A method and device for generating expression animations

Abstract

An embodiment of the present invention provides a method and apparatus for generating an expression animation. The method includes: obtaining audio data and / or text data; driving the audio data and / or text data to obtain phoneme-based facial animation data; converting the phoneme-based facial animation data into animation data of a preset standard; obtaining a set of bone expressions corresponding to facial bones; and driving the set of bone expressions according to the animation data of the preset standard to generate a first expression animation of the facial bones. This ensures the generality of the expression animation production process, is compatible with different facial data sources, provides a basic condition for subsequent decoupling of facial animation data, and the method of driving animation data based on a set of bone expressions can decouple the animation data from specific facial skeleton data. The same set of bone expressions can be output for different skeletons, enabling animation retargeting and effectively improving the time for producing expression animations.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method for generating expression animations and a device for generating expression animations. Background Art

[0002] For games with a large amount of plot interactions, a programmatic expression generation method is usually adopted to improve the production efficiency of expression animations in the game. With the development of AI (Artificial Intelligence) technology, the method of generating expression animations based on audio by AI is a generation method with relatively high efficiency and less dependence on external conditions among various programmatic expression generation methods, and thus is often adopted in game production.

[0003] Currently, for AI voice generation of expression animations, the general solution only requires providing audio and the corresponding text file, generating the weight values of specific expression Pose postures through an AI engine, and finally generating the corresponding expression animations by mixing the weight values of different expressions. Although this solution for generating expression animations by AI voice has strong commonality in the process of AI processing, the animation data extracted by the AI engine from audio and text is only weight animation data based on Phoneme phonemes. These animation data are not easy to standardize and have poor scalability, and the animation data and bone data are not decoupled. If the bone data needs to be changed, additional production iteration time is required, which is likely to form a bottleneck in production efficiency. Summary of the Invention

[0004] In view of the problem that the weight animation data extracted from audio and text is only based on Phoneme phonemes, is not easy to standardize and has poor scalability, and the animation data and bone data are not decoupled, resulting in a bottleneck in production efficiency, embodiments of the present invention are proposed to provide a method for generating expression animations and a corresponding device for generating expression animations that overcome the above problems or at least partially solve the above problems.

[0005] Embodiments of the present invention disclose a method for generating expression animations, including:

[0006] Obtaining audio data and / or text data;

[0007] Driving the audio data and / or text data to obtain phoneme-based facial animation data;

[0008] Converting the phoneme-based facial animation data into animation data of a preset standard;

[0009] Obtaining a bone expression set corresponding to facial bones;

[0010] Drive the skeletal expression set according to the animation data of the preset standard to generate the first expression animation of the facial skeleton.

[0011] Optionally, the animation data of the preset standard is FACS standard animation data. The conversion of the phoneme-based facial animation data into the animation data of the preset standard includes:

[0012] According to a preset mapping table, convert the phoneme-based facial animation data into the FACS standard animation data; wherein, the preset mapping table is used to describe the mapping relationship between the phoneme expression set and the FACS expression set.

[0013] Optionally, after the step of converting the phoneme-based facial animation data into standard animation data, it further includes:

[0014] Perform post-processing operations on the animation data of the preset standard to improve the accuracy of the animation data of the preset standard.

[0015] Optionally, the method further includes:

[0016] Extract features from the audio data and / or text data to obtain a target emotion weight value;

[0017] Drive the skeletal expression set according to the target emotion weight value to generate the second expression animation of the facial skeleton;

[0018] Fuse the first expression animation and the second expression animation to obtain a full-face expression animation.

[0019] Optionally, the extraction of features from the audio data and / or text data to obtain a target emotion weight value includes:

[0020] Extract features from the audio data and / or text data to obtain a first emotion weight value;

[0021] Obtain text annotation data, and extract features from the text annotation data to obtain an annotated emotion weight value;

[0022] Perform a fusion process on the annotated emotion weight value and the first emotion weight value to obtain a target emotion weight value.

[0023] Optionally, the first expression animation is an expression animation of the lower half of the face, and the second expression animation is an expression animation of the upper half of the face.

[0024] Optionally, it further includes:

[0025] Bind the full-face expression animation to the facial skeleton to play the expression animation on the facial skeleton.

[0026] An embodiment of the present invention also discloses an apparatus for generating expression animations, including:

[0027] An audio data acquisition module, configured to acquire audio data and / or text data;

[0028] An animation data driving module, configured to drive the audio data and / or text data to obtain phoneme-based facial animation data;

[0029] An animation data conversion module, configured to convert the phoneme-based facial animation data into animation data of a preset standard;

[0030] A skeletal expression set acquisition module, configured to acquire a skeletal expression set corresponding to facial bones;

[0031] A first expression animation generation module, configured to drive the skeletal expression set according to the animation data of the preset standard to generate a first expression animation of the facial bones.

[0032] An embodiment of the present invention also discloses an electronic device, including:

[0033] A processor and a storage medium, where the storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor executes the machine-readable instructions to execute the method according to any one of the embodiments of the present invention.

[0034] An embodiment of the present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the method according to any one of the embodiments of the present invention.

[0035] The embodiments of the present invention have the following advantages:

[0036] In the embodiments of the present invention, by acquiring audio data and / or text data; driving the audio data and / or text data to obtain phoneme-based facial animation data; converting the phoneme-based facial animation data into animation data of a preset standard; acquiring a skeletal expression set corresponding to facial bones; driving the skeletal expression set according to the animation data of the preset standard to generate a first expression animation of the facial bones. By converting the facial animation data into animation data of a preset standard, the generality of the expression animation production process can be ensured, different facial data sources can be compatible, and a basic condition for subsequent decoupling of facial animation data can be provided. Moreover, by driving the animation data based on the skeletal expression set, the animation data can be decoupled from specific facial skeleton data. The same skeletal expression set can be output for different skeletons, and animation redirection can be completed, enabling the independent iteration of animation data and bone binding data, effectively improving the time for producing expression animations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required for the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0038] Figure 1 is a flowchart of the expression generation of a voice-driven AI engine in the prior art;

[0039] Figure 2 is a step flowchart of a method for generating an expression animation provided by an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of a two-dimensional mapping table provided by an embodiment of the present invention;

[0041] Figure 4 is a flowchart of generating an expression animation provided by an embodiment of the present invention;

[0042] Figure 5 is a step flowchart of generating a facial expression animation provided by an embodiment of the present invention;

[0043] Figure 6 is a flowchart of an expression set driven by FACS standard animation data for different skeletons provided by an embodiment of the present invention;

[0044] Figure 7 is a structural block diagram of a device for generating an expression animation provided by an embodiment of the present invention;

[0045] Figure 8 is a structural block diagram of an electronic device of the present invention;

[0046] Figure 9 is a structural block diagram of a computer-readable storage medium of the present invention. Detailed Embodiments

[0047] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0048] In the prior art, such as Figure 1As shown in the figure, the expression generation scheme of the voice-driven AI engine generally includes the following steps: (1) Prepare the data pairs of audio and text (in some schemes, either audio or text can be selected), and import them into the AI engine for analysis; (2) Export the animation weights related to the voice Phoneme from the AI engine, and prepare the corresponding expression poses; (3) According to the generated animation weights, drive the corresponding Phoneme expression base poses, and output the animation sequence of a specific skeleton.

[0049] As mentioned above, the existing expression generation schemes of voice-driven AI engines have strong commonality in the process of AI processing. However, in the subsequent application methods of animation data and animation production processes, there is a lack of a production scheme with strong scalability, strong generality, and low coupling degree. It can be summarized as the following aspects:

[0050] (1) Animation data is not easy to standardize and has poor scalability: The AI engine only extracts weight animation data based on Phoneme from audio and text. This data is highly related to voice, lacks reference to the facial muscle system, lacks good facial anatomy basis, and is not easy to standardize for production. When wanting to fuse animation data such as facial capture, it cannot be directly used because facial capture animation data is usually animation data based on the facial muscle system.

[0051] (2) Animation data and bone data are not decoupled: In the existing expression animation production process, when the bone binding data is changed, the animation data needs to be updated and output again, which consumes additional production iteration time and is prone to form a bottleneck in production efficiency.

[0052] (3) It is not convenient for animation data redirection: Since the bone animation driving method with higher running efficiency is usually used in games, when wanting to perform facial lip-sync animation redirection between two different skeletons, there is a lack of an effective redirection mechanism.

[0053] (4) Lack of support for upper half-face expression animation: For the expression animation generation based on the voice-driven AI engine, because the input information is only related to voice, there is a lack of sufficient information support when generating upper half-face expression animations that are not related to voice.

[0054] To solve the problems that the above-mentioned general expression animation production scheme based on AI voice driving will encounter, the present invention specifically proposes the following technical precautions, and the technical principle is as follows:

[0055] (1) Build a standardized facial data recording method based on FACS (Facial Action Coding System). Through a two-dimensional mapping table between phonemes and the FACS expression set, the mutual conversion between FACS data and phoneme data can be realized, so that the seamless connection between speech facial animation data and facial capture animation data can be achieved. FACS is a set of standard expression coding methods with a mature history. The present invention uses FACS as the basis of data, and corresponding conversions are performed between other types of facial expression data and it. The conversion technology relies on a two-dimensional mapping conversion table (lookup table).

[0056] (2) In order to completely decouple the facial animation data and the facial bone binding data, an independent FACS standard animation data layer is introduced. This animation data corresponds to the Pose of the FACS standard bone expression set. When it is necessary to update the animation data or the bone data, as long as the same set of FACS expression sets is followed, conversion can be performed between different FACS standard bone Poses based on the FACS expression set.

[0057] (3) In order to obtain the animation weight of the upper half face expression, based on NLP (Natural Language Processing) semantic feature extraction, the emotional semantic weight (such as joy, anger, sorrow, and happiness) of the input text of the corresponding AI engine is obtained, and combined with the stress strength of the speech, post-processing is performed on the extracted emotional semantic weight. For scenarios that require manual intervention, the method of manually marking text emotions with data can be combined to obtain automatic expression data. In the final data fusion, the upper and lower half faces are split into corresponding FACS control parameters for full-face animation fusion.

[0058] Refer to Figure 2 , which shows the step flow chart of an embodiment of a method for generating an expression animation provided by an embodiment of the present invention, and specifically may include the following steps:

[0059] Step 201, obtain audio data and / or text data;

[0060] Among them, the audio data refers to some audio that can express the emotions of a person, and the text data refers to some text that can express the emotions of a person. For example, the text "The weather is really nice today" can show the emotional characteristics of a person's joy.

[0061] Step 202, drive the audio data and / or text data to obtain phoneme-based facial animation data;

[0062] Specifically, by calling the AI engine interface, voice driving is performed on the audio data and / or text data to obtain phoneme-based facial animation data. Among them, the AI engine is a general engine for voice driving, and the output of the AI engine is facial animation data around phonemes. Among them, the phoneme Phoneme is the smallest speech unit, including vowels and consonants. For example, vowels a / o / e, etc.

[0063] Step 203, convert the phoneme-based facial animation data into animation data of a preset standard;

[0064] Due to the recording method of phoneme-based facial animation data, it conforms to the speech intuition of people, but it is not convenient for artists to make standardized expression animations. In the embodiment of the present invention, the phoneme-based facial animation data can be converted into animation data of a preset standard. For example, if the preset standard is the FACS standard, the facial animation data can be converted into animation data recorded in the FACS manner. On the one hand, it is more in line with human biological characteristics, because FACS is based on the human face muscle movement system, and each AU (Action Unit) is independent of each other, which is convenient for linear mixing and interpolation between each movement unit; on the other hand, FACS can well adapt to the animation data generated by facial capture and has been fused between various data sources.

[0065] Step 204, obtain the bone expression set corresponding to the facial bones;

[0066] Among them, the bone expression set corresponding to the facial bones can be a bone expression set based on the FACS standard, and each facial bone has a corresponding bone expression set. Among them, the bone expression set can be composed of Visemes. A Viseme is a visual unit of an expression Pose. The visual effects of expressions of the same Viseme are the same, and the visual effects of expressions of different Visemes are different.

[0067] Step 205, drive the bone expression set according to the animation data of the preset standard to generate the first expression animation of the facial bones.

[0068] In the embodiment of the present invention, the bone expression set can be driven according to the animation data of the preset standard to generate the first expression animation of the facial bones, that is, when driving the expression animation, it is realized based on the Pose driven by the bones. In most cases, the performance of the Pose driven by the bones is better than other Pose schemes. For example, many games use the mesh Pose based on the blend shape animation to drive, but the blend shape animation usually has a greater overhead on the computing performance, and the production of art resources is more cumbersome, and the efficiency iteration of assets is lower. Therefore, the above idea is not adopted in the present invention example, but this process method is completely compatible with the driving method of the blend shape animation Pose.

[0069] In a specific implementation, in a 3D modeling software tool (such as Maya software), first obtain the phoneme-based facial animation data driven by the AI engine programmatically. After the above transformation of the phoneme-based facial animation data, the corresponding animation data of the bone Pose can be obtained. Then, write the preset standard animation data into the custom data segment of the root bone, and export the expression animation in the form of custom data by means of a 3D asset exchange file (such as the FBX format). This is different from the traditional way of exporting based on bone data. It should be noted that by using the method of driving through the preset standard written into the custom data segment of the root bone, the impact on the original data of the root bone can be avoided, and it is convenient for technicians to flexibly adjust the bone data to adjust the performance effect of the expression animation generated on the bone.

[0070] In a preferred embodiment of the present invention, the preset standard animation data is FACS standard animation data, and the step 203 may specifically include the following sub-steps:

[0071] According to a preset mapping table, convert the phoneme-based facial animation data into the FACS standard animation data; wherein, the preset mapping table is used to describe the mapping relationship between the phoneme expression set and the FACS expression set.

[0072] Among them, the preset mapping table is a pre-set mapping relationship table used to describe the mapping relationship between the phoneme expression set and the FACS expression set. The preset mapping table can be a two-dimensional mapping table, including two dimensions of the phoneme expression set and the FACS expression set.

[0073] In order to facilitate the mutual conversion between the phonemes output by the AI engine and FACS, a preset mapping table is designed. Since the relationship between phonemes and FACS movement units is not one-to-one, this table is used to find out which FACS movement units affect a phoneme and record them in the form of weights. The actual storage method of the preset mapping table can be recorded in the form of CSV (Comma-Separated Values, character-separated value file) or JSON format. Since FACS only defines a framework specification based on facial movements, which specific movement units to select can be adjusted according to the implementation of the actual project. For example, 52 bone expression Poses can be selected, and each expression Pose is single-frame facial expression bone data or facial mesh data, such as Figure 3As shown, the phoneme expression set contains 12, and the bone expression set contains 52. Among them, the 52 poses consist of 1 neutral expression pose (i.e., AU0, the pose of the initial binding of facial bones, the basic motion unit of AU (Action Unit) FACS) and 51 common FACS motion units, including jawForward (lower jaw forward), jawLeft (lower jaw to the left), noseSneer_L (left nasal wing lift), noseSneer_R (right nasal wing lift), etc.

[0074] As an example, the preset mapping table is shown in Table 1 below. Among them, the sum of the weight components of each phoneme corresponding to each FACS in the table does not require normalization processing.

[0075]

[0076] Table 1

[0077] In the embodiments of the present invention, by looking up the preset mapping table, the phoneme-based facial animation data can be converted into FACS standard animation data.

[0078] In a preferred embodiment of the present invention, after step 203, the following may further be included:

[0079] Perform post-processing operations on the preset standard animation data to improve the accuracy of the preset standard animation data.

[0080] In the embodiments of the present invention, post-processing operations can be performed on the converted preset standard animation data to improve the accuracy of the preset standard animation data, and further optimize the performance effect of the expression animation generated subsequently according to the preset standard animation data.

[0081] Among them, the post-processing operations include at least one of the following: noise smoothing operation, time synchronization calibration operation, and data cleaning operation.

[0082] In a preferred embodiment of the present invention, the method further includes:

[0083] Extract features from the audio data and / or text data to obtain the target emotion weight value; drive the bone expression set according to the target emotion weight value to generate the second expression animation of the facial bones; fuse the first expression animation and the second expression animation to obtain the full-face expression animation.

[0084] Specifically, in the above step 202, the phoneme-based facial animation data is obtained by driving audio data and / or text data. This facial animation data is usually used to represent the animation effect of the lower half of the face, but the expression effect of the upper half of the face is not good. In the embodiments of the present invention, text annotation data is introduced to better represent the expression animation effect of the full face of a person.

[0085] In a specific implementation, in order to obtain the data of the emotional animation of the upper half of the facial expression, with the help of the emotional semantic extraction framework of NLP (Natural Language Processing), the tendency and weight of the emotion are first analyzed from the text. For example, for the text "The weather is really nice today", a positive happy emotion weight of 70% is analyzed. Then, combined with the stress of the audio and the keywords of the text emotion, the weight at the single-word granularity can be located. For example, the weight of "really nice" is greater than that of "The weather today". The weights between single words are smoothed by means of cosine interpolation and other methods.

[0086] After obtaining the target emotion weight value through feature extraction, the skeletal expression set can be driven according to the target emotion weight value to generate the second expression animation of the facial skeleton. The full-face expression animation can be obtained by fusing the first expression animation and the second expression animation, where the first expression animation is the expression animation of the lower half of the face, and the second expression animation is the expression animation of the upper half of the face.

[0087] In a preferred embodiment of the present invention, performing feature extraction on the audio data and / or text data to obtain a target emotion weight value includes:

[0088] Performing feature extraction on the audio data and / or text data to obtain a first emotion weight value;

[0089] Obtaining text annotation data, and performing feature extraction on the text annotation data to obtain an annotated emotion weight value; performing a fusion process on the annotated emotion weight value and the first emotion weight value to obtain a target emotion weight value.

[0090] Among them, the text annotation data refers to some text data annotated with the emotions of a person. In the embodiments of the present invention, if it is necessary to fuse the manually marked emotion data, on the basis of the emotion weight output by the program, the annotated emotion weight value obtained by performing feature extraction on the text annotation data and the first emotion weight value calculated above can be fused to obtain the final required target emotion weight value.

[0091] In a specific implementation, the labeled emotion weight value and the first emotion weight value can be fused by linear interpolation to obtain the final target emotion weight. The fused target emotion weight is referenced to the corresponding emotion FACS Pose, and the FACS expression animation corresponding to the upper half of the face can be generated. Finally, it is fused with the lower half face lip animation to synthesize the final full-face animation data. As Figure 4 shown, by extracting emotion weight values (such as joy, anger, sorrow, happiness, etc.) from audio data and text data through NLP emotion semantic features, and fusing them with the weight values extracted from the manually labeled data representing text emotion features, and then driving the corresponding FACS Pose with the fused emotion weight values to generate the upper half face expression animation of the face. Fusing the upper half face expression animation of the face and the lower half face expression animation of the face can obtain the full-face animation of the face.

[0092] In a preferred embodiment of the present invention, the method further includes:

[0093] Binding the full-face expression animation to the facial bones to play the expression animation on the facial bones.

[0094] In an embodiment of the present invention, by binding the full-face expression animation to the facial bones, the expression animation is played on the facial bones. Specifically, in a game engine, the custom expression animation exported from a 3D asset exchange file (such as FBX format) can be parsed and stored in the game engine for binding to different facial bones to play the facial expression animation. To accelerate the entire production process, the present invention has developed an automated execution script tool in a 3D modeling software (such as Maya). This tool follows the above-mentioned operation process of the embodiment of the present invention and is used to automatically batch export the corresponding custom facial expression animation data, such as Figure 5 shown, including step 501, generating a phoneme expression weight animation by calling the AI engine interface and caching it locally; step 502, performing the mapping process from phonemes to FACS to generate FACS-based expression animation data; step 503, performing post-processing operations on the FACS expression animation data, such as noise smoothing, time synchronization calibration, etc.; step 504, writing the processed FACS expression animation data into the custom attribute data segment of the root bone; step 505, exporting it to the game engine in the FBX file exchange format.

[0095] In an embodiment of the present invention, the FACS standard animation data and specific facial skeleton data are completely decoupled. The FACS standard animation data can be used in multiple bones, and different expression animations with different performance effects can be generated when used in different bones. Moreover, after changing the bone binding data, the FACS standard animation data does not need to be updated and output again, avoiding consuming additional production iteration time and improving the production efficiency of expression animations. AsFigure 6 As shown in the figure, assume there are two facial skeletons, namely facial skeleton A and facial skeleton B. Among them, the FACS expression set A can be derived from facial skeleton A, and the FACS expression set B can be derived from facial skeleton B. The FACS standard animation data is input into the FACS expression set A for driving to obtain the facial animation of skeleton A, and the FACS standard animation data is input into the FACS expression set B for driving to obtain the facial animation of skeleton B.

[0096] Through the above processing steps, the embodiment of the present invention proposes an AI-programmed expression animation generation scheme for voice-driven expressions with strong versatility, low data coupling degree, and separate processing of the upper and lower half faces, which can be used as a general production process for voice-driven AI expression production in game production. This scheme has the following advantages:

[0097] (1) The method of recording and converting facial animation data based on FACS can ensure the versatility of this production process, be compatible with different facial data sources, and provide basic conditions for subsequent decoupling of facial animation data.

[0098] (2) By writing the animation data into the custom data segment of the root bone based on the bone Pose driving method according to the FACS standard, the animation data can be decoupled from the specific facial skeleton data. Different bones output the same FACS Pose set, and then the animation can be redirected based on FACS, enabling the independent iteration of the animation data and the bone binding data, effectively improving the animation production time.

[0099] (3) Based on the fact that voice driving is good at generating lower half face animations, a highly efficient method based on NLP is proposed to automatically extract the emotional semantic features of the text, and fuse the stress weight of the audio and the artificial marking weight to adapt to various production methods and enhance applicability.

[0100] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present invention.

[0101] Referring to Figure 7 , a structural block diagram of a device for generating expression animations provided by an embodiment of the present invention is shown, which may specifically include the following modules:

[0102] The audio data acquisition module 701 is used to acquire audio data and / or text data;

[0103] The animation data driving module 702 is used to drive the audio data and / or text data to obtain phoneme-based facial animation data;

[0104] The animation data conversion module 703 is used to convert the phoneme-based facial animation data into animation data of a preset standard;

[0105] The skeletal expression set acquisition module 704 is used to acquire the skeletal expression set corresponding to the facial skeleton;

[0106] The first expression animation generation module 705 is used to drive the skeletal expression set according to the animation data of the preset standard to generate the first expression animation of the facial skeleton.

[0107] In a preferred embodiment of the present invention, the animation data of the preset standard is FACS standard animation data, and the animation data conversion module 703 includes:

[0108] The data conversion sub-module is used to convert the phoneme-based facial animation data into the FACS standard animation data according to a preset mapping table; wherein, the preset mapping table is used to describe the mapping relationship between the phoneme expression set and the FACS expression set.

[0109] In a preferred embodiment of the present invention, it further includes:

[0110] The data post-processing module is used to perform post-processing operations on the animation data of the preset standard to improve the accuracy of the animation data of the preset standard.

[0111] In a preferred embodiment of the present invention, the method further includes:

[0112] The feature weight extraction module is used to perform feature extraction on the audio data and / or text data to obtain the target emotion weight value;

[0113] The second expression animation generation module is used to drive the skeletal expression set according to the target emotion weight value to generate the second expression animation of the facial skeleton;

[0114] The expression animation fusion module is used to fuse the first expression animation and the second expression animation to obtain the full-face expression animation.

[0115] In a preferred embodiment of the present invention, the feature weight extraction module includes:

[0116] The first emotion weight value extraction sub-module is used to perform feature extraction on the audio data and / or text data to obtain the first emotion weight value;

[0117] The labeled emotion weight value extraction sub-module is used to obtain text annotation data, and perform feature extraction on the text annotation data to obtain the labeled emotion weight value;

[0118] The weight value fusion sub-module is used to perform a fusion process on the labeled emotion weight value and the first emotion weight value to obtain the target emotion weight value.

[0119] In a preferred embodiment of the present invention, the first expression animation is an expression animation of the lower half of the face, and the second expression animation is an expression animation of the upper half of the face.

[0120] In a preferred embodiment of the present invention, it further includes:

[0121] The expression animation playback module is used to bind the full-face expression animation to the facial bones to play the expression animation on the facial bones.

[0122] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment.

[0123] The embodiment of the present invention also provides an electronic device, as Figure 8 shown, including:

[0124] A processor 801 and a storage medium 802, the storage medium 802 stores machine-readable instructions executable by the processor 801. When the electronic device runs, the processor 801 executes the machine-readable instructions to execute the method according to any one of the embodiments of the present invention. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0125] The embodiment of the present invention also provides a computer-readable storage medium, as Figure 9 shown, a computer program 901 is stored on the storage medium. When the computer program 901 is run by a processor, it executes the method according to any one of the embodiments of the present invention. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0126] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to describe the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other.

[0127] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, an apparatus, or a computer program product. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0128] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0129] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices, such that a series of operation steps are executed on the computer or other programmable terminal devices to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable terminal devices provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0131] Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the embodiments of the present invention.

[0132] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

[0133] The above provides a detailed introduction to a method and a device for generating expression animations according to the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for generating expression animations, characterized in that, Including: Obtaining audio data and / or text data; Driving the audio data and / or text data to obtain phoneme-based facial animation data; Converting the phoneme-based facial animation data into animation data of a preset standard; Obtaining a set of bone expressions corresponding to facial bones; Driving the set of bone expressions according to the animation data of the preset standard to generate a first expression animation of the facial bones; The driving refers to writing the animation data of the preset standard into a custom data segment of the root bone; The method further includes: Performing feature extraction on the audio data and / or text data to obtain a target emotion weight value; Driving the set of bone expressions according to the target emotion weight value to generate a second expression animation of the facial bones; Fusing the first expression animation and the second expression animation to obtain a full-face expression animation.

2. The method according to claim 1, wherein The animation data of the preset standard is FACS standard animation data, and the converting the phoneme-based facial animation data into animation data of the preset standard includes: Converting the phoneme-based facial animation data into the FACS standard animation data according to a preset mapping table; wherein, the preset mapping table is used to describe the mapping relationship between the set of phoneme expressions and the set of FACS expressions.

3. The method according to claim 1, characterized in that, After the step of converting the phoneme-based facial animation data into standard animation data, it further includes: Performing post-processing operations on the animation data of the preset standard to improve the accuracy of the animation data of the preset standard.

4. The method according to claim 1, characterized in that Performing feature extraction on the audio data and / or text data to obtain a target emotion weight value, including: Performing feature extraction on the audio data and / or text data to obtain a first emotion weight value; Obtaining text annotation data and performing feature extraction on the text annotation data to obtain an annotation emotion weight value; Performing fusion processing on the annotation emotion weight value and the first emotion weight value to obtain a target emotion weight value.

5. The method according to claim 4, wherein The first expression animation is an expression animation of the lower half of the face, and the second expression animation is an expression animation of the upper half of the face.

6. The method according to claim 5, wherein It further includes: Binding the full-face expression animation to the facial bones to play the expression animation on the facial bones.

7. An apparatus for generating an expression animation, characterized in that, Including: An audio data acquisition module for obtaining audio data and / or text data; An animation data driving module for driving the audio data and / or text data to obtain phoneme-based facial animation data; An animation data conversion module for converting the phoneme-based facial animation data into animation data of a preset standard; A set of bone expressions acquisition module for obtaining a set of bone expressions corresponding to facial bones; A first expression animation generation module for driving the set of bone expressions according to the animation data of the preset standard to generate a first expression animation of the facial bones; The driving refers to writing the animation data of the preset standard into a custom data segment of the root bone; The device further includes: A feature weight extraction module for performing feature extraction on the audio data and / or text data to obtain a target emotion weight value; A second expression animation generation module for driving the set of bone expressions according to the target emotion weight value to generate a second expression animation of the facial bones; An expression animation fusion module, configured to fuse the first expression animation and the second expression animation to obtain a full-face expression animation.

8. An electronic device, characterized in that, It includes: A processor and a storage medium, where the storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor executes the machine-readable instructions to perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is run by the processor, it performs the method according to any one of claims 1-6.

Images