Method, apparatus and program product for generating avatar animation

Abstract

The invention relates to a method, a device and a computer program product for generating an avatar animation. The method includes generating an animation instruction vector for the avatar animation based on the text input. The method also includes determining an animation sequence of the avatar animation based on the animation instruction vector, where the animation sequence indicates a plurality of frames of the avatar animation and transformations between the plurality of frames. The method also includes determining a facial mix shape of the avatar animation based on the animation instruction vector, where the facial mix shape indicates a facial expression of the avatar animation. In addition, the method also includes generating an avatar animation corresponding to the text based on the animation sequence and the face mix shape. Through the mode, the input text can be accurately understood, so that a more natural and smoother coherent animation rich in facial expression details can be generated, and the user experience is further improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computers, and more particularly, to a method, an electronic device and a product for generating avatar animation. BACKGROUND

[0002] Avatar animation refers to the process of creating and animating virtual characters using technical means. These virtual characters, often referred to as "avatars", can be used in a variety of applications. In the process of making avatar animation, the appearance of the avatar is first designed and modeled, which includes its shape, clothing, facial features, etc. Then, animators add actions and expressions to these avatars, making them able to present realistic dynamic effects. These actions and expressions can be achieved through techniques such as keyframe animation, motion capture, etc.

[0003] Avatar animation is gradually penetrating into more industries and scenarios. In the game industry, exquisite avatar animation can enhance the immersion and game experience of players; in film and television production, avatar animation can be used to create special effect characters or achieve actions that actors cannot complete; in virtual reality and augmented reality applications, avatar animation can provide users with a more realistic and personalized virtual experience. SUMMARY

[0004] Embodiments of the present disclosure provide a method, an electronic device and a computer program product for generating avatar animation.

[0005] In a first aspect of embodiments of the present disclosure, a method for generating avatar animation is provided. The method includes generating, based on a text input, an animation instruction vector for the avatar animation. The method further includes determining, based on the animation instruction vector, an animation sequence of the avatar animation, wherein the animation sequence indicates a plurality of frames of the avatar animation and transformations between the plurality of frames. The method further includes determining, based on the animation instruction, a facial blend shape of the avatar animation, the facial blend shape indicating a facial expression of the avatar animation. In addition, the method further includes generating the avatar animation corresponding to the text based on the animation sequence and the facial blend shape.

[0006] In a second aspect of embodiments of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device storing one or more programs, when executed by the one or more processors, cause the one or more processors to implement a method for generating an avatar animation. The method includes generating, based on a text input, an animation instruction vector for the avatar animation. The method further includes determining, based on the animation instruction vector, an animation sequence of the avatar animation, wherein the animation sequence indicates a plurality of frames of the avatar animation and a transformation between the plurality of frames. The method further includes determining, based on the animation instruction vector, a blendshape of the avatar animation, the blendshape indicating a facial expression of the avatar animation. In addition, the method further includes generating the avatar animation corresponding to the text based on the animation sequence and the blendshape.

[0007] In a third aspect of embodiments of the present disclosure, a computer-readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements a method for generating an avatar animation. The method includes generating, based on a text input, an animation instruction vector for the avatar animation. The method further includes determining, based on the animation instruction vector, an animation sequence of the avatar animation, wherein the animation sequence indicates a plurality of frames of the avatar animation and a transformation between the plurality of frames. The method further includes determining, based on the animation instruction vector, a blendshape of the avatar animation, the blendshape indicating a facial expression of the avatar animation. In addition, the method further includes generating the avatar animation corresponding to the text based on the animation sequence and the blendshape.

[0008] It is to be understood that the description in the Summary section is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features, aspects, and advantages of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other features, aspects, and advantages of embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart showing a method for generating an avatar animation according to some embodiments of the present disclosure;

[0012] Figure 3 A schematic diagram showing generation of an avatar animation based on an avatar animation system according to some embodiments of the present disclosure;

[0013] Figure 4A diagram showing a method for determining an animation sequence according to some embodiments of the present disclosure is shown.

[0014] Figure 5 A diagram showing a method for determining an animation sequence according to some embodiments of the present disclosure is shown.

[0015] Figure 6 A diagram showing a method for determining a facial blendshape according to some embodiments of the present disclosure is shown.

[0016] Figure 7 A block diagram of a device that can implement various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather should be interpreted within the fullest scope consistent with the appended claims. It is to be understood that the drawings and the embodiments are merely for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0018] In the description of embodiments of the present disclosure, the term “includes” and its variants are to be read as open-ended terms that mean “comprises.” The term “based on” is to be interpreted as “based, at least in part, on.” The term “one embodiment” or “the embodiment” are to be interpreted as “at least one embodiment.” The terms “first,” “second,” etc. can refer to different or the same objects. Other explicit or implicit definitions can also be included below.

[0019] Animating avatar characters based on text input is a challenging task that requires capturing subtle nuances, emotions, and context in the text and translating them into lifelike, coherent, and contextually appropriate avatar animations. Traditional approaches rely on hand-crafted rules, pre-defined templates, or limited datasets. Rule-based approaches use hand-crafted rules to map text input to animation parameters such as facial expressions, gestures, and body movements. However, rule-based approaches are limited by the expressiveness and diversity of pre-defined rules, which can not cover all possible text input scenarios and variations. Template-based approaches use pre-defined templates or scripts to generate animations based on text input. However, template-based approaches are limited by the availability and quality of templates, which can not match the style and context of text input.

[0020] To this end, an embodiment of the present disclosure proposes a solution for generating avatar animation. In an embodiment of the present disclosure, first, a text is input, and this text will be used by the system to generate action instruction vectors for the avatar animation. Then, the system will determine a series of actions of the avatar animation based on these instruction vectors. These actions include multiple different postures of the avatar animation, as well as the transition process from one posture to another. In this way, an animation sequence can be obtained. At the same time, the system will also determine the facial expressions of the avatar animation based on these animation instruction vectors. Finally, the system will combine these action sequences with facial expressions to generate a complete avatar animation. This avatar animation will make corresponding actions and expressions based on the input text.

[0021] This approach allows for a more accurate understanding of text input, resulting in more natural and fluid animation sequences. Furthermore, it generates animations with detailed and coherent facial expressions, greatly enhancing the vividness and realism of the generated avatar animations and improving the user experience.

[0022] Figure 1 1 shows a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1 As shown, input text 110 can be processed by animation generation system 120 to generate avatar animation 130, which corresponds to input text 110. For example, if the input text is "He jumped up with joy," animation generation system 120 can generate an avatar animation with a smiling facial expression and jumping off the ground. The facial expression can also have more facial details, such as slightly narrowed eyes and a raised corner of the mouth. Animation generation system 120 includes an animation interpreter 122 based on a large language model (LLM), an avatar animation engine 124, and a facial expression generative adversarial network 126.

[0023] In some embodiments, the large language model based animation interpreter 124 can be used to generate the animation instruction vector. For example, when the input text is “he jumped up happily”, it can generate an animation instruction vector indicating [expression: smile; body action: jump with both feet off the ground]. In some embodiments, the avatar animation engine 126 can be used to generate the concrete animation sequence. For example, for the animation instruction vector indicating [expression: smile; body action: jump with both feet off the ground], the avatar animation engine 126 can output an animation sequence of a continuous jump, in which the avatar’s expression is a smile. In some embodiments, the facial expression generation adversarial network 126 can generate facial expression details of slightly squinting eyes and upturned corners of the mouth for the animation instruction vector indicating [expression: smile]. In some embodiments, the synthesis of the generated facial expression details and the generated animation sequence of a continuous jump can generate an avatar animation 130 that is both continuous and expressive corresponding to the input text “he jumped up happily”.

[0024] Figure 2 A flowchart of a method 200 for generating an avatar animation is shown, according to some embodiments of the present disclosure. At block 202, based on a text input, an animation instruction vector for an avatar animation is generated. In some embodiments, the large language model based animation interpreter 124 can be used to generate the animation instruction vector. For example, when the input text is “he jumped up happily”, it can generate an animation instruction vector indicating [expression: smile; body action: jump with both feet off the ground].

[0025] At block 204, based on the animation instruction vector, an animation sequence of the avatar animation is determined, the animation sequence indicating a plurality of frames of the avatar animation and a transformation between the plurality of frames. Animation sequence refers to a series of consecutive pictures arranged in a specific order and frame rate to form a smooth dynamic image in animation production. In some embodiments, the avatar animation engine 126 can be used to generate the concrete animation sequence. For example, for the animation instruction vector indicating [expression: smile; body action: jump with both feet off the ground], the avatar animation engine 126 can output an animation sequence as follows: at the beginning, the avatar animation character stands on the ground, and the character’s facial expression is relaxed and happy; in preparation for jumping, the avatar animation character starts to bend the knees and the body slightly sinks, preparing for the jump; jump up, the avatar animation character suddenly kicks the ground with force, and the body quickly rises; in the air, the character’s body remains stretched, and the arms may be spread upwards.

[0026] At block 206, facial blend shapes for the avatar animation are determined based on the animation instruction vector. The facial blend shapes indicate the facial expression of the avatar animation. Facial blend shapes can divide the facial region into different parts, such as eyes, eyebrows, nose, and mouth, and create a series of preset shapes for each part. In some embodiments, the facial expression generative adversarial network 126 can generate facial expression details such as slightly narrowed eyes and raised corners of the mouth for the animation instruction vector indicating [expression: smile].

[0027] At block 208, an avatar animation corresponding to the text is generated based on the animation sequence and the facial blend shapes. In some embodiments, the generated animation sequence can be merged with the facial expression details at corresponding moments to generate an avatar animation with rich and coherent facial expression details.

[0028] By analyzing the semantics of the input text to generate coherent animation sequences and combining this with facial blend shape technology, the system can deeply understand the input text content and create more natural and smooth avatar animations based on this. This method not only makes the overall rhythm and movement transitions of the avatar animation more harmonious, but also ensures the coherence of the plot development, providing users with a more comfortable visual experience. At the same time, with the help of facial blend shape technology, it can generate delicate and coherent animation effects of facial expressions, enhancing the vividness and realism of the generated avatar animation, thereby further improving the user experience.

[0029] Figure 3 FIG. 3 shows a schematic diagram of generating an avatar animation 300 based on an avatar animation system according to some embodiments of the present disclosure. Figure 3 As shown, the input text 310 can be transformed into an input animation 330 corresponding to the input text by the animation generation system 320. Figure 3 The text processing component 322 can process the input text 310 to obtain emotional features or contextual features. The text processing component 322 is the initial module of the system, which is mainly responsible for parsing and understanding the input text. It uses natural language processing (NLP) technology to analyze the emotions, moods and intentions of the text, laying the foundation for the creation of accurate avatar animation. During the processing process, the text processing component 322 will perform a series of complex operations, including lexical analysis, syntactic analysis, semantic understanding, etc., to extract key information and features from the text. These features include not only the meaning of words, phrases and sentences, but also the emotional tendency, emotional intensity and intention behind the text. Specifically, formula (1) represents the process of natural language processing of the input text T by the text processing component 322, and the output result is E. E represents the emotional and contextual features extracted from the text input T, which will serve as an important basis for subsequent animation creation.

[0030] E=NPL(T) (1)

[0031] For example, if the text input 310 is "She jumped for joy," the text processing component 322 can identify the characteristic "happy" and pass this emotional characteristic to the subsequent animation interpreter 324 to generate an avatar animation expressing a happy emotion. In some embodiments, the text processing component 322 can process the input long text into short text to facilitate the animation interpreter 324 based on the language model to better understand the input text 310.

[0032] By accurately extracting and analyzing these features, the text processing component 322 can provide strong support for subsequent animation sequence design and facial blend shape generation, which enables the generated avatar animation to be closer to the text content and achieve more accurate emotional and action expression, thereby improving the quality of the entire avatar animation work and further enhancing the user experience.

[0033] Continue to refer Figure 3 , inputting the input text 310 into the animation interpreter 324 based on the large language model can obtain the animation instruction vector corresponding to the input text 310. Figure 4 A schematic diagram illustrating a method 400 for determining animation instruction vectors according to some embodiments of the present disclosure. Figure 3 The architecture of the animation interpreter 324 based on the large language model is shown as Figure 4 As shown in 420. Reference Figure 4 The large language model-based animation interpreter 420 adopts a sequence-to-sequence (seq2seq) architecture 424 with a sparse attention mechanism 422 and is also combined with a hard-aligned retrieval mechanism 426 in a pre-set animation instruction library, which enables a fine mapping of text to animation.

[0034] In some embodiments, the animation parameters at the current moment can be determined based on the animation parameters at the previous moment and the emotional characteristics at the previous moment. Specifically, as shown in formula (2):

[0035] A t =LLM(E t-1 ,A t-1 ) (2)

[0036] Assume there is an input text 410 "She smiled happily", in which A t Indicates the animation state parameters or animation instructions of the current time step (i.e. the moment of "smiling"), and indicates the animation parameters of her smiling face. t-1 Contains the emotional and contextual feature information of the description "she is happy", which is the feature vector of the previous time step. t-1is the animation state at the previous time step, such as the animation parameters of her calm expression or the start of a smile. The large language model will predict and generate the animation state A t at the current time step based on these input information. t In some embodiments, the form of A i is a sequence of numerical codes.

[0037] In some embodiments, assume there is an input text 410 "She smiled happily". Through the sparse attention mechanism 422 of the large language model, it can be identified that "happily" and "smiled" are the most important parts of this text, because they directly describe the emotion and action. Therefore, the sparse attention mechanism will mainly focus on these parts and assign them higher weights. The weight calculation formula (3) in the sparse attention mechanism is as follows:

[0038]

[0039] For the sentence "She smiled happily", the sparse attention mechanism will focus on the words most relevant to the smile. In this example, "happily" and "smiled" can be the selected words. i and h j represent the hidden states of the words "happily" and "smiled" respectively. Score(h i ,h j ) calculates the relevance between the two words, measuring their importance in emotional expression and action. Due to the characteristics of sparse attention, only the words in the selected word set Selected are focused on. Then, for these words, their attention weights a ij are calculated, which determine their relative importance in generating animations. Through this sparse attention mechanism, the computational load of the system can be reduced, thereby improving the efficiency of the system.

[0040] Based on these focused text parts, the large language model can generate corresponding animation instruction vectors. In this example, it may generate animation parameters related to the "happily" emotion (such as the smiling expression of the character's face) and animation parameters related to the "smiled" action (such as the movement trajectory of the character's head).

[0041] In some embodiments, to avoid errors in the animation instructions generated by the large language model, a hard alignment mechanism can be used to accurately match the key information in the text with pre-stored animation instructions. As shown in formula (4):

[0042] A seq = Seq2Seq(T input ,H align ) (4)

[0043] Assume there is an input text 410 "She jumped for joy", T input It is the sentence "She jumped for joy", H align It is the hard-aligned animation instruction corresponding to this text pattern, that is, the general animation instruction for the character to jump. The Seq2Seq model combines the input text T input and hard alignment information H align To generate animation instruction sequence A seq , A seq A at all times is included t Parameters. This animation instruction sequence will describe in detail the whole process from standing to jumping. In some embodiments, A seq Can exist in the form of a vector.

[0044] In some embodiments, the animation generation system 320 has pre-stored animation instructions corresponding to specific text patterns. For example, for the word "smile," there may be a pre-stored animation instruction that describes the entire process of a character going from a blank expression to a full smile. When the seq2seq model encounters the word "smile," it retrieves this pre-stored animation instruction and hard-codes it into the generated animation instruction.

[0045] This architecture leverages the power of large language models to parse nuances in text and accurately translate them into animation instructions. By combining a sparse attention mechanism with a seq2seq architecture with hard alignment, it can more accurately understand the text content and generate animation instruction vectors that closely match the text.

[0046] Return to Figure 3 The large language model-based animation interpreter 324 can generate an animation instruction vector corresponding to the input text 310. For example, if the input text 310 is "He jumped up with joy," the language model-based animation interpreter 324 can output an animation instruction vector indicating [expression: smile; body movement: jump off the ground]. This animation instruction vector is input into the avatar animation engine 326 to generate an animation sequence corresponding to the animation instruction vector. These animation sequences are frame-by-frame images and the transformation relationships between the images.

[0047] The following will be combined Figure 5 5. A flowchart for determining an animation sequence 500 according to some embodiments of the present disclosure is described below. Figure 5 FIG. 5 shows a flowchart for determining an animation sequence 500 according to some embodiments of the present disclosure. Figure 5The animation instruction vector 510 in vector form generated by the animation interpreter is input into a cycle generative adversarial network (cycleGANseq) 520 with a sequence discriminator to generate a series of key frame images 530. To ensure that the generated series of key frame images 530 are coherent and can dynamically change, the generated series of key frame images 530 can be interpolated 540, so that the avatar animation engine 326 can generate a dynamic and coherent animation sequence 550.

[0048] Alternatively, in the process of generating the animation sequence 550, the cycle generative adversarial network can be used to generate the key frame of each current time step. Specifically, as shown in equation (5):

[0049] M t =CycleGAN(A t ,M t-1 ) (5)

[0050] Where A t is the animation parameter of the current time step, and M t-1 is the animation motion key frame of the previous time step.

[0051] With reference to Figure 5 , in some embodiments, the cycle generative adversarial network 520 with a sequence discriminator does not evaluate each frame separately, but evaluates the entire animation sequence. This ensures that the generated animation sequence is not only reasonable on each frame, but also presents consistency and fluency on the entire sequence. In this way, the system can generate a coherent and natural animation sequence. Specifically, as shown in equation (6):

[0052] M seq =CycleGAN seq (A seq ) (6)

[0053] Where A seq is the input animation instruction set vector, which contains text or parameter information describing the action, expression, position change, etc. of the character. CycleGAN seq is a modified CycleGAN model that generates a corresponding animation sequence M seq according to the input animation instruction. This motion sequence M seq is a continuous action set that describes the key frame images of the motion trajectory and changes of the avatar animation character during the entire animation process. In this way, the system can generate an animation sequence that meets the requirements of the input instruction set.

[0054] In some embodiments, the parameters of the recurrent generative adversarial network with sequence discriminator 520 can be adjusted using the loss between the real series of keyframes and the generated series of keyframes.

[0055] In some embodiments, for an animation instruction vector indicating [expression: smile; body movement: jump with both feet off the ground], in order to ensure that the generated series of key frames 530 is a coherent animation sequence of smile and jump, an interpolation operation may be performed on the generated series of key frames 530. Specifically, the interpolation formula (7) is as follows:

[0056] M t =Interpolate(M t-1 ,M T+1 ) (7)

[0057] Among them, M t-1 is the previous key frame, M t is the inserted keyframe, M T+1 is the next key frame. For example, if an avatar animation character transitions from a standing state to a jumping state, interpolation 540 will calculate each intermediate posture of the avatar animation character between standing and jumping, thereby generating a continuous, smooth animation sequence. For example, for the instruction [expression: smile; body movement: jump with both feet off the ground], a coherent jumping and smiling head expression animation sequence can be generated through the recurrent generation network 520 with a sequence discriminator and interpolation 540. In this way, the avatar animation engine can generate more natural and smooth movements that match the input text description, thereby enhancing the realism and expressiveness of the animation.

[0058] Return to Figure 3 , through the animation interpreter 324 based on the large language model, an animation instruction vector corresponding to the input text 310 can be generated. For example, if the input text 310 is "He jumped up with joy", the animation interpreter 324 based on the language model can output an animation instruction vector indicating [expression: smile; body movement: jump with both feet off the ground]. This animation instruction vector is input to the facial expression generation adversarial network 328 to generate facial expression details corresponding to the animation instruction. Figure 6 6 is a schematic diagram for determining a facial blend shape 600 according to some embodiments of the present disclosure. Figure 6 FIG2 shows a schematic diagram for determining facial blend shapes 600 according to some embodiments of the present disclosure. Figure 6 , the animation instruction vector 610 is input into the facial expression generation adversarial network 620, and the facial blend shape 630 related to the instruction can be obtained. Specifically, as shown in formula (8):

[0059] F t =GANface (E t ) (8)

[0060] where F t denotes the facial blendshape parameters at time t. E t is the emotion feature vector. The facial expression generative adversarial network 328 is a generative adversarial network specifically designed for facial blendshape editing, which generates the corresponding blendshape parameters based on the input emotion feature vector. Blendshape is a technique in computer graphics used to represent facial expressions by combining different base shapes (i.e., blendshapes) to create complex facial expressions. The emotion feature vector contains the emotional information extracted from the input text, such as happiness, sadness, anger, etc.

[0061] In some embodiments, for the emotion feature vector indicating [expression: smile] in the animation instruction vector indicating [expression: smile; body action: jumping with both feet off the ground], a facial expression detail of slightly squinting eyes and upturned corners of the mouth can be generated. In this way, the system can generate facial expressions that match the text description, are realistic and natural, and improve the emotional expression ability and interactive experience of the virtual character.

[0062] Returning Figure 3 , after inputting the animation instruction vector generated by the large language model-based animation interpreter 324 into the animated avatar engine 326 and the facial expression generative adversarial network 328, a coherent and dynamically changing animation sequence and a head portrait with facial expression details can be obtained respectively. By fusing the two at the corresponding time, an animation 330 with realistic expression details and coherent consistency can be obtained. Specifically, as shown in equation (9):

[0063] Avatar animation = Integrate(M t , F t ) (9)

[0064] where Avatar animation denotes the final animated avatar character, M t is the body motion at time t, which contains the image of the body posture and action of the animated avatar character at that moment, and F t is the facial expression at the same time, which represents the emotional state and emotional expression of the animated avatar character at that time. This process involves the synchronization of time, rhythm, and expression between the two, ensuring that they can coordinate and consistently exhibit the behavior and emotions of the virtual character. By providing highly realistic and attractive animated avatar characters, this system can enhance the interactive experience of users and improve the immersion of virtual worlds, bringing more lively and expressive visual effects to various applications.

[0065] Figure 7 A schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. As shown, the device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with computer program instructions stored in a read-only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708. Various programs and data required for operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0066] Various components in the device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc., an output unit 707, such as various types of displays, speakers, etc., a storage unit 708, such as a magnetic disk, an optical disk, etc., and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0067] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the method 300. For example, in some embodiments, the method 300 can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method 300 described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the method 300 by any other appropriate means, such as by means of firmware.

[0068] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, non- transitory machine-readable media can include ROM, RAM, programmable ROM (e.g., EEPROM), electrically erasable programmable ROM (EEPROM), flash memory, or any other storage technology, peripheral device, special-purpose logic circuitry (e.g., an application specific integrated circuit), or the like. The aforementioned components (e.g., a module) can be configured to operate together to provide for a process that performs (at least in part) the functionality described herein.

[0069] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program codes, when executed by the processor or controller, enable the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can execute entirely on a machine, partly on a machine, partly on a remote machine or entirely on a remote machine or server.

[0070] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include one or more lines of a program of electrical connection with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Further, while operations are depicted in a particular, sequential order, this should not be understood as requiring, or

[0071] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for generating an avatar animation, comprising: generating, based on a text input, an animation instruction vector for an avatar animation; determining, based on the animation instruction vector, an animation sequence of the avatar animation, the animation sequence indicating a plurality of frames of the avatar animation and transformations between the plurality of frames; determining, based on the animation instruction vector, a blendshape of the avatar animation, the blendshape indicating a facial expression of the avatar animation; and generating, based on the animation sequence and the blendshape, an avatar animation corresponding to the text.

2. The method of claim 1, wherein generating, based on a text input, an animation instruction vector for an avatar animation comprises: determining, based on a sentiment feature, a context feature, and an animation instruction of a first time instance of the text input, an animation instruction of a second time instance.

3. The method of claim 2, wherein determining, based on a sentiment feature, a context feature, and an animation instruction of a first time instance of the text input, an animation instruction of a second time instance comprises: determining, by a large language model with a sparse attention mechanism, based on the text input, the sentiment feature and the context feature of the text input; and determining, based on the sentiment feature and the context feature, an animation instruction.

4. The method of claim 3, wherein determining, based on the sentiment feature and the context feature, an animation instruction comprises: determining, based on a predefined animation instruction library, the animation instruction corresponding to the text input by a hard alignment mechanism.

5. The method of claim 4, further comprising: generating, based on a sequence-to-sequence of the large language model, an animation instruction vector of the animation instruction, the animation instruction in the animation instruction vector being arranged in a time sequence.

6. The method of claim 3, wherein determining, based on the animation instruction vector, an animation sequence of the avatar animation comprises: generating, by a cycle generative adversarial network with a sequence discriminator, based on the animation instruction vector, a series of keyframes; and interpolating between a first keyframe and a second keyframe in the series of keyframes to generate the animation sequence.

7. The method of claim 6, further comprising: adjusting the cycle generative adversarial network with a sequence discriminator based on a loss between real series of keyframes and the generated series of keyframes.

8. The method of claim 6, further comprising: determining, based on an animation sequence of a third time instance and an animation instruction of a fourth time instance, an animation sequence of a third time instance.

9. The method of claim 1, wherein determining, based on the animation instruction vector, a blendshape of the avatar animation comprises: determining, by the facial expression generative adversarial network, based on a sentiment feature of a fifth time instance, a blendshape of a fifth time instance.

10. The method of claim 9, wherein generating, based on the animation sequence and the blendshape, an avatar animation corresponding to the text input comprises: generating, based on an animation sequence of a sixth time instance and a blendshape of a sixth time instance, an avatar animation of a sixth time instance.

11. An electronic device, comprising: ​ ​ ​ at least one processor; and coupled to the at least one processor and having instructions stored thereon that, when executed by the at least one processor, cause the electronic device to perform actions comprising: generating, based on a text input, an animation instruction vector for an avatar animation; determining, based on the animation instruction vector, an animation sequence of the avatar animation, the animation sequence indicating a plurality of frames of the avatar animation and transformations between the plurality of frames; determining, based on the animation instruction vector, a blendshape of the avatar animation, the blendshape indicating a facial expression of the avatar animation; and generating, based on the animation sequence and the blendshape, an avatar animation corresponding to the text.

12. The device of claim 11, wherein generating, based on a text input, an animation instruction vector for an avatar animation comprises: determining, based on a sentiment feature, a context feature of a first time instance of the text input, and an animation instruction of the first time instance, an animation instruction of a second time instance.

13. The device of claim 12, wherein determining, based on a sentiment feature, a context feature of a first time instance of the text input, and an animation instruction of the first time instance, an animation instruction of a second time instance comprises: determining, based on the text input, the sentiment feature and the context feature of the text input by a large language model with a sparse attention mechanism; and determining, based on the sentiment feature and the context feature, an animation instruction.

14. The device of claim 13, wherein determining, based on the sentiment feature and the context feature, an animation instruction comprises: determining, based on the text input, the animation instruction corresponding to the text input from a predefined animation instruction library by a hard alignment mechanism.

15. The device of claim 13, further comprising: generating, based on a sequence-to-sequence of the large language model, an animation instruction vector of the animation instruction, the animation instruction in the animation instruction vector being arranged in a time sequence.

16. The device of claim 13, wherein determining, based on the animation instruction vector, an animation sequence of the avatar animation comprises: generating, based on the animation instruction vector, a series of keyframes by a cycle generative adversarial network with a sequence discriminator; and interpolating between a first keyframe and a second keyframe in the series of keyframes to generate the animation sequence.

17. The device of claim 16, further comprising: adjusting the cycle generative adversarial network with the sequence discriminator based on a loss between real series of keyframes and the generated series of keyframes.

18. The device of claim 16, further comprising: determining, based on an animation sequence of a third time instance and an animation instruction of a fourth time instance, an animation sequence of the third time instance.

19. The device of claim 11, wherein determining, based on the animation instruction vector, a blendshape of the avatar animation comprises: determining, based on a sentiment feature of a fifth time instance, a blendshape of the fifth time instance by the facial expression generative adversarial network. ​ ​ 20. A computer program product, the computer program product being tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions that, when executed, cause a machine to: generate, based on a text input, animation instruction vectors for an avatar animation; determine, based on the animation instruction vectors, an animation sequence for the avatar animation, the animation sequence indicating a plurality of frames of the avatar animation and transformations between the plurality of frames; determine, based on the animation instruction vectors, a blendshape of a face of the avatar animation, the blendshape indicating a facial expression of the avatar animation; and generate, based on the animation sequence and the blendshape, an avatar animation corresponding to the text. ​

Images