Facial animation control that automatically generates facial motion units using text and speech.

By using a machine learning engine to process modal data and generate facial motion units, the problem of low efficiency in manually creating facial motion units is solved, and the automation and realism of avatar facial animation are improved.

CN116508100BActive Publication Date: 2026-11-17SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202180070558.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-08-07
Filing Date
2021-09-02
Publication Date
2026-11-17
Estimated Expiration
2041-09-02

AI Technical Summary

Technical Problem

In existing technologies, the computerized avatar facial animation process requires the manual creation of facial action units (FAUs), resulting in inefficiency and difficulty in achieving realistic facial animation effects.

Method used

The machine learning (ML) engine identifies and processes first-modal and second-modal data to generate facial action units (FAUs), and animates' faces based on emotional action information and time-aligned word-level emotional probabilities.

Benefits of technology

It automates the facial animation process, improving efficiency and enhancing the realism and naturalness of facial animation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116508100B_ABST
    Figure CN116508100B_ABST
Patent Text Reader

Abstract

The computer-generated text and speech are processed by a machine learning engine to animate the computer avatar’s face (202) (306).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application generally relates to facial animation control of automatically generated facial action units (FAUs) using text and speech in computer simulations and other applications. Background Technology

[0002] For example, the faces of computer-generated images in computer games are animated during simulated playback to provide a realistic effect. As understood in this article, artists must generally carefully create Facial Animation Units (FAUs), which are used to animate corresponding parts of the face on a frame-by-frame basis by evaluating the contraction or relaxation of corresponding muscle points on the human face. Summary of the Invention

[0003] An apparatus includes: at least one processor configured with instructions for recognizing an image of the face of a computerized avatar. The instructions are executable to input first modality data associated with the avatar into a machine learning (ML) engine, input second modality data associated with the avatar into the ML engine, and receive from the ML engine output useful for animaterizing the image of the avatar's face. The instructions are executable to animate the avatar's face based on the output.

[0004] In some implementations, the output includes facial action units (FAUs), where each FAU relates to a corresponding portion of the image of the face. In a non-limiting example, the first modal data includes text and the second modal data includes speech.

[0005] In a non-limiting example, the instructions may be executable to derive sentiment action information from the first modality data and the second modality data using the ML engine. The output may be based at least in part on time-aligned word-level sentiment probabilities generated based on the sentiment action information.

[0006] In another approach, one method includes training a machine learning (ML) model using a training set of animated faces speaking known words. The method further includes generating an image of a first face to be animated to speak words based on a first text. The method also includes inputting the first text into the ML model and animateing the image of the first face to speak the first word indicated by the first text based on the output of the ML model.

[0007] In an exemplary implementation, the method may include: detecting sentiment and emotion from the first text; aligning the first text with speech representing the first text to render aligned text / speech; and inputting the sentiment, the emotion, and the aligned text / speech into the ML model. The method may further include: inputting a target sentiment into the ML module. An exemplary implementation of the method may include: receiving a first probability representing facial actions from the ML modality; receiving a second probability representing an emotion from the ML model; and using the first probability and the second probability to construct a Facial Action Unit (FAU). The method may then include: animate the image of the first face according to the FAU.

[0008] In another aspect, a component includes: at least one display configured to present an animated computer avatar; and at least one processor configured with instructions for executing a machine learning (ML) model. The instructions are capable of: receiving text instructing the avatar to speak; processing the text using the ML model to generate facial action units (FAUs); and animate the computer avatar based on the FAUs.

[0009] The details of both the structure and operation of this application can be best understood with reference to the accompanying drawings, in which the same reference numerals refer to the same parts, and in the drawings: Attached Figure Description

[0010] Figure 1 It is a block diagram of an exemplary system including examples based on this principle;

[0011] Figure 2 It is a screenshot of the face of an exemplary computerized embodiment;

[0012] Figure 3 The overall exemplary logic is illustrated in an exemplary flowchart format;

[0013] Figure 4 This demonstrates the training process of a machine learning (ML) engine; and

[0014] Figure 5 This illustrates the inference process of the ML engine. Detailed Implementation

[0015] This disclosure generally relates to a computer ecosystem, which includes various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. Systems described herein may include server components and client components that can be network-connected, enabling data exchange between the client components and the server components. Client components may include one or more computing devices, including game consoles (such as Sony...). This includes game consoles made by Microsoft, Nintendo, or other manufacturers; virtual reality (VR) headsets; augmented reality (AR) headsets; portable televisions (such as smart TVs and internet-enabled TVs); portable computers (such as laptops and tablets); and other mobile devices (including smartphones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some client computers may use operating systems such as Linux, Microsoft operating systems, or Unix operating systems, or operating systems made by Apple, Inc. or Google. These operating environments can be used to execute one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. Furthermore, the operating environment according to the principles of the present invention can be used to execute one or more computer game programs.

[0016] The server and / or gateway may include one or more processors that execute instructions to configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be a game console (such as Sony). Instantiation of personal computers, etc.

[0017] Information can be exchanged between clients and servers over a network. For this purpose, and for security reasons, servers and / or clients may include firewalls, load balancers, temporary storage devices, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements methods for providing secure communities (such as online social networking sites) to network members.

[0018] The processor can be a single-chip or multi-chip processor, which can perform logic by means of various lines (such as address lines, data lines and control lines) as well as registers and shift registers.

[0019] Components included in one embodiment can be used in any suitable combination in other embodiments. For example, any of the various components described herein and / or depicted in the figures can be combined, interchanged, or excluded from other embodiments.

[0020] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes: a system having only A; a system having only B; a system having only C; a system having both A and B; a system having both A and C; a system having both B and C; and / or a system having both A, B and C, etc.

[0021] Now for specific reference Figure 1 An exemplary system 10 is illustrated, which may include one or more of the exemplary devices mentioned above and further described below according to the principles of the invention. The first device among the exemplary devices included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, such as, but not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling a TV). Alternatively, the AVD 12 may also be a computerized internet-enabled (“smart”) phone, tablet computer, laptop computer, HMD, wearable computerized device, computerized internet-enabled music player, computerized internet-enabled headset, computerized internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the invention (e.g., communicate with other CE devices implementing the principles of the invention, perform the logic described herein, and perform any other functions and / or operations described herein).

[0022] Therefore, in order to implement this principle, AVD 12 can be... Figure 1Some or all of the components shown are constructed. For example, AVD 12 may include one or more displays 14, which may be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen and may be touch-enabled for receiving user input signals via touch on the display. AVD 12 may include one or more speakers 16 for outputting audio according to the principles of the invention, and includes at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to control AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. A graphics processor 24A may also be included. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12, and other elements described herein including AVD 12, to implement the principles of the invention, such as controlling display 14 to display images thereon and to receive input from said display. Furthermore, it should be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above.

[0023] In addition to the foregoing, the AVD 12 may also include one or more input ports 26, such as a High Definition Multimedia Interface (HDMI) port or a USB port that physically connects to another CE device and / or a headphone port that connects headphones to the AVD 12 to present audio from the AVD 12 to the user. For example, input port 26 may be wired or wirelessly connected to a wired or satellite source 26a of audio-visual content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disk player containing content. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 44.

[0024] AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices. In some cases, the one or more computer memories are embodied as stand-alone devices within the chassis of the AVD, or as personal video recording devices (PVRs) or video disk players for playing back AV programs, either inside or outside the chassis of the AVD, or as removable memory media. Furthermore, in some embodiments, AVD 12 may include a location or positioning receiver, such as, but not limited to, a cellular phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular phone base station and provide said information to processor 24 and / or in conjunction with processor 24 to determine the altitude at which the AVD 12 is set. Component 30 may also be implemented by an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions.

[0025] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras (such as webcams), and / or cameras integrated into AVD 12 and capable of being controlled by processor 24 to acquire pictures / images and / or videos according to the principles of the present invention. AVD 12 may also include a Bluetooth transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0026] In addition, AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, odometers, or magnetometers, infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to processor 24. AVD 12 may include an over-the-air (OTA) TV broadcast port 38 for receiving OTA TV broadcasts that provide input to processor 24. In addition to the foregoing, it should be noted that AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power AVD 12, such as a kinetic energy harvester that can convert kinetic energy into electrical energy to charge the battery and / or power AVD 12.

[0027] Still referencing Figure 1In addition to AVD 12, System 10 may also include one or more other CE device types. In one example, the first CE device 44 may be a computer game console that can be used to send audio and video of a computer game to AVD 12 via commands sent directly to AVD 12 and / or via a server, while the second CE device 46 may include components similar to the first CE device 44. In the example shown, the second CE device 46 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player 47. In the example shown, only two CE devices 44, 46 are shown; it should be understood that fewer or more devices may be used. The devices described herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may be combined with some or all of the components shown in the case of AVD 12.

[0028] Referring now to the aforementioned at least one server 50, said at least one server includes at least one server processor 52, at least one tangible computer-readable storage medium 54 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 56, said at least one network interface, under the control of the server processor 52, allowing communication via network 22 with... Figure 1 It communicates with other devices and can actually facilitate communication between the server and client devices according to the principles of the invention. It should be noted that network interface 56 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as, for example, a wireless telephone transceiver).

[0029] Therefore, in some implementations, server 50 may be an internet server or an entire server "farm," and in exemplary implementations for, for example, online gaming applications, the server may include and perform "cloud" functionality, enabling devices of system 10 to access a "cloud" environment via server 50. Alternatively, server 50 may be a combination of one or more game consoles or... Figure 1 The other devices shown are implemented in the same room or on other computers nearby.

[0030] Figure 2 A display 200 is shown, such as any display on which a computer-generated face 202 is presented herein. The face 202 has a plurality of evaluation anchors 204 (only four are shown for clarity), which can be animated according to the requirements of a corresponding Facial Animation Unit (FAU) to evaluate the relaxation or contraction of corresponding parts of a human face.

[0031] Figure 3This document illustrates the overall logic that can be adopted to automatically generate FAUs to animate Face 202. The logic described herein can be executed by any one or more of the processors described herein.

[0032] Starting from frame 300, train the machine learning (ML) engine, as shown in the reference. Figure 4 Described. Proceeding to box 302, a video image 202 of the face is generated via, for example, computer simulation (such as a computer game), and at box 304, corresponding multimodal data associated with the face is input into the ML engine. Based on the multimodal data, the ML engine generates a predicted Facial Active User Assessment (FAU), and at box 306, the predicted FAU is used to animate the evaluated muscles of the avatar face 202, as described in the reference. Figure 5 A more complete description.

[0033] Figure 4 This illustrates an exemplary training process for the ML engine. Access a training dataset from a database of 400. This training dataset includes images of faces expressing emotions while speaking. In other words, the database contains ground truth animations of avatars speaking known text when the correct facial expressions are animated.

[0034] At box 402, for each input training image, the corresponding anchor point 204 is extracted from the image. In other words, box 402 generates anchor points for the input image to be used for the Facial Action Unit (FAU). Furthermore, at box 404, speech signals and associated text are extracted. It should be understood that the speech and text at box 404 are associated with the FAU extracted at box 402.

[0035] Box 406 indicates that the speech and text are temporarily aligned with each other, as further disclosed below. Figure 4 As shown in graph 407, phoneme boundaries 407A are extracted from the speech signal and phoneme boundaries 407B of the corresponding text are extracted. In this way, each incremental block along the x-axis represents a phoneme generated by speech, wherein the phoneme is confined between adjacent boundaries 407C and 407D of the incremental block.

[0036] Forced alignment box 406 uses phoneme boundaries to temporally align the emotion in emotion annotation box 408, the emotion in emotion annotation box 410 in which experts annotate the emotion input, and the speech features in speech feature box 412. Therefore, the corresponding emotion, emotion, and speech features are aligned using extracted phonemes. As will be discussed further below, "emotion" can refer to the degree of feeling, such as positive or negative, and "feeling" can refer to the speaker's emotional state, such as happy, sad, angry, etc. Note that the sliding window of words in speech can be handled in the following way: for example, words one to three can be annotated with the first emotion / feeling, words two to four can be annotated with the second emotion / feeling, and so on. Speech features 412 are generated to represent the combined characteristics of the input. Speech features can be generated using techniques including the Wigner-Ville distribution function and time-frequency filtering.

[0037] The results from boxes 408, 410, and 412 are provided to the ML emotion-action generation model 414 to train the model. Furthermore, the input to model 414 is the FAU from box 402 and the target emotion 416, which is the ground truth of the input (annotated) emotion used to train model 414. In this way, model 414 is trained on multiple sets of data from database 400 to associate the FAU with corresponding speech and text and the target emotion, so as to be able to output facial action anchor probabilities 418 and time-aligned word-level emotion probabilities 420 for subsequent animation of the face (e.g., by appropriately animateting the anchor 204 according to the FAU). Figure 2 It is used in the animation of the face (202).

[0038] Continuing to move Figure 5 Previously, one or both of categorical sentiment classification and dimensional sentiment classification could be used to detect sentiment in the input training set for comparison with the ground truth 416. Categorical classification produces sentiment categories including happiness, sadness, anger, fear, etc. Dimensional classification produces sentiment dimensions in terms of valence and arousal.

[0039] Figure 5 The subsequent inference processing of model 414 is shown. Figure 5 The text 500 spoken by the animated avatar is input into the text-speech box 502 to convert the text into a speech signal. Alternatively, recorded or live human speech corresponding to the text can be supplied to box 502. Speech features are extracted at box 504, and at box 506, features are extracted for… Figure 4 The training process describes the extraction of phoneme boundaries from text and corresponding speech in the same manner, as indicated by graph 507, which has the same characteristics as... Figure 4 The curve in Figure 407 is the same as that used for time alignment as referenced above. Figure 4 Instruct on the characteristics of the elements in the corresponding aligned boxes.

[0040] The avatar from box 500 will send the spoken text to the emotion and sentiment detection box 508 for analysis. Figure 4 The training extracts sentiment and emotion from the text. At box 508, the sentiment and emotion extracted from the text, along with speech features from box 504 and aligned text / speech signals from box 506, are input into model 510. If needed, a target sentiment can be input into the model, as indicated at box 512. The target sentiment can be a portion of annotated text derived from the input text using machine learning.

[0041] Based on the training of model 510, and according to the input it receives, the model outputs a probability 512 for each anchor point 204 of facial movements and a probability 516 of emotion derived from time-aligned text / speech signals. Based on one or more of these probabilities, an estimated anchor position 518 (i.e., FAU) is obtained to establish the facial expression to be portrayed for each animated frame. The FAU is applied to the avatar's animation to give the avatar the correct expression when the avatar speaks the text input at box 500.

[0042] It should be understood that although the principles of the invention have been described with reference to some exemplary embodiments, these embodiments are not intended to be limiting, and various alternative arrangements may be used to achieve the subject matter claimed herein.

Claims

1. A device for facial animation control, comprising: At least one processor, said at least one processor being configured with instructions for: Image recognition of the face of a computerized avatar; Identify first modal data associated with the avatar, the first modal data including text; Identify second modal data associated with the avatar, the second modal data including speech; Receive instructions for text to be spoken by the avatar and speech to be spoken by the avatar, the text being at least partially based on the first modal data and the speech being at least partially based on the second modal data; The instruction is at least partially used to align the speech and text temporally with each other using phoneme boundaries extracted from the speech and text, and the instruction is also capable of being executed to align emotion and emotional intensity temporally with the speech using the phoneme boundaries extracted from the speech and text. Input the text and the speech into a machine learning (ML) model; The ML model is capable of receiving anchors derived from an image of the avatar's face to generate a Facial Action Unit (FAU) and associating the speech and text with the FAU. Receive from the ML model the output that animates the image of the avatar's face; as well as The avatar's face is animated based on the output.

2. The device of claim 1, wherein the instructions are executable to derive emotional action information from the first modal data and the second modal data.

3. The device of claim 2, wherein the output is at least in part based on time-aligned word-level sentiment probabilities generated according to the sentiment action information.

4. A method for controlling facial animation, comprising: Image recognition of the face of a computerized avatar; Identify first modal data associated with the avatar, the first modal data including text; Identify second modal data associated with the avatar, the second modal data including speech; Receive instructions for text to be spoken by the avatar and speech to be spoken by the avatar, the text being at least partially based on the first modal data and the speech being at least partially based on the second modal data; At least in part, the speech and the text are aligned temporally with each other using phoneme boundaries extracted from the speech and the text, and the emotion and emotion level are aligned temporally with the speech using the phoneme boundaries extracted from the speech and the text. Input the text and the speech into a machine learning (ML) model; The ML model is capable of receiving anchors derived from an image of the avatar's face to generate a Facial Action Unit (FAU) and associating the speech and text with the FAU. Receive from the ML model the output that animates the image of the avatar's face; as well as The avatar's face is animated based on the output.

5. A component for facial animation control, comprising: At least one display, the at least one display being configured to present an animated computer avatar; At least one processor, the at least one processor being configured with instructions for executing machine learning (ML) models, the instructions being capable of performing: Receive the text instructing the avatar to speak the following words; Receive the words that the avatar is about to speak; The instructions at least partially use phoneme boundaries extracted from the speech and the text to align the text and the speech in time, and the instructions are also capable of being executed to align emotion and emotional intensity with the speech in time using the phoneme boundaries extracted from the speech and the text. The text is processed using the ML model to generate Facial Action Units (FAUs); and The computer avatar is animated based on the FAU.

6. The component of claim 5, wherein the instructions are capable of performing: Detecting sentiment and sentiment level from the text; Align the text with the speech representing the text to render the aligned text / speech; and Input the emotion, the emotion level, and the aligned text / speech into the ML model.

7. The component of claim 6, wherein the instructions are capable of performing: Input the target sentiment into the ML model.

8. The component of claim 5, wherein the instructions are capable of performing: Receive the first probability representing facial movements from the ML model.

9. The component of claim 8, wherein the instructions are capable of performing: Receive a second probability representing emotion from the ML model.

10. The component of claim 9, wherein the instructions are capable of performing: The FAU is established using the first probability and the second probability.

Citation Information

Patent Citations

  • Automatically generating motions of an avatar

    WO2020010530A1