Creating images, grids, and speech animations from mouth shape data

Abstract

Creating images and animations of lip motion from mouth shape data includes providing vectors of a plurality of coefficients as one or more input features to a neural network model. Each vector of the plurality of coefficients corresponds to a different mouth shape. Using the neural network model, a data structure output is generated specifying a visual representation of a mouth, the mouth including a lip having a shape corresponding to the vector.

Description

Technical Field

[0001] The present disclosure relates to creating images, meshes, and / or animations that include accurate lip movements. Background Art

[0002] The ability to generate high-quality images of characters (whether digital humans, avatars, and other anthropomorphic objects) is becoming increasingly important for a variety of reasons, including the growing popularity of the metaverse, the adoption of virtual experiences across different parts of society, and recent advances in hardware and other technologies (such as neural networks that facilitate rapid virtualization). For example, in the case of a digital human, its rendering is intended to closely (if not perfectly) resemble a real person. The quality of the textures used and the resolution of the resulting image play an important role in achieving this goal. As an example, preferably, the body parts of the digital human are rendered with the correct textures, so that the hair looks like it is composed of individual strands; the shirt looks like it is made of fabric; the skin has pores; etc. Even for other use cases such as the generation of avatars, whether humanoid characters, humanoid characters, or other anthropomorphic objects, the accurate generation of certain features such as the mouth and / or lips is important because actual human perception of these areas is very sensitive to any perceptible visual artifacts (e.g., visual irregularities and / or misalignment relative to audio such as speech).

[0003] However, available techniques are generally unable to produce accurate and realistic lip movements. So-called "deep fakes" are illustrative of the current state of the art. Even some of the most realistic deep fakes available exhibit inaccurate lip movements. This is particularly true for lip movements corresponding to consonants. Some three-dimensional (3D) models have been created with more accurate lip geometry. However, renderings of 3D models often suffer from a synthetic appearance that lacks the degree of realism achieved with deep fakes. Therefore, creating accurate lip movements remains a technical challenge.

[0004] In many cases, characters such as digital humans, avatars, etc. are presented on large screens, which tend to highlight any visual irregularities. For example, a digital human may be displayed on a screen of a kiosk or other system, where the screen is sized so that the digital human appears life-size when rendered (e.g., having an average height and dimensions corresponding to a real person). These displays may be located in a variety of locations, including, but not limited to, airport lobbies, hotels, public spaces, etc. For all of these various use cases and others, it is important to render the avatar with high-quality features, including lip movements. Summary of the invention

[0005] Technical Solutions

[0006] In one or more embodiments, a computer-implemented method includes providing a vector of multiple coefficients as one or more input features of a neural network model. There may be one or more vectors of the multiple coefficients. Each vector of the multiple coefficients corresponds to a different mouth shape. The method includes using the neural network model to generate a data structure output specifying a visual representation of a mouth, the mouth including lips having a shape corresponding to the vector.

[0007] In one aspect, the plurality of coefficients of each vector are viseme coefficients. The method may include generating the viseme coefficients from audio data of the speech. The method may include encoding the viseme coefficients into the vector provided to the neural network model.

[0008] In another aspect, a data structure output is generated directly from a vector of multiple coefficients without using any facial landmark data.

[0009] In another aspect, the data structure output is a red-green-blue (RGB) two-dimensional (2D) image.

[0010] In another aspect, the data structure output is a grid.

[0011] In another aspect, generating a data structure output includes generating a plurality of images using the neural network model. Each image is generated from a different vector having a plurality of coefficients. The plurality of images collectively represent an animation of lip movement.

[0012] In another aspect, the method includes providing a first mesh of a face to a neural network model using a vector of a plurality of coefficients. The data structure output is a second mesh generated by the neural network model based on the vector and the first mesh of the face. The second mesh is a version of the first mesh of the face that specifies a mouth that includes lips having a shape corresponding to the vector.

[0013] In one or more embodiments, a system includes one or more processors configured to initiate operations. The operations include providing a vector of multiple coefficients as one or more input features to a neural network model. Each vector of the multiple coefficients corresponds to a different mouth shape. The operations include generating a data structure output specifying a visual representation of a mouth using the neural network model, the mouth including lips having a shape corresponding to the vector.

[0014] In one aspect, the plurality of coefficients of each vector are viseme coefficients. The operations may include generating the viseme coefficients from audio data of the speech. The operations may include encoding the viseme coefficients into the vector provided to the neural network model.

[0015] In another aspect, a data structure output is generated directly from a vector of a plurality of coefficients without using any facial landmark data.

[0016] In another aspect, the data structure output is an RGB 2D image.

[0017] In another aspect, the data structure output is a grid.

[0018] In another aspect, generating a data structure output includes generating a plurality of images using the neural network model. Each image is generated from a different vector having a plurality of coefficients. The plurality of images collectively represent an animation of lip movement.

[0019] In another aspect, operations include providing a first mesh of a face to a neural network model using a vector of a plurality of coefficients. A data structure output is a second mesh generated by the neural network model based on the vector and the first mesh of the face. The second mesh is a version of the first mesh of the face that specifies a mouth including lips having a shape corresponding to the vector.

[0020] In one or more embodiments, a computer program product includes one or more computer-readable storage media having program code stored thereon. The program code is executable by one or more processors to perform operations. The operations include providing a vector of multiple coefficients as one or more input features to a neural network model. Each vector of the multiple coefficients corresponds to a different mouth shape. The operations include generating a data structure output specifying a visual representation of a mouth using the neural network model, the mouth including lips having a shape corresponding to the vector.

[0021] In one aspect, the plurality of coefficients of each vector are viseme coefficients. The operations may include generating the viseme coefficients from audio data of the speech. The operations may include encoding the viseme coefficients into the vector provided to the neural network model.

[0022] In another aspect, a data structure output is generated directly from a vector of a plurality of coefficients without using any facial landmark data.

[0023] In another aspect, the data structure output is an RGB 2D image.

[0024] In another aspect, the data structure output is a grid.

[0025] In another aspect, generating a data structure output includes generating a plurality of images using the neural network model. Each image is generated from a different vector having a plurality of coefficients. The plurality of images collectively represent an animation of lip movement.

[0026] In another aspect, operations include providing a first mesh of a face to a neural network model using a vector of a plurality of coefficients. A data structure output is a second mesh generated by the neural network model based on the vector and the first mesh of the face. The second mesh is a version of the first mesh of the face that specifies a mouth including lips having a shape corresponding to the vector.

[0027] This summary is provided only to introduce certain concepts, and does not identify any key or essential features of the claimed subject matter. Many other features and embodiments of the invention will be apparent from the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings illustrate one or more embodiments; however, the drawings should not be considered to limit the invention to only the embodiments shown. Various aspects and advantages will become apparent by reading the following detailed description and referring to the drawings.

[0029] Figure 1 An example of an architecture that may be executed by a data processing system to perform image generation is shown.

[0030] 2A , 2B and 2C show examples of facial feature point data.

[0031] Figure 3 shows that it can be executed by Figure 1 An example method for performing a system of the architecture.

[0032] Figure 4 Another example of an architecture adapted to operate with a grid is shown.

[0033] Figure 5 shows that it can be executed by Figure 4 An example method for performing a system with an architecture.

[0034] Figure 6 Another example of an architecture that can adapt a mesh to a specific mouth shape is shown.

[0035] Figure 7 shows that it can be executed by Figure 6 An example method for performing a system of the architecture.

[0036] Figure 8 Another example of an architecture that can adapt a mesh to a specific mouth shape is shown.

[0037] Fig. 9 shows that it can be executed by Figure 8 An example method for performing a system of the architecture.

[0038] Fig.10 An example implementation of a data processing system capable of executing the architecture described within this disclosure is shown. DETAILED DESCRIPTION

[0039] Although the present disclosure ends with claims defining novel features, it is believed that the various features described herein will be better understood by considering the description in conjunction with the drawings. The process(es), machine(s), manufacture(s), and any variations thereof described within the present disclosure are provided for illustrative purposes. Any specific structural and functional details described should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to employ the features described in nearly any appropriately detailed structure in a variety of ways. Furthermore, the terms and phrases used within the present disclosure are not intended to be limiting, but rather to provide an understandable description of the features described.

[0040] The present disclosure relates to creating images, meshes and / or animations (e.g., speech animations) that include accurate lip movements. According to the inventive arrangements disclosed herein, methods, systems and computer program products that are capable of generating visual representations of a mouth region that includes lips are disclosed. The lips can be generated, drawn or rendered with a specific lip geometry (e.g., shape). Images are generated using a generative neural network model configured or trained on mouth shape data. Given an input vector of specified mouth shape data, the generative neural network model is capable of generating a visual representation, such as an image or mesh that includes a region with highly accurate lips. The lips have a highly realistic shape or geometry. The generated visual representation can be used to create an animation of lip movement to simulate a character talking or speaking. As used within the present disclosure, the term "animation" means a series of images (e.g., frames) or meshes that can be played in sequence as a video to show movement. Because the generated visual representation is of high quality, the animation generated from such a visual representation depicts highly accurate and realistic lip movements.

[0041] Other aspects of the arrangement of the present invention are described in more detail below with reference to the accompanying drawings. For the purpose of simplicity and clarity of explanation, the elements shown in the figures are not necessarily drawn to scale. For example, for the sake of clarity, the size of some elements may be enlarged relative to other elements. In addition, where it is considered appropriate, reference numerals are repeated in the accompanying drawings to indicate corresponding, similar or similar features.

[0042] Figure 1 An example of architecture 100 is shown. Architecture 100 is capable of generating visual representations such as images. Architecture 100 can be implemented as a software framework that can be executed by a data processing system. Fig.10 Examples of data processing systems suitable for executing the architecture 100 as described herein and / or suitable for training a neural network model included in the architecture 100 are described.

[0043] In this example, the architecture 100 includes a mouth shape extractor 102 and a neural network model 104. The mouth shape extractor 102 and the neural network model 104 are arranged in series as a pipeline. Typically, the mouth shape extractor 102 receives audio data 110 as input. The audio data 110 may contain speech. The mouth shape extractor 102 extracts mouth shape data encoded as mouth shape coefficients 112 from the audio data 110. In this example, the neural network model 104 is conditioned on the output (e.g., mouth shape coefficients 112) from the mouth shape extractor 102. That is, the mouth shape coefficients 112, which are outputs from the mouth shape extractor 102, are fed or provided to the neural network model 104 as input.

[0044] In one or more example embodiments, throughout this disclosure, mouth shape coefficients 112 may be specified in terms of visemes. The term "viseme" is derived from the phrase "visual phoneme" and is a geometric representation of a phoneme. A viseme specifies the shape of the mouth at the vertex of a given phoneme. Each phoneme is associated with or generated by a viseme. Each viseme may represent one or more phonemes. In this regard, there is a many-to-one mapping of phonemes to visemes. Visemes are typically generated as an artistic rendering of the shape of the mouth (e.g., lips) when a particular phoneme is spoken. Visemes convey 3D data of the shape of the mouth when generating the phoneme mapped to it. Visemes are also largely speaker (e.g., voice) independent.

[0045] Example viseme extraction techniques that can be used to implement the mouth shape extractor 102 are described in Edwards et al., "JALI: An Animator-Centric Viseme Model for Expressive Lip Synchronization," ACM SIGGRAPH'16 Technical Paper, July 24-28, 2016, Anaheim, CA, which is incorporated herein by reference in its entirety. It should be understood that the mouth shape extractor 102 can be implemented using any available or known viseme extraction technique.

[0046] In one or more examples, the mouth shape extractor 102 can encode the mouth shape data into a plurality of mouth shape coefficients 112. The mouth shape coefficients 112 can be encoded or represented as a vector. As an illustrative example, there may be dozens of different visemes for a given language. The mouth shape coefficients 112 as output from the mouth shape extractor 102 can be encoded as a vector that includes a coefficient for each different mouth shape or viseme used. The coefficients for each viseme represent the contribution of that particular viseme to the shape / geometry of the mouth when a particular segment or portion of sound (e.g., a portion of speech) is produced from the audio data 110. Therefore, different segments of the audio data 110 can be specified as vectors of viseme coefficients. The vector can be a 1-dimensional vector of multiple coefficients.

[0047] In one or more example embodiments, the mouth shape extractor 102 can encode the mouth shape data into one or more vectors of multiple coefficients (eg, each vector having multiple coefficients).

[0048] In one or more example embodiments, the neural network model 104 may be implemented as an image-to-image conversion network. The image-to-image conversion network is a generative neural network model. Examples of generative neural network models may include, but are not limited to, generative adversarial networks (GANs) variational autoencoders (VAEs).

[0049] Typically, a GAN includes two neural networks, referred to as a generator and a discriminator. The generator and the discriminator engage in a zero-sum game with each other. Given a training set, the GAN is able to learn to generate new data with the same statistics as the training set. As an illustrative example, a GAN trained on an image or image library is able to generate different images that look real to a human observer. In a GAN, the generator generates images. The discriminator determines a measure of the realism of the images generated by the generator. Since the two neural networks can be dynamically updated during operation (e.g., continuously trained during operation), the GAN is able to learn in an unsupervised manner, with the generator attempting to generate images with an increasing measure of realism determined by the discriminator.

[0050] An autoencoder is an unsupervised artificial neural network that learns how to compress and encode data efficiently. An autoencoder learns how to reconstruct data from a simplified encoded representation back to a representation that is as close as possible to the original input. A VAE is an autoencoder whose encoding distribution is regularized during training in order to ensure that the latent space has properties sufficient to allow the generation of a certain portion of new data.

[0051] In one or more example implementations, the neural network model 104 can be implemented as a U-Net neural network. An example of a neural network implemented as a U-Net neural network is described in Ronneberger et al., “U-Net Convolutional Networks for Biomedical Image Segmentation” (May 18, 2015), which is incorporated herein by reference in its entirety.

[0052] exist Figure 1 In the example of , in addition to the mouth shape coefficients 112, the neural network model 104 also receives the head pose data 114. The neural network model 104 is trained to generate a data structure output, such as an image 116, given the vector of mouth shape coefficients 112 and the head pose data 114. More specifically, the neural network model 104 is trained to generate an image 116 of a mouth. The image 116 includes lips having a shape (e.g., a geometric shape) corresponding to the vector of mouth shape coefficients 112.

[0053] Typically, the neural network model can be trained to a point where the parameters of the neural network model have converged or substantially converged. The goal of the training process is to change the input-output relationship of the neural network model. The training process determines a set of parameters, such as weights and / or biases, that provide the desired input-output relationship for the neural network model. As an illustrative example, the learning process can repeatedly adjust the weights of the neural network model to change the input-output relationship so that the input-output accuracy cost function is optimized. During the training process, computational efficiency and / or power efficiency may not be a consideration. The term "computational efficiency" as used within the present disclosure means the ability to perform operations as part of executing a neural network model without wasting time or energy, where operations include but are not limited to calculations (e.g., multiplication, addition, etc.) and memory accesses. Although power efficiency can be considered independent of computational efficiency, increased power efficiency may be caused by increased computational efficiency of the system.

[0054] The image 116 of the mouth can be generated as a 2-dimensional (2D) red-green-blue (RGB) image. In one or more example embodiments, the image 116 can be incorporated into the character or used as part of the character. In this way, the character includes a high-quality image as the mouth area, including accurate mouth shape and lip shape. The character can be an avatar or digital person in any of a variety of different computing contexts, such as a video game, an Internet forum, and / or a virtual environment that can include a metaverse. Therefore, given conditions such as audio data 110, the architecture 100 is capable of generating high-quality images of the mouth. The image sequence 116 can be played as a video to animate the mouth talking and / or the human face talking. The architecture 100 is capable of generating an image 116 for each received vector of the mouth shape coefficients 112 extracted from the audio data 110, so that the resulting image and / or animation of the lip movement accurately tracks the speech contained in the audio data 110 or is synchronized with the speech contained in the audio data 110 (e.g., speech animation).

[0055] The example embodiments described herein do not rely on any particular representation of visemes. In one or more example embodiments, 34 visemes are used, resulting in a vector of 34 coefficients being provided to the neural network model 104. For example, the neural network model 104 may generate an image as an output for each vector of 34 coefficients received. However, it should be understood that other viseme frameworks with different numbers of visemes may be used. Other example mouth shape frameworks that may be used in place of visemes or suitable for use herein may include, but are not limited to, "blendshapes," also referred to as "morph targets."

[0056] In one or more other example implementations, the architecture described herein can be generalized beyond visemes. Because there are no assumptions about what the audio representation should be, mechanisms other than visemes can be used as long as the mechanism or representation closely corresponds to the mouth shape.

[0057] Figure 1 The example architecture 100 of differs from other conventional image generation systems in several respects. One difference is that the architecture 100 utilizes mouth shape coefficients 112 extracted from audio data 110, rather than extracting Mel-frequency cepstral coefficients (MFCCs). MFCCs describe the overall shape of the spectral envelope of an audio clip. Unlike visemes, MFCCs tend to have speech bias (e.g., speech dependency). That is, MFCCs tend to exhibit biases for a particular speaker captured in the audio data. For example, when using MFCCs, significant training must be performed using a large number of different identifiers (e.g., speech) with different speech amplitudes to overcome any such biases. This is often not feasible when seeking high-quality results.

[0058] Another difference is that the architecture 100 generates the image 116 directly from the mouth shape data (e.g., specified or encoded as the mouth shape coefficients 112). More specifically, the architecture 100 does not use or generate any facial landmark data when generating the image 116. The term "facial landmark data" means one or more key points. Facial landmark data or key points are individual points corresponding to different parts of the face (such as the corners of the lips, the corners of the eyes, etc.). Many conventional image generation systems use key points when generating mouth images. Key points are used as feature points to guide image generation. Facial landmark data lacks accuracy because such data is a prediction of the location of facial features (such as lips) within the image to be generated. The inaccuracy of facial landmark data also stems from the fact that facial landmark data are 2D data points that lack depth information. In terms of generating images of the mouth and lips, facial landmark data cannot represent certain shapes, such as "ahhh" and "ooo". More specifically, when flattened from a 3D representation to a 2D representation, the facial landmark data generated for the two different sounds appear identical in 2D space, despite having different depth information for each sound. In contrast, the mouth shape data, which is inherently 3D, retains such data, allowing for disambiguation between the shapes corresponding to "ahhh" and "ooo".

[0059] By avoiding the generation of and / or any reliance on facial landmark data, each of the various architectures described herein retains 3D information of the mouth shape data. The mouth shape coefficients 112 of the vector provided to the neural network model 104 retain depth information different from the facial landmark data, thereby allowing the neural network model 104 to generate images with high-quality lip shapes.

[0060] For purposes of illustration, the neural network model 104 includes an encoder 120 coupled to one or more layers 122. The layers 122 are coupled to a decoder 124. Figure 1 It is shown that due to the use of the vector of mouth shape coefficients 112, the size of the encoder 120 is smaller than that required by other conventional techniques for image generation, such as those utilizing MFCC and / or facial landmark data. Figure 1 Also shown is that image generation is performed directly from the vector of mouth shape coefficients 112. Smaller encoder implementations facilitate a data processing system to more efficiently execute architecture 100 (eg, faster runtime and / or execute using fewer hardware resources).

[0061] Figures 2A, 2B and 2C show examples of facial landmark data. While facial landmark data can be used for large, obvious expressions such as smiling, facial landmark data cannot capture the subtle shapes of the lips that typically occur during human speech. The examples of Figures 2A, 2B and 2C are derived from real-world data to illustrate the inaccuracies typically associated with facial landmark data. In the examples, the lines superimposed on top of the corresponding images are drawn from facial landmark data (e.g., key points). As can be observed, the lines drawn based on the facial landmark data are not aligned with the edges of the lips. For example, the lines miss the curvature of the lips.

[0062] In the example of FIG. 2A , the top of the lower lip is not aligned with the corresponding line drawn from the facial landmark data. In the example of FIG. 2B , the line drawn based on the facial landmark data indicates that the lips are closed when this is not the case, thereby not making a mouth opening. In the example of FIG. 2C , the facial landmark data is not aligned with the top of the lower lip. Image generation techniques that rely on facial landmark data are constrained by the limitations of the facial landmark data itself. That is, inaccuracies in the facial landmark data manifest themselves in images that are generated in a form that is less realistic with respect to the lip geometry and / or movement.

[0063] Figure 3 An example method 300 is shown that may be performed by a system that executes the architecture 100. As discussed, the architecture 100 may be implemented by, for example, Fig.10 The described data processing system (eg, a computer) or another suitable computing system may be used for execution.

[0064] In box 302, the system receives audio data 110. The audio data includes human speech. In box 304, the system extracts mouth shape data from the audio data 110 using a mouth shape extractor 102. In box 306, the mouth shape extractor 102 encodes the extracted mouth shape data into a vector of multiple mouth shape coefficients 112. The mouth shape coefficients can be viseme coefficients. In box 308, the system provides the vector of mouth shape coefficients 112 to the neural network model 104 as input. In box 310, the system optionally provides head posture data 114 to the neural network model 104. In box 312, the neural network model 104 generates a data structure output specifying a visual representation. For example, the neural network model 104 generates an image 116 of a mouth area. The image 116 includes lips having a shape corresponding to the vector of mouth shape coefficients 112.

[0065] It should be understood that the method 300 can be iterated to generate additional images 116. In general, for each vector of mouth shape coefficients 112 extracted from the audio data 110 corresponding to a particular moment in time, the neural network model 104 can generate an image 116. Thus, for a vector sequence of mouth shape coefficients 112 extracted from the audio data 110, where the audio data 110 includes human speech, the system can generate multiple images 116 that together form an animation of lip movements (e.g., a speech animation). In one or more example embodiments, the human speech includes speech generated using a technique such as text-to-speech.

[0066] In some cases, realism may not be desired, required, or the primary goal. As an example, many virtual environments, such as 3D game worlds, contain specific, intentionally unrealistic, and stylized artistic features. Even in such environments, accurate lip movement of a character is still necessary to convey the sense that the character is actually speaking. The arrangements of the present invention described in this disclosure can be used to drive accurate lip movement of a 3D character. For example, the mouth shapes of the "mesh" pipeline can be used to provide an enhanced sense of mouth and / or lip movement for a character, even though these characters have an intentionally unrealistic appearance.

[0067] Conventional techniques for creating mouth motion may include redirection. Redirection refers to a process in which motion from one item hierarchy is transferred to another item hierarchy. As an example, a motion such as the way a character walks can be applied to another character. In the case of redirection, there is no guarantee that the motion is transferable, or that the transferred motion will not be unnatural to the character to which the motion is transferred. By generating a mesh that specifies the location of features, the resulting image(s) can be controlled with greater precision. A mesh refers to a 3D representation of one or more objects. A mesh can be any of a variety of different types. A mesh represents an object in 3D with intersecting contour lines. A polygonal mesh represents an object in 3D with vertices, edges, and faces. A mesh is able to specify features and contours of the surface of an object in 3D. Thus, for example, a mesh of a mouth or face can provide increased accuracy of lip motion for a 3D character.

[0068] Figure 4 Another example implementation of an architecture 100 suitable for operating in a grid is shown. Figure 4 In the example of , the neural network model 104 is trained to generate a mesh 402. In addition, the architecture 100 includes a texture generator 404 that can apply or render a texture of the mesh 402 generated by the neural network model 104.

[0069] Figure 4 Some parts of the architecture 100 are substantially similar to Figure 1For example, the mouth shape extractor 102 extracts mouth shape data from the audio data 110 and encodes the mouth shape data into a vector of mouth shape coefficients 112, which are provided to the neural network model 104 as input. As described above, the mouth shape coefficients 112 can be viseme coefficients. The neural network model 104 also receives head pose data 114 as input.

[0070] In this example, the neural network model 104 is trained to generate the grid 402. Figure 4 In the example of , mesh 402 specifies a 3D representation of a head and face. Mesh 402 also specifies a mouth area including lips. Mesh 402 as generated by neural network model 104 is provided to texture generator 404 as input. Texture generator 404 is capable of generating a texture on top of mesh 402 to produce image 406 (e.g., a 2D representation of the mesh). In this example, texture generator 404 generates image 406 having skin texture, eye texture, eyebrow texture, hair texture, and lip texture superimposed on mesh 402. Mesh 402 and image 406 each include lips having a shape (e.g., a geometric shape) corresponding to a vector of mouth shape coefficients 112.

[0071] exist Figure 4 In the example of , the process can be repeated to generate additional meshes 402 and / or images 406 to produce an animation of lip movement (e.g., a speech animation). The meshes 402 and images 406 each include lips having a shape (e.g., a geometric shape) corresponding to the vector of the mouth shape coefficients 112.

[0072] Figure 5 shows that it can be executed by Figure 4 As discussed, the architecture 100 may be implemented by a system such as a Fig.10 The described data processing system (eg, a computer) or another suitable computing system may be used for execution.

[0073] In box 502, the system receives audio data 110. The audio data includes speech. In box 504, the system extracts mouth shape data from the audio data 110 using the mouth shape extractor 102. In box 506, the mouth shape extractor 102 encodes the extracted mouth shape data into a vector including a plurality of mouth shape coefficients 112. The mouth shape coefficients may be viseme coefficients. In box 508, the system provides the vector of mouth shape coefficients 112 to the neural network model 104 as input. In box 510, the system optionally provides head pose data 114 to the neural network model 104. In box 512, the neural network model 104 generates a data structure output specifying a visual representation. For example, the neural network model 104 generates a mesh 402 specifying a 3D representation of a head and face including a mouth area. The mesh 402 specifies lips having a geometric shape corresponding to the vector of mouth shape coefficients 112. In box 514, the system applies a texture to the mesh 402 to generate a final image, such as image 406. Similar to mesh 402 , image 406 includes lips having a shape corresponding to a vector of mouth shape coefficients 112 .

[0074] It should be understood that the method 500 can be iterated to generate additional grids and / or final images. In general, for each vector of mouth shape coefficients 112 extracted from the audio data 110 corresponding to a particular moment in time, the neural network model 104 can generate a grid 402 and / or an image 406. Thus, for a vector sequence of mouth shape coefficients 112 extracted from the audio data 110, where the audio data 110 includes speech, the system can generate multiple grids 402 and / or images 406 that together form an animation of lip movement (e.g., a speech animation).

[0075] In one or more other example embodiments, the architecture 100 is adapted to receive an existing mesh and adapt the mesh to a particular mouth shape corresponding to a portion of audio.

[0076] Figure 6 Another example implementation of an architecture 100 that can adapt a mesh to a specific mouth shape is shown. Figure 6 In the example of , the neural network model 104 is trained to modify the mesh based on the vector of mouth shape coefficients 112.

[0077] Figure 6 Some parts of the architecture 100 are substantially similar to Figure 1 and Figure 4 For example, the mouth shape extractor 102 extracts mouth shape data from the audio data 110 and encodes the mouth shape data into a vector of mouth shape coefficients 112, which is provided to the neural network model 104 as input. As described above, the mouth shape coefficients 112 can be viseme coefficients. For illustration purposes, the audio data 110 can be for a voice saying the sound "uuuu" (e.g., a long u sound).

[0078] exist Figure 6 In the example of , neural network model 104 also receives mesh 602 of a head and face, where the mouth region has a particular shape including lips. The mouth shape of mesh 602 is a shape corresponding to a particular viseme. For example, mesh 602 may correspond to a mouth shape for saying "eeee" (e.g., a long e sound). Thus, the shape of the mouth region in mesh 602 is different from the shape represented by mouth shape coefficients 112 provided to neural network model 104.

[0079] Neural network model 104 is trained to modify mesh 602 based on mouth shape coefficients 112 to generate mesh 604. In one aspect, mesh 604 is a version of mesh 602, e.g., an adjustment, derivative, or new mesh of mesh 602 having a different mouth region. Mesh 604 specifies a head and face including a mouth region having lips shaped according to mouth shape coefficients 112. Mesh 604 includes lips having a shape (e.g., a geometric shape) corresponding to a vector of mouth shape coefficients 112. For example, as from Figure 6 The mesh 604 output by the neural network model 104 in is suitable for specifying a mesh having a mouth area and lips corresponding to the sound “uuuu” as the specified mouth shape coefficient 112.

[0080] Figure 7 shows that it can be executed by Figure 6 As discussed, the architecture 100 may be implemented by a system such as a Fig.10 The described data processing system (eg, a computer) or another suitable computing system may be used for execution.

[0081] In box 702, the system receives audio data 110. The audio data 110 includes human speech. In box 704, the system extracts mouth shape data from the audio data 110 using the mouth shape extractor 102. In box 706, the mouth shape extractor 102 encodes the extracted mouth shape data into a vector including a plurality of mouth shape coefficients 112. In box 708, the system provides the vector of mouth shape coefficients 112 to the neural network model 104 as input. In box 710, the system provides the mesh 602 to the neural network model 104. The mesh 602 specifies a head and face having a mouth region including lips having a shape corresponding to a specific or selected mouth shape (e.g., viseme). In box 712, the neural network model 104 generates a data structure output specifying a visual representation. For example, the neural network model 104 generates a mesh 604 specifying a head and face including a mouth region and lips based on the vector of mouth shape coefficients 112 and the mesh 602. Mesh 604 includes lips having a shape corresponding to a vector of mouth shape coefficients 112 .

[0082] Figure 6 and Figure 7The example of illustrates a technique where, given a mesh for a first mouth shape or viseme, the mesh may be adapted to specify a second and different mouth shape or viseme. In one or more example embodiments, the mesh 602 may be from Figure 6 The image (e.g., grid 604) output by the previous iteration of the architecture 100. That is, Figure 6 In the operating architecture 100 for multiple sequential portions of audio to generate an animation of lip movements (e.g., a speaking animation), the grid 602 (also known as a "historical frame") can be the output generated by a previous iteration of the architecture 100, which is used to adjust the neural network model 104 for the next iteration.

[0083] In one or more other example embodiments, the mouth region of the original mesh may be masked as provided to the neural network model 104. By masking the mouth region of the mesh 602, the neural network model 104 generates lip shapes using only the mouth shape coefficients 112 without relying on the mouth shapes of the historical frames. This approach also allows the neural network model 104 to generate the rest of the facial motion (e.g., the mouth region including the lips) conditioned on the vise of the mouth shape coefficients 112 and the mesh 602 without the installation region.

[0084] Figure 8 Another example implementation of an architecture 100 that can be used to adapt a mesh to a specific mouth shape is shown. Figure 8 In the example of , the neural network model 104 is trained to modify the mesh based on the vector of mouth shape coefficients 112. In addition, the mouth area of ​​the mesh 602 is covered by the mask 802 (eg, masked).

[0085] Figure 8 Some parts of the architecture 100 are substantially similar to Figure 6 For example, the mouth shape extractor 102 extracts mouth shape data from the audio data 110 and encodes the mouth shape data into a vector of mouth shape coefficients 112, which are provided to the neural network model 104 as input. As described above, the mouth shape coefficients can be viseme coefficients. Figure 8 In the example of , neural network model 104 also receives mesh 602 in which the mouth region is masked by applying mask 802. Thus, mesh 602 does not specify any mouth shape or lips or any particular viseme.

[0086] The neural network model 104 is trained to modify the mesh 602 based on the vector of mouth shape coefficients 112 and fill the mouth area to generate the mesh 604. In one aspect, the mesh 604 is a version of the mesh 602, e.g., an adaptation, derivative, or new mesh of the mesh 602, having a filled mouth area. On the other hand, the mesh 604 specifies a head and a face, which includes a mouth area having lips shaped according to the mouth shape coefficients 112. The mesh 604 includes lips having a shape (e.g., a geometric shape) corresponding to the vector of the mouth shape coefficients 112. In this example, if the vector of the mouth shape coefficients 112 is for the sound "uuuu", the mesh 604 will include lips shaped to produce the sound "uuuu".

[0087] Fig. 9 shows that it can be executed by Figure 8 As discussed, the architecture 100 may be implemented by a system such as a Fig.10 The described data processing system (eg, a computer) or another suitable computing system may be used for execution.

[0088] In box 902, the system receives audio data 110. The audio data 110 includes speech. In box 904, the system extracts mouth shape data from the audio data 110 using the mouth shape extractor 102. In box 906, the mouth shape extractor 102 encodes the extracted mouth shape data into a vector including a plurality of mouth shape coefficients 112. In box 908, the system provides the vector of mouth shape coefficients 112 to the neural network model 104 as input. In box 910, the system provides the grid 602 to the neural network model 104. The grid 602 specifies a head and a face, wherein the mouth area is masked so as not to specify any particular mouth shape, lip shape, or viseme. In box 912, the neural network model 104 generates a data structure output specifying a visual representation. For example, the neural network model 104 generates a grid 604 specifying a head and a face having a mouth area with lips based on the vector of mouth shape coefficients by filling the masked area of ​​the grid 602. The grid 604 includes lips having a shape corresponding to the vector of mouth shape coefficients 112.

[0089] Fig.10 An example implementation of a data processing system 1000 is shown. As defined herein, the term "data processing system" refers to one or more hardware systems configured to process data, each hardware system including at least one processor and memory, wherein the processor is programmed with computer readable instructions that, when executed, initiate operations. Data processing system 1000 may include processor 1002, memory 1004, and bus 1006 that couples various system components including memory 1004 to processor 1002.

[0090] Processor 1002 may be implemented as one or more processors. In an example, processor 1002 is implemented as a central processing unit (CPU). Processor 1002 may be implemented as one or more circuits capable of executing instructions contained in a program code. The circuit may be an integrated circuit or embedded in an integrated circuit. Processor 1002 may be implemented using a complex instruction set computer architecture (CISC), a reduced instruction set computer architecture (RISC), a vector processing architecture, or other known architectures. Example processors include, but are not limited to, processors with a 10x6 type architecture (IA-32, IA-64, etc.), a Power architecture, an ARM processor, etc.

[0091] Bus 1006 represents one or more of any of a variety of communication bus structures. By way of example and not limitation, bus 1006 may be implemented as a Peripheral Component Interconnect Express (PCIe) bus. Data processing system 1000 typically includes various computer system readable media. Such media may include computer readable volatile and non-volatile media and computer readable removable and non-removable media.

[0092] Memory 1004 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) 1008 and / or cache memory 1010. Data processing system 1000 may also include other removable / non-removable, volatile / non-volatile computer storage media. As an example, storage system 1012 may be provided for reading and writing from non-removable, non-volatile magnetic and / or solid-state media (not shown and generally referred to as "hard drives") to non-removable, non-volatile magnetic and / or solid-state media. Although not shown, a disk drive for reading and writing removable non-volatile disks (e.g., "floppy disks") and an optical disk drive for reading or writing removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs or other optical media) may be provided. In this case, each may be connected to bus 1006 via one or more data media interfaces. Memory 1004 is an example of at least one computer program product.

[0093] Memory 1004 can store computer-readable program instructions that can be executed by processor 1002. For example, computer-readable program instructions can include an operating system, one or more application programs, other program codes, and program data. Computer-readable program instructions can implement any different examples of architecture 100 as described herein. Processor 1002 can perform various operations attributable to a computer described herein when executing computer-readable program instructions. It should be understood that the data items used, generated, and / or operated by data processing system 1000 are functional data structures that are given functions when adopted by data processing system 1000. As defined in this disclosure, the term "data structure" means a physical implementation of a data model organization of data in a physical memory. Therefore, a data structure is formed by specific electrical or magnetic structural elements in a memory. The data structure applies a physical organization to the data stored in the memory used by the application program executed using the processor. Examples of data structures include images and grids.

[0094] Data processing system 1000 may include one or more input / output (I / O) interfaces 1018 communicatively linked to bus 1006. I / O interfaces 1018 allow data processing system 1000 to communicate with one or more external devices and / or over one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or a public network (e.g., the Internet). Examples of I / O interfaces 1018 may include, but are not limited to, a network card, a modem, a network adapter, a hardware controller, etc. Examples of external devices may also include devices that allow a user to interact with data processing system 1000 (e.g., a display, a keyboard, a microphone for receiving or capturing audio data, a speaker, and / or a pointing device).

[0095] Data processing system 1000 is only one example implementation. Data processing system 1000 may be practiced as a stand-alone device (e.g., as a user computing device or server, as a bare metal server), in a cluster (e.g., two or more interconnected computers), or in a distributed cloud computing environment (e.g., as a cloud computing node), where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0096] Fig.10 The examples are not intended to suggest any limitation on the scope of use or functionality of the example embodiments described herein. Data processing system 1000 is an example of computer hardware capable of performing the various operations described within this disclosure. In this regard, data processing system 1000 may include, for example, Fig.10The fewer components shown in the figure or the additional components not shown, this depends on the specific type of the equipment and / or system implemented. The specific operating system and / or (one or more) applications included can change according to the equipment and / or system type, and the type of the I / O device included can also change according to the equipment and / or system type. In addition, one or more of the illustrative components can be incorporated into another component or otherwise form a part of another component. For example, the processor may include at least some memory.

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.Nevertheless, several definitions that apply throughout this document will now be presented.

[0098] As defined herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0099] The term "about" refers to nearly correct, an exact value or amount that is close but not exact. For example, the term "about" may indicate that the property, parameter or value is within a predetermined amount of the exact property, parameter or value.

[0100] As defined herein, unless expressly stated otherwise, the terms "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0101] As defined herein, the term "automatically" means without user intervention.

[0102] As defined herein, the term "computer-readable storage medium" refers to a storage medium that contains or stores program code for use by or in conjunction with an instruction execution system, device, or apparatus. As defined herein, a "computer-readable storage medium" is not itself a temporary propagating signal. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. As described herein, different types of memory are examples of computer-readable storage media. A non-exhaustive list of more specific examples of computer-readable storage media may include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and the like.

[0103] As defined herein, the term "if" means "when" or "upon" or "in response to" or "in response to," depending on the context. Thus, the phrase "if it is determined" or "if [a stated condition or event] is detected" may be interpreted to mean "upon the determination" or "in response to the determination" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0104] As defined herein, the terms "one embodiment," "an embodiment," "one or more embodiments," or similar language mean that a particular feature, structure, or characteristic described in conjunction with an embodiment is included in at least one embodiment described within the present disclosure. Thus, appearances of the phrases "in one embodiment," "in an embodiment," "in one or more embodiments," and similar language throughout the present disclosure may, but do not necessarily, all refer to the same embodiment. The terms "embodiment" and "arrangement" may be used interchangeably within the present disclosure.

[0105] As defined herein, the term "processor" means at least one hardware circuit. The hardware circuit may be configured to execute instructions contained in a program code. The hardware circuit may be an integrated circuit. Examples of processors include, but are not limited to, a central processing unit (CPU), an array processor, a vector processor, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), an application specific integrated circuit (ASIC), a programmable logic circuit, and a controller.

[0106] As defined herein, the term "in response to" and similar language, such as "if," "when," or "upon," as described above, means to readily respond to or react to an action or event. The response or reaction is automatically performed. Thus, if a second action is performed "in response to" a first action, there is a causal relationship between the occurrence of the first action and the occurrence of the second action. The term "in response to" indicates a causal relationship.

[0107] The term "substantially" means that the characteristic, parameter or value described need not be achieved precisely, but may deviate or vary by an amount that does not eliminate the effect that the characteristic is intended to provide, including, for example, tolerances, measurement errors, measurement precision limitations and other factors known to those skilled in the art.

[0108] As defined herein, the term "user" refers to a human being.

[0109] The terms first, second, etc. may be used herein to describe various elements. These elements should not be limited by these terms, because unless otherwise specified or the context clearly indicates otherwise, these terms are only used to distinguish one element from another element.

[0110] The computer program product may include (one or more) computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform various aspects of the present invention. In the present disclosure, the term "program code" may be used interchangeably with the term "computer-readable program instructions". The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, and / or a wireless network). The network may include a copper transmission cable, an optical transmission fiber, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge device including an edge server. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in the corresponding computing / processing device.

[0111] The computer-readable program instructions for performing the operation of the invention arrangement described herein can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​and / or process programming languages). Computer-readable program instructions can specify state setting data. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or executed completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including LAN or WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In some cases, the electronic circuit including, for example, a programmable logic circuit, FPGA or PLA can execute computer-readable program instructions with personalized electronic circuits by utilizing the state information of computer-readable program instructions, so as to perform the various aspects of the invention arrangement described herein.

[0112] Certain aspects of the inventive arrangement are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products. It should be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions (e.g., program code).

[0113] These computer-readable program instructions can be provided to a processor of a computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the function / action specified in one or more boxes of the flow chart and / or block diagram. In this way, the processor is operably coupled to the program code instructions to convert the machine of the processor into a special-purpose machine for executing the instructions of the program code. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, and / or other equipment to act in a particular manner, so that the computer-readable storage medium having instructions stored therein includes a product, which includes instructions for implementing various aspects of the operation specified in one or more boxes of the flow chart and / or block diagram.

[0114] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operations are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0115] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products of various aspects arranged according to the invention. In this regard, each frame in the flow chart or block diagram can represent a part of a module, segment or instruction, which includes one or more executable instructions for implementing a specified operation. In some alternative embodiments, the operation indicated in the frame may not occur in the order indicated in the figure. For example, the two frames shown in succession can be performed substantially simultaneously, or the frame can sometimes be performed in reverse order, depending on the functions involved. It should also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a system based on dedicated hardware that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0116] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements found in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed.

[0117] The description of the embodiments provided herein is for illustrative purposes and is not intended to be exhaustive or limited to the disclosed forms and examples. The terms used herein are selected to explain the principles of the present invention, practical applications, or technical improvements to the technology found in the market, and / or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. Modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described inventive arrangements. Therefore, reference should be made to the appended claims rather than the foregoing disclosure to indicate the scope of these features and embodiments.

Claims

1. A computer-implemented method comprising: Providing a vector of multiple coefficients as one or more input features of a neural network model, wherein each vector of the multiple coefficients corresponds to a different mouth shape; as well as A data structure output specifying a visual representation of a mouth including lips having a shape corresponding to the vector is generated using the neural network model.

2. The computer-implemented method of claim 1, wherein the plurality of coefficients of each vector are viseme coefficients.

3. The computer-implemented method of claim 2, further comprising: Viseme coefficients are generated from audio data of speech.

4. The computer-implemented method of claim 2, further comprising: The viseme coefficients are encoded into the vector provided to the neural network model.

5. The computer-implemented method of claim 1, wherein the data structure output is generated directly from the vector of the plurality of coefficients without using any facial landmark data.

6. The computer-implemented method of claim 1 , wherein: The data structure output is a red-green-blue (RGB) two-dimensional image.

7. The computer-implemented method of claim 1 , wherein: The data structure output is a grid.

8. The computer-implemented method of claim 1 , wherein: Generating the data structure output includes: A plurality of images are generated using the neural network model, wherein each image is generated from a different vector having a plurality of coefficients, and wherein the plurality of images collectively represent an animation of lip movement.

9. The computer-implemented method of claim 1 , further comprising: providing a first mesh of a face to the neural network model using the vector of the one or more coefficients; wherein the data structure output is a second mesh generated by the neural network model based on the vector and the first mesh of the face, wherein the second mesh is a version of the first mesh of the face that specifies a mouth including lips having a shape corresponding to the vector.

10. A system comprising: One or more processors configured to initiate operations comprising: providing a vector of multiple coefficients as one or more input features of a neural network model, wherein each vector of the multiple coefficients corresponds to a different mouth shape; and A data structure output specifying a visual representation of a mouth including lips having a shape corresponding to the vector is generated using the neural network model.

11. The system of claim 10, wherein the plurality of coefficients of each vector are viseme coefficients.

12. The system according to claim 11, wherein: The one or more processors are configured to initiate operations comprising: Viseme coefficients are generated from audio data of speech.

13. The system according to claim 11, wherein: The one or more processors are configured to initiate operations comprising: The viseme coefficients are encoded into the vector provided to the neural network model.

14. The system of claim 10, wherein the data structure output is generated directly from the vector of the plurality of coefficients without using any facial landmark data.

15. The system according to claim 10, wherein: The data structure output is a red-green-blue (RGB) two-dimensional image.

Images