Multi-modal deentanglement for generating virtual human avatars

Abstract

The multi-modal deentanglement may include generating a set of silhouette images corresponding to the face, the generating cancelling a correlation between an upper portion and a lower portion of the face depicted by each silhouette image. A single-modal machine learning model may be trained with the set of silhouette images. As trained, the single-mode machine learning model may generate a composite image of the face. A composite image generated by the trained single-modal machine learning model may be used to train a multi-modal rendering network. A multi-modal rendering network may be trained to generate a voice animated digital person. Training the multi-modal rendering network may be based on minimizing a difference between the composite image and an image generated by the multi-modal rendering network.

Description

Technical Field

[0001] The present disclosure relates to machine learning and image processing, and more particularly, to generating virtual humans and other avatars using machine learning. Background Art

[0002] The ability to generate high-quality images of characters (whether digital humans, avatars, or other anthropomorphic objects) is increasingly important. Among the reasons for the increasing importance are the growing popularity of the metaverse, the adoption of virtual experiences across different parts of society, and recent advances in hardware and other technologies (e.g., neural networks) that facilitate rapid virtualization. For example, the goal of rendering a digital human is to be as close to a real person as possible (if not completely). In achieving this goal, the quality of both the textures used and the resolution of the resulting image play an important role. Preferably, the digital human image is rendered with accurate textures so that the hair looks like it is composed of individual hair strands; the skin looks like it has pores; the clothes look like they are made of fabric, etc. Even for other use cases, such as the generation of avatars (whether humanoid characters, humanoid characters, or other anthropomorphic objects), the accuracy and realism of the rendered images are important.

[0003] In many cases, digital humans, avatars, etc. are presented on large screens. A digital human may be presented on a screen, for example, at an airport or hotel kiosk, in a life-size rendering (e.g., with an average height and size corresponding to a real person). Life-size renderings tend to highlight any visual irregularities, so the generation of high-quality images of digital humans, avatars, and other anthropomorphic objects remains challenging. Accurate generation of certain features such as the movement of the mouth and / or lips of an image is important because actual human perception of these areas is very sensitive to any perceptible visual artifacts (such as the movement of the digital human's lips and misalignment of the digital human's speech). Summary of the invention

[0004] Technical issues

[0005] In one or more embodiments, a computer-implemented method includes generating a set of silhouette images corresponding to a human face, the generation disassociating the upper and lower portions of the human face depicted by each silhouette image. The method includes training a unimodal machine learning model to generate a synthetic image of the human face, the unimodal model being trained using the set of silhouette images. The method includes using the output of the unimodal machine learning model to train a multimodal rendering network to generate a synthetic image of a speech-animated digital human. Training the multimodal rendering network is based on minimizing the difference between the synthetic image and the image generated by the multimodal rendering network.

[0006] In one aspect, generating the silhouette image may include randomly pairing a first silhouette image with a second silhouette image, the first silhouette image and the second silhouette image depicting an upper half and a lower half of a face, respectively. The silhouette image is generated by merging the first silhouette image and the second silhouette image.

[0007] On the other hand, merging the first silhouette image and the second silhouette image may include extracting associated sets of keypoints from each of the first silhouette image and the second silhouette image, frontalizing the sets of keypoints of each of the first silhouette image and the second silhouette image, generating a frontalized silhouette by adding the frontalized sets of keypoints of the first silhouette image and the second silhouette image, and defrontalizing the frontalized silhouette to generate the silhouette image.

[0008] On the other hand, frontalizing the plurality of sets of key points may include multiplying the plurality of sets of key points of the first silhouette image and the second silhouette image by the inverse of the head pose matrix of the first image and the inverse of the head pose matrix of the second silhouette image, respectively. Defrontalizing the frontalized silhouette may include multiplying the frontalized silhouette by the head pose matrix of the first image or the head pose matrix of the second image.

[0009] In another aspect, generating each composite image may include performing an image-to-image transformation of the corresponding merged silhouette image.

[0010] In one or more embodiments, a system includes one or more processors configured to initiate operations. The operations include generating a set of silhouette images corresponding to human faces, the generation disassociating the upper and lower portions of the human faces depicted by each silhouette image. The operations include training a unimodal machine learning model to generate a synthetic image of the human face, the unimodal model being trained using the set of silhouette images. The operations include outputting from the unimodal machine learning model to train a multimodal rendering network to generate a synthetic image of a speech-animated digital human. Training the multimodal rendering network is based on minimizing the difference between the synthetic image and an image generated by the multimodal rendering network.

[0011] In one aspect, generating the silhouette image may include randomly pairing a first silhouette image with a second silhouette image, the first silhouette image and the second silhouette image depicting an upper half and a lower half of a face, respectively. The silhouette image is generated by merging the first silhouette image and the second silhouette image.

[0012] On the other hand, merging the first silhouette image and the second silhouette image may include extracting multiple sets of key points associated from each of the first silhouette image and the second silhouette image, frontalizing the multiple sets of key points of each of the first silhouette image and the second silhouette image, generating a frontalized silhouette by adding the frontalized multiple sets of key points of the first silhouette image and the second silhouette image, and defrontalizing the frontalized silhouette to generate the silhouette image.

[0013] On the other hand, frontalizing the multiple sets of key points may include multiplying the multiple sets of key points of the first silhouette image and the second silhouette image by the inverse of the head pose matrix of the first image and the inverse of the head pose matrix of the second silhouette image, respectively. Defrontalizing the frontalized silhouette may include multiplying the frontalized silhouette by the head pose matrix of the first image or the head pose matrix of the second image.

[0014] In another aspect, generating each composite image may include performing an image-to-image transformation of the corresponding merged silhouette image.

[0015] In one or more embodiments, a computer program product includes one or more computer-readable storage media having program code stored thereon. The program code is executable by one or more processors to perform operations. The operations include generating a set of silhouette images corresponding to human faces, the generation disassociating the upper and lower portions of the human faces depicted by each silhouette image. The operations include training a unimodal machine learning model to generate a composite image of a human face, the unimodal model being trained using the set of silhouette images. The operations include outputting from the unimodal machine learning model to train a multimodal rendering network to generate a composite image of a speech-animated digital human. Training the multimodal rendering network is based on minimizing the difference between the composite image and an image generated by the multimodal rendering network.

[0016] In one aspect, generating the silhouette image may include randomly pairing a first silhouette image with a second silhouette image, the first silhouette image and the second silhouette image depicting an upper half and a lower half of a face, respectively. The silhouette image is generated by merging the first silhouette image and the second silhouette image.

[0017] On the other hand, merging the first silhouette image and the second silhouette image may include extracting multiple sets of key points associated from each of the first silhouette image and the second silhouette image, frontalizing the multiple sets of key points of each of the first silhouette image and the second silhouette image, generating a frontalized silhouette by adding the frontalized multiple sets of key points of the first silhouette image and the second silhouette image, and defrontalizing the frontalized silhouette to generate the silhouette image.

[0018] On the other hand, frontalizing the multiple sets of key points may include multiplying the multiple sets of key points of the first silhouette image and the second silhouette image by the inverse of the head pose matrix of the first image and the inverse of the head pose matrix of the second silhouette image, respectively. Defrontalizing the frontalized silhouette may include multiplying the frontalized silhouette by the head pose matrix of the first image or the head pose matrix of the second image.

[0019] In another aspect, generating each composite image may include performing an image-to-image transformation of the corresponding merged silhouette image.

[0020] This summary is provided only to introduce certain concepts, and does not identify any key or essential features of the claimed subject matter. Various other features and embodiments of the present invention will be apparent from the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings illustrate one or more embodiments; however, the drawings should not be considered to limit the invention to only the embodiments shown. Various aspects and advantages will become apparent by reading the following detailed description and referring to the drawings.

[0022] Figure 1 An example of an architecture that may be executed by a data processing system to perform image generation is shown.

[0023] Figure 2 Shows Figure 1 Certain operational aspects of the architecture.

[0024] Figure 3 Shows Figure 1 Certain operational aspects of the architecture.

[0025] Figure 4 Shows Figure 1 Certain operational aspects of the architecture.

[0026] Figure 5 shows that it can be executed by Figure 1 An example method for performing a system of the architecture.

[0027] Figure 6 Shows the use of Figure 1 An example of the output of training a multimodal rendering network.

[0028] Figure 7 An example implementation of a data processing system capable of executing the architecture described within this disclosure is shown. DETAILED DESCRIPTION

[0029] Although the present disclosure ends with claims defining novel features, it is generally recognized that the various features described herein will be better understood by considering the description in conjunction with the accompanying drawings. The process, machine, manufacture, and any variations thereof described within the present disclosure are provided for illustrative purposes. Any specific structural and functional details described should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the features described in virtually any appropriately detailed structure. Furthermore, the terms and phrases used within the present disclosure are not intended to be limiting, but rather to provide an understandable description of the features described.

[0030] The present disclosure relates to machine learning and image processing, and more specifically, to generating virtual humans and other avatars using machine learning. The process of manually creating digital humans is extremely labor-intensive and may not produce realistic representations. In order to overcome these challenges, neural networks and deep learning techniques are increasingly relied upon to generate virtual human avatars. However, creating virtual human avatars using machine learning (e.g., neural networks, deep learning) brings other unique challenges. One of the challenges is training machine learning models on multimodal data. Multimodal data imposes multiple model constraints corresponding to speech, body posture, body shape, and other features of multimodal data. Speech can be represented as a vector (corresponding to an audio wave), body posture can be represented as a rotation matrix, and body shape can be represented as a 2D image contour. Training deep neural networks on multimodal data is particularly challenging because multimodal data tends to be correlated. Therefore, the model tends to overfit relative to the modality that the model is easiest to learn, and therefore affects the training of the model to the greatest extent. Overfitting causes machine learning models to have very poor performance in processing multimodal test data. Conventional techniques for mitigating overfitting (such as reducing the number of parameters, regularization, etc.) fail if overfitting is caused by multiple modalities of the training data.

[0031] According to the inventive arrangements disclosed herein, methods, systems, and computer program products capable of disentangling multiple modalities and disentangling correlations between modalities are described. One aspect of the inventive arrangements is the generation of synthetic data that can be used to train a multimodal rendering network. "Synthetic" refers to generated for the specific purpose of disentangling different modalities and disentangling correlations between modalities. On the one hand, "synthetic" refers to machine-generated for the aforementioned purpose. The term applies to both images and data in this article. By disentangling image modalities and audio modalities, synthetic data can be used to train a multimodal rendering network to render a speech-animated digital human, wherein head movements are accurately aligned with the lip movements of the digital human. On the one hand, "speech-animated digital human" refers to multiple humanoid images rendered sequentially and combined with audio rendering of human speech.

[0032] The inventive arrangement can be applied to a variety of tasks where modal entanglement is a problem or generalization of machine learning is required.

[0033] Other aspects of the arrangement of the present invention are described in more detail below with reference to the accompanying drawings. For the purpose of simplicity and clarity of explanation, the elements shown in the figures are not necessarily drawn to scale. For example, for the sake of clarity, the size of some elements may be enlarged relative to other elements. In addition, where it is considered appropriate, reference numerals are repeated in the accompanying drawings to indicate corresponding, similar or similar features.

[0034] Figure 1 An example computer-implemented architecture 100 capable of generating synthetic images is shown. The synthetic images can be used to train a multimodal rendering network to generate a speech-animated digital human. The architecture 100 can be implemented as a software framework that can be executed by a data processing system. Figure 7 Examples of data processing systems suitable for executing architecture 100 and / or training machine learning models included in architecture 100 are described.

[0035] The architecture 100 illustratively includes a synthetic data generator 102 and a machine learning model 104. The synthetic data generator 102 and the machine learning model 104 are illustratively arranged in series as a pipeline.

[0036] Operably, the synthesized data generator 102 generates synthesized data based on the input of training data 106. The training data 106 is multimodal data whose modalities include image modalities and audio modalities. The operation of the synthesized data generator 102 disentangles the modalities of the training data 106. Disentangling disentangles the correlation between the image modality and the audio modality of the training data 106. After disentangling, the synthesized data generator 102 outputs a silhouette image 108 corresponding to a human face. The silhouette image 108 is generated by extracting key points and contours from an image of a human face using a filter or a processor during preprocessing. Therefore, the key points and contours of the silhouette image 108 are pixel-by-pixel aligned features of the human face.

[0037] The silhouette image includes the edge outline and key points of the face. Key points are individual points corresponding to different parts of the face (such as the corners of the mouth, the corners of the eyes, etc.). Key points can be used to generate an image of the face (including the mouth) and can be used as landmarks to guide image generation.

[0038] Figure 2An example process 200 of generating a silhouette image 108 performed by the synthetic data generator 102 is shown. The synthetic data generator 102 generates each of the silhouette images 108 in a manner that eliminates the correlation between the head motion and the mouth motion of the image of the human face. As generated, the upper head of each of the silhouette images 108 is uncorrelated with the lower head (particularly the mouth) of each of the silhouette images 108. Initially, the synthetic data generator 102 randomly selects a pair of image frames i and j from the training data 106. Illustratively, the randomly paired image frames are the i-th frame 202 and the j-th frame 204. The synthetic data generator 102 performs operation 206 on the two frames. Operation 206 frontalizes each of the i-th frame 202 and the j-th frame 204, generating a frontalized frame 208 and a frontalized frame 210, respectively. By replacing the key point K of the i-th frame 202 with the frontalized frame 210, the frontalized frame 210 is generated. i and the key point K of the j-th frame 204 j Multiply by the matrix P i and P j Operation 206 is performed by inverse of i is the head pose matrix of the i-th frame 202, and P j is the head pose matrix of the j-th frame 204.

[0039] The key point K i and K j Each of them is multiplied by the inverse of their respective head pose matrix to obtain the key point K i ' and K j '. Frame 208 is passed through a filter or processor that passes only the upper portion of the frame, generating frame 212. Frame 210 is passed through a filter or processor that passes only the lower portion of the frame, generating frame 214. Therefore, frame 212 is the upper portion of frame 208, and frame 214 is the lower portion of frame 210. After the frames are frontalized, the key point K' of frame 212 (upper portion) i upper and the key point K′ of frame 214 (lower part) j lower In the same 2D space, keypoints can be merged by matrix addition.

[0040] K′ merged =K′ iupper +K′ j lower ,

[0041] Operation 216 is then performed. Operation 216 merges the upper keypoint and the lower keypoint into a single merged frame. Operation 218 is performed to de-frontalize the merged frame and generate a merged keypoint K merged By transforming the positive key point K' merged Multiply by the head pose matrix Pi Or the head pose matrix P j To perform de-frontalization on the frontalized merged frame. For example, if the key point K' merged Multiply by the head pose matrix P i To perform operation 218, then

[0042] K merged =P i K′ merged .

[0043] Two sets of key points (key points K of the i-th frame 202 i and the key point K of the j-th frame 204 j ) can be used to generate synthetic data, so the matrix multiplication obtains a new image that disentangles the mouth from the pose of the image. Therefore, the key point K' can be used merged and the head pose matrix P i Or the head pose matrix P j De-frontalization of the frontalized merged frame is performed by matrix multiplication of .

[0044] The synthetic data generator 102 is constructed from the key points K merged Generate silhouette image X ij By repeating process 200 for the rest of the training data 106, the corresponding frame 220 can be obtained with the silhouette image X ij Each of the other silhouette images in the silhouette image 108 is generated in the same manner.

[0045] The silhouette image 108 is fed into the machine learning model 104 to train the model to generate the synthetic image 110 .

[0046] Figure 3The machine learning model 104 is shown generating a composite image 110 from a silhouette image 108. Illustratively, the machine learning model 104 is a neural network that includes an encoder 112 coupled to one or more layers 114, which in turn are coupled to a decoder 116. In some embodiments, the machine learning model 104 is a Pix2Pix conditional generative adversarial network (GAN) trained to perform image-to-image conversion. Based on the conditional GAN, the machine learning model 104 generates a target image conditioned on a given input image. In general, a GAN includes two neural networks, referred to as a generator and a discriminator, which participate in a zero-sum game with each other. Given a training set, the GAN is able to learn to generate new data with the same statistics as the training set. As an illustrative example, a GAN trained on an image or image library is able to generate different images that look real to a human observer. The GAN generator generates an image. The GAN discriminator determines a measure of the authenticity of the image generated by the generator. Because the two neural networks can be dynamically updated during operation (e.g., continuously trained during operation), the GAN is able to learn in an unsupervised manner, where the generator attempts to generate images that increase in a measure of realism as determined by the discriminator. Specifically, with the Pix2Pix GAN, the GAN changes the loss function so that the generated images are reasonable in the context of the target domain and are reasonable transformations of the images input to the model. As a Pix2Pix GAN, the machine learning model 104 can perform an image-to-image transformation of the merged silhouette image to generate a synthetic image.

[0047] The synthetic images 110 generated by the machine learning model 104 are unimodal, but are used to train a multimodal machine learning model (e.g., a multimodal rendering network) to generate a speech-animated digital human. The multimodal machine learning model trained using the synthetic images 110 generates a speech-animated digital human whose head movement and mouth movement (including lips) are accurately aligned. Each of the synthetic images 110 can depict a human face with a different mouth shape.

[0048] Figure 4The use of a synthetic image 110 generated by the architecture 100 for training a multimodal rendering network 400 is shown. Note that the operations performed by the architecture 100 can occur simultaneously with or before the training of the multimodal rendering network 400. The multimodal rendering network 400 illustratively includes a silhouette encoder 402, an audio encoder 404, a machine learning model 406, a decoder 408, a determiner 410, and a preprocessor 412. The multimodal rendering network 400 processes the training data 106 in learning to generate a speech animated digital human 414. The training data 106 includes image modality data 416 and audio modality data 418. The image modality data 416 used to train the machine learning model 406 includes an image, a corresponding silhouette (e.g., key points, contours), and pose data of the image. The silhouette can be efficiently represented by 2D coordinates. The audio modality data 418 includes a vector representation (corresponding to an audio wave) that digitally encodes human speech. The preprocessor 412 generates a partial drawing 420 based on the image modality data 416. The partial drawing 420 may be generated using a filter or processor to filter out the mouth and chin areas, thereby creating a portion of the silhouette from the silhouette that includes only the upper head portion. Multiple images of the voice-animated digital human are similarly generated along with the voice-animated digital human 414 and rendered in sequence to produce an animation of the digital human. The sequential rendering of the multiple images is performed in coordination with the audio rendering of the human's speech. Therefore, the voice-animated digital human appears to speak like a human and speaks in a human voice.

[0049] Note that the mouth of the partial drawing 420 is omitted. The mouth of the speech animation digital human 414 is generated based on the viseme corresponding to the audio modality data 418. The viseme specifies the shape of the mouth at the vertex of a given phoneme. Each phoneme is associated with or generated by a viseme. Each viseme can represent one or more phonemes. In this regard, there is a many-to-one mapping of phonemes to visemes. When a specific phoneme is spoken, the viseme is usually generated as an artistic rendering of the shape of the mouth (e.g., lips). The audio encoder 404 pushes each viseme corresponding to the audio modality into a potential space shared with the image modality data 416. The partial drawing 420 is input to the silhouette encoder 402, which compresses the corresponding data. The data is processed by multiple layers of the machine learning model 406 and decoded by the decoder 408 to generate the speech animation digital human 414.

[0050] The synthetic image 110 provides ground truth for training the multimodal rendering network 400 through continuous learning cycles (epochs). The multimodal rendering network 400 undergoes iterative changes to reduce the difference between the image 414 generated by the network and the synthetic image 110 generated by the architecture 100. The difference is determined by the determiner 410. In various embodiments, the determiner 410 calculates a quantitative measure of the difference based on L1 loss, GAN loss, VGG loss or other perceptual loss metrics. Because the synthetic image 110 is trained using synthetic data of disentangled image modalities and audio modalities, the multimodal rendering network 400 is able to learn to generate a speech animation digital human, in which the lip movements of the digital human are realistically aligned with the audio (speech) of the digital human. A more realistic alignment is achieved because the image data is not dominant compared to the audio when training the multimodal rendering network 400, which would otherwise result in overfitting and poor alignment of the image and audio.

[0051] The architecture 100 generates a synthetic image 110 from synthetic data. But the synthetic data is derived from the same data that was input to the multimodal rendering network 400. However, the synthetic data is essentially image data disentangled from the other modalities of the training data that was input to the multimodal rendering network 400. Therefore, the synthetic image 110 generated from the synthetic data can provide a ground truth in which the correlation between head motion (image modality data 416) and mouth motion (audio modality data 418) is eliminated. In one aspect, the mouth motion is determined by the visemes pushed into a latent space shared with the image modality data 416 by the audio encoder 404 as described above. The process operates to synchronize the mouth motion and speech rendering of the speech animated digital human so that, as rendered, the speech animated digital human appears to have the natural speaking ability of the human.

[0052] The synthetic image 110 is the ground truth for training the machine learning model 406. The image modality data 416 (which is also input to the synthetic data generator 102 to generate the synthetic data for deriving the synthetic image 110) is the same as the image modality data 416 used to generate the partial drawing 420 (although the mouth is eliminated) along with the audio modality data 418. Nevertheless, the mouth key points excluded from the partial drawing 420 are related to the audio modality data 418. The key points excluded from the partial drawing 420 are the key points used in deriving the synthetic image 110 (the ground truth) and matching the audio used by the multimodal rendering network 400 to generate the speech animated digital human 414. Specifically, the key points of the mouth position of the lower head silhouette used to generate the synthetic image 110 can be indexed, labeled or otherwise matched to the audio and visemes input to the machine learning model 406. Therefore, the audio and mouth portions of the speech animated digital human 414 that are trained to generate following the machine learning model 406 are related. Since the audio modality data 418 corresponds to the key points for deriving the mouth position of the lower head portion of the synthetic image 110, the mouth portion of the synthetic image 110 corresponds to the mouth portion of the speech-animated digital human 414. However, since the correlation between the mouth and the head of the speech-animated digital human 414 has been released, there is no correlation between the head movement and the mouth movement of the speech-animated digital human 414.

[0053] Figure 5 An example method 500 is shown that may be performed by a system that executes the architecture 100. As described, the architecture 100 may be implemented by, for example, Figure 7 The described data processing system (eg, computer) executes.

[0054] In block 502, the system randomly pairs a first silhouette image with a second silhouette image. The first silhouette image and the second silhouette image depict the upper half and the lower half of a human face, respectively. The silhouette image can be extracted from or generated based on the training data 106, which is also input into the multimodal rendering network for generating a speech animation digital human.

[0055] In block 504, the system generates a silhouette image. The silhouette image is generated by merging the first silhouette image and the second silhouette image. At block 506, the system continues pairing the silhouette images until all images that are part of the training data 106 have been paired. The generated silhouette images collectively represent a set of silhouette images 108.

[0056] In block 508, the system uses the set of silhouette images 108 to train the unimodal machine learning model 104. Once trained, the machine learning model 104 is able to generate the composite image 110. Any correlation between the original images of the training data 106 has been resolved by the process of merging to generate the silhouette image 108. Therefore, the head and mouth (including lips) motion in the composite image 110 are uncorrelated.

[0057] In block 510, the system outputs a composite image 110. The composite image 110 is input to a multimodal rendering network.

[0058] In block 512, a multimodal rendering network is trained using the synthetic image 110. The multimodal rendering network is trained to generate a speech-animated digital human. The multimodal rendering network may be trained through successive cycles to reduce the difference between the speech-animated digital human generated by the network and the synthetic image 110. Each network-generated image of the digital human may uniquely correspond to one of the synthetic images 110. By iteratively training the multimodal rendering network to minimize the difference with the synthetic image 110, the multimodal rendering network learns to generate a speech-animated digital human whose head movement and mouth movement are realistically aligned.

[0059] Although primarily described herein in the context of image and audio disentanglement, architecture 100 performs other tasks where modal disentanglement is a problem or where the rendering system benefits from generalization. For example, architecture 100 can be used to generalize a multimodal rendering network for multiple identity rendering. With identity images as well as silhouette images, architecture 100 can train a multimodal rendering network using modalities of multiple identities so that the network learns to render multiple arbitrary identities.

[0060] exist Figure 6 , the output of the architecture 100 is used to train a multimodal rendering network 600 to render an entity whose physical appearance has one identity but has the voice and speaking style of a different identity. The multimodal rendering network 600 illustratively includes a generator 602 and a determiner 604. The input 606 of the multimodal rendering network 600 includes a silhouette 608 and audio 610 of a first identity corresponding to a voice-animated digital human 614. In addition, an image template 612 corresponding to a second identity is input to the generator 602 of the multimodal rendering network 600. The architecture 100 disentangles the modality of the input 606 to generate a synthetic image 616 corresponding to the second identity. The multimodal rendering network 600 uses the silhouette 608 and audio 610 corresponding to the first identity to generate the speaking style and mouth movement of the second identity. The image template 612 provides the texture of the second identity, such as facial features, hair color, clothing fabric, etc.

[0061] Thus, using the image template 612, the multimodal rendering network 600 generates a speech-animated digital human 618 corresponding to the second identity. Multiple images of the speech-animated digital human are similarly generated together with the speech-animated digital human 618, and are sequentially rendered in coordination with the human speech audio to produce an animation of the digital human. The speech-animated digital human 618 has the facial features, hair color, and other texture aspects of the second identity. However, the speech of the speech-animated digital human 618 has the mouth movement and voice of the first identity. Using different image templates, the multimodal rendering network 600 is able to learn how to generate multiple identity renderings by minimizing the difference between the speech-animated images generated by the network and the synthetic images generated by the architecture 100.

[0062] In other embodiments, the architecture 100 performs knowledge distillation. Knowledge distillation enables the multimodal rendering network to be implemented as a smaller network. As a smaller network, the multimodal rendering network learns only relevant features from the separated modalities to generate a voice-animated digital human. The multimodal rendering network is able to reverse-learn any unimportant features and prevent overfitting, while also keeping the network smaller to perform inference faster.

[0063] Figure 7 An example implementation of a data processing system 700 is shown. As defined herein, the term "data processing system" refers to one or more hardware systems configured to process data, each hardware system including at least one processor and memory, wherein the processor is programmed with computer readable instructions that initiate operations when executed. Data processing system 700 may include processor 702, memory 704, and bus 706 that couples various system components including memory 704 to processor 702.

[0064] Processor 702 can be implemented as one or more processors. In an example, processor 702 is implemented as a central processing unit (CPU). Processor 702 can be implemented as one or more circuits capable of executing instructions contained in a program code. The circuit can be an integrated circuit or embedded in an integrated circuit. Processor 702 can be implemented using a complex instruction set computer architecture (CISC), a reduced instruction set computer architecture (RISC), a vector processing architecture, or other known architectures. Example processors include, but are not limited to, processors with a 10x6 type architecture (IA-32, IA-64, etc.), a Power architecture, an ARM processor, etc.

[0065] Bus 706 represents one or more of any of a variety of communication bus structures. By way of example and not limitation, bus 706 may be implemented as a peripheral component interconnect express (PCIe) bus. Data processing system 700 typically includes various computer system readable media. Such media may include computer readable volatile and non-volatile media and computer readable removable and non-removable media.

[0066] Memory 704 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) 708 and / or cache memory 710. Data processing system 700 may also include other removable / non-removable, volatile / non-volatile computer storage media. As an example, storage system 712 may be provided for reading from and writing to non-removable, non-volatile magnetic and / or solid-state media (not shown and generally referred to as a "hard drive"). Although not shown, a disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") may be provided, as well as an optical disk drive for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media). In this case, each may be connected to bus 706 via one or more data media interfaces. Memory 704 is an example of at least one computer program product.

[0067] The memory 704 can store computer-readable program instructions that can be executed by the processor 702. For example, the computer-readable program instructions may include an operating system, one or more application programs, other program code, and program data. The computer-readable program instructions can implement any of the different examples of the architecture 100 as described herein, including the synthetic data generator 102 and the machine learning model 104. The processor 702, when executing the computer-readable program instructions, can perform various operations attributable to computers described herein, including in conjunction with Figure 5 The operations of the method 500 described herein. It should be understood that the data items used, generated and / or operated by the data processing system 700 are functional data structures that impart functionality when employed by the data processing system 700. As defined within this disclosure, the term "data structure" refers to a physical implementation of a data model organization of data within a physical memory. As such, a data structure is formed by specific electrical or magnetic structural elements in the memory. The data structure imposes a physical organization on the data stored in the memory used by the application program executed using the processor. Examples of data structures include images and grids.

[0068] Data processing system 700 may include one or more input / output (I / O) interfaces 718 communicatively linked to bus 706. I / O interfaces 718 allow data processing system 700 to communicate with one or more external devices and / or over one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or a public network (e.g., the Internet). Examples of I / O interfaces 718 may include, but are not limited to, a network card, a modem, a network adapter, a hardware controller, etc. Examples of external devices may also include devices that allow a user to interact with data processing system 700 (e.g., a display, a keyboard, a microphone for receiving or capturing audio data, a speaker, and / or a pointing device).

[0069] Data processing system 700 is only one example implementation. Data processing system 700 may be practiced as a standalone device (e.g., as a user computing device or server, as a bare metal server), in a cluster (e.g., two or more interconnected computers), or in a distributed cloud computing environment (e.g., as a cloud computing node), where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0070] Figure 7 The examples are not intended to suggest any limitation on the scope of use or functionality of the example implementations described herein. Data processing system 700 is an example of computer hardware capable of performing the various operations described within this disclosure. In this regard, data processing system 700 may include, for example, Figure 7 Fewer components or components not shown in Figure 7 The additional components shown in the figure may depend on the specific type of device and / or system implemented. The specific operating system and / or applications included may vary depending on the type of device and / or system, and the type of I / O devices included may also vary depending on the type of device and / or system. In addition, one or more of the illustrative components may be incorporated into another component or otherwise form a part of another component. For example, a processor may include at least some memory.

[0071] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.Nevertheless, several definitions that apply throughout this document will now be presented.

[0072] As defined herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0073] The term "about" means almost correct or exact, a value or amount is close but not exact. For example, the term "about" may indicate that the property, parameter or value is within a predetermined amount of the exact property, parameter or value.

[0074] As defined herein, unless expressly stated otherwise, the terms "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0075] As defined herein, the term "automatically" means without user intervention.

[0076] As defined herein, the term "computer-readable storage medium" refers to a storage medium that contains or stores program code for use by or in conjunction with an instruction execution system, device, or apparatus. As defined herein, a "computer-readable storage medium" is not itself a temporary propagating signal. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. As described herein, different types of memory are examples of computer-readable storage media. A non-exhaustive list of more specific examples of computer-readable storage media may include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and the like.

[0077] As defined herein, the term "if" means "when" or "upon" or "in response to" or "in response to," depending on the context. Thus, the phrase "if it is determined" or "if [a stated condition or event] is detected" may be interpreted to mean "upon determination" or "in response to determination" or "upon detection of [a stated condition or event]" or "in response to detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0078] As defined herein, the terms "one embodiment," "an embodiment," "one or more embodiments," or similar language mean that a particular feature, structure, or characteristic described in conjunction with an embodiment is included in at least one embodiment described within the present disclosure. Thus, appearances of the phrases "in one embodiment," "in an embodiment," "in one or more embodiments," and similar language throughout the present disclosure may, but do not necessarily, all refer to the same embodiment. The terms "embodiment" and "arrangement" may be used interchangeably within the present disclosure.

[0079] As defined herein, the term "processor" refers to at least one hardware circuit. The hardware circuit may be configured to execute instructions contained in a program code. The hardware circuit may be an integrated circuit. Examples of processors include, but are not limited to, a central processing unit (CPU), an array processor, a vector processor, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), an application specific integrated circuit (ASIC), a programmable logic circuit, and a controller.

[0080] As defined herein, the term "in response to," and similar language as described above, such as "if," "when," or "upon," refers to the ready response or reaction to an action or event. The response or reaction is automatically performed. Thus, if a second action is performed "in response to" a first action, there is a causal relationship between the occurrence of the first action and the occurrence of the second action. The term "in response to" indicates a causal relationship.

[0081] As defined herein, the term "substantially" means that the characteristic, parameter or value described need not be achieved precisely, but may deviate or vary by an amount that does not eliminate the effect that the characteristic is intended to provide, including, for example, tolerances, measurement errors, measurement accuracy limitations and other factors known to those skilled in the art.

[0082] As defined herein, the term "user" refers to a human being.

[0083] The terms first, second, etc. may be used herein to describe various elements. These elements should not be limited by these terms, because unless otherwise specified or the context clearly indicates otherwise, these terms are only used to distinguish one element from another element.

[0084] The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform various aspects of the present invention. In the present disclosure, the term "program code" may be used interchangeably with the term "computer-readable program instructions". The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, and / or a wireless network). The network may include a copper transmission cable, an optical transmission fiber, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge device including an edge server. The network adapter card or network interface in each computing / processing device receives a computer-readable program instruction from the network, and forwards the computer-readable program instruction to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0085] The computer-readable program instructions for performing the operation of the present invention arrangement described herein can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, or a source code or an object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​and / or process programming languages). Computer-readable program instructions can specify state setting data. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including LAN or WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In some cases, the electronic circuit including, for example, a programmable logic circuit, FPGA or PLA can execute computer-readable program instructions with personalized electronic circuits by utilizing the state information of computer-readable program instructions, so as to perform the various aspects of the invention arrangement described herein.

[0086] This article describes certain aspects of the present invention with reference to the flowcharts and / or block diagrams of methods, devices (systems) and computer program products. It should be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions (e.g., program code).

[0087] These computer-readable program instructions can be provided to a processor of a computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a component for implementing the function / action specified in one or more boxes of the flow chart and / or block diagram. In this way, the processor is operably coupled to the program code instructions to transform the machine of the processor into a special-purpose machine for executing the instructions of the program code. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, and / or other equipment to act in a particular manner, so that the computer-readable storage medium in which the instructions are stored includes a product, which includes instructions for implementing various aspects of the operation specified in one or more boxes of the flow chart and / or block diagram.

[0088] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operations are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0089] The flow chart and block diagram in the accompanying drawings illustrate the architecture, function and operation of the possible implementation of the system, method and computer program product of various aspects arranged according to the present invention.In this respect, each frame in the flow chart or block diagram can represent the module, segment or part of instruction, which includes one or more executable instructions for realizing the specified operation.In some alternative implementations, the operation pointed out in the frame may not occur in the order pointed out in the figure.For example, the two frames shown in succession can be performed substantially at the same time, or the frame can sometimes be performed in reverse order, depending on the function involved.It should also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be realized by a system based on special hardware that performs a specified function or action or performs a combination of special hardware and computer instructions.

[0090] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements found in the following claims are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed.

[0091] The description of the embodiments provided herein is for illustrative purposes and is not intended to be exhaustive or limited to the disclosed forms and examples. The terms used herein are selected to explain the principles of the present invention, practical applications, or technical improvements to the technology found in the market, and / or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. Modifications and variations are clear to those of ordinary skill in the art without departing from the scope and spirit of the described inventive arrangements. Therefore, the scope of these features and implementations should be indicated by reference to the appended claims rather than the foregoing disclosure.

Claims

1. A computer-implemented method comprising: generating a set of silhouette images corresponding to human faces, wherein the generating dissociates correlations between upper and lower portions of the human face depicted by each silhouette image; training a unimodal machine learning model to generate a synthetic image of a human face, wherein the unimodal model is trained using the set of silhouette images; and The output from a unimodal machine learning model is used to train a multimodal rendering network to generate synthetic images of a speaking animated digital human.

2. The method according to claim 1, wherein: The method further comprises: The multimodal rendering network is trained based on minimizing the difference between the synthesized images and the images generated by the multimodal rendering network.

3. The method according to claim 1, wherein: Generating a silhouette image includes: randomly pairing a first silhouette image with a second silhouette image, wherein the first silhouette image and the second silhouette image depict an upper half and a lower half of a human face, respectively; and A silhouette image is generated by merging the first silhouette image and the second silhouette image.

4. The method according to claim 3, wherein: Merging the first silhouette image and the second silhouette image includes: extracting a plurality of sets of key points associated with each of the first silhouette image and the second silhouette image; frontalizing a plurality of sets of key points in each of the first silhouette image and the second silhouette image; generating a frontalized silhouette by adding the frontalized sets of key points of the first silhouette image and the second silhouette image; and The frontalized silhouette is defrontalized to generate a silhouette image.

5. The method according to claim 4, wherein: The key points of positive transformation include: The groups of key points of the first silhouette image and the second silhouette image are multiplied by the inverse of the head pose matrix of the first image and the inverse of the head pose matrix of the second silhouette image, respectively.

6. The method according to claim 4, wherein: De-frontalizing a frontalized silhouette includes: The frontalized silhouette is multiplied by the head pose matrix of the first image or the head pose matrix of the second image.

7. The method according to claim 1, wherein: Generating each synthetic image involves: Perform the corresponding image-to-image transformation of the merged silhouette image.

8. A system comprising: One or more processors are configured to initiate operations, the operations comprising: generating a set of silhouette images corresponding to human faces, wherein the generating dissociates correlations between upper and lower portions of the human face depicted by each silhouette image; training a unimodal machine learning model to generate a synthetic image of a human face, wherein the unimodal model is trained using the set of silhouette images; and The output of the unimodal machine learning model is used to train a multimodal rendering network to generate a synthetic image of a speech-animated digital human, wherein training the multimodal rendering network is based on minimizing the difference between the synthetic image and the image generated by the multimodal rendering network.

9. The system according to claim 8, wherein: The one or more processors are configured to initiate operations further comprising: The multimodal rendering network is trained based on minimizing the difference between the synthesized images and the images generated by the multimodal rendering network.

10. The system according to claim 8, wherein: Generating a silhouette image includes: randomly pairing a first silhouette image with a second silhouette image, wherein the first silhouette image and the second silhouette image depict an upper half and a lower half of a human face, respectively; and A silhouette image is generated by merging the first silhouette image and the second silhouette image.

11. The system according to claim 10, wherein: Merging the first silhouette image and the second silhouette image includes: extracting a plurality of sets of key points associated with each of the first silhouette image and the second silhouette image; frontalizing a plurality of sets of key points in each of the first silhouette image and the second silhouette image; generating a frontalized silhouette by adding the frontalized sets of key points of the first silhouette image and the second silhouette image; and The frontalized silhouette is defrontalized to generate a silhouette image.

12. The system according to claim 11, wherein: The key points of positive transformation include: The groups of key points of the first silhouette image and the second silhouette image are multiplied by the inverse of the head pose matrix of the first image and the inverse of the head pose matrix of the second silhouette image, respectively.

13. The system according to claim 11, wherein: De-frontalizing a frontalized silhouette includes: The frontalized silhouette is multiplied by the head pose matrix of the first image or the head pose matrix of the second image.

14. The system according to claim 8, wherein: Generating each synthetic image involves: Perform the corresponding image-to-image transformation of the merged silhouette image.

15. A computer program product comprising: One or more computer-readable storage media, and program instructions stored collectively on the one or more computer-readable storage media, wherein the program instructions are executable by the one or more processors to initiate operations, the operations comprising: generating a set of silhouette images corresponding to human faces, wherein the generating dissociates correlations between upper and lower portions of the human face depicted by each silhouette image; training a unimodal machine learning model to generate a synthetic image of a human face, wherein the unimodal model is trained using the set of silhouette images; and The output of the unimodal machine learning model is used to train a multimodal rendering network to generate a synthetic image of a speech-animated digital human, wherein training the multimodal rendering network is based on minimizing the difference between the synthetic image and the image generated by the multimodal rendering network.

Images