Generation device, learning device, generation method, and learning method

Abstract

This generation device comprises a storage unit and a control unit. The storage unit stores network data including a multilayer network trained using training voice data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training voice data. The control unit acquires input voice data having the first dimension, acquires input emotion vector data having the second dimension, and inputs the input voice data and the input emotion vector data to the multilayer network to generate a face animation involving a lip sync corresponding to the input voice data.

Description

Generation device, learning device, generation method, and learning method

[0001] The present disclosure relates to a generation device, a learning device, a generation method, and a learning method.

[0002] In the field of 3DCG (3-Dimensional Computer Graphics) animation, a technique has been known for generating facial animation with lip synchronization from an arbitrary voice, making it appear as if a specific person or their avatar is actually speaking.

[0003] In this technology, methods have been studied to make the generated facial animation more realistic, that is, to make it look more natural and natural. For example, Patent Document 1 discloses a technology that uses a neural network to identify facial expression parameters from linguistic information corresponding to speech, and further considers an emotion vector corresponding to speech to identify the expression parameters more accurately.

[0004] Special Publication No. 2022-537011

[0005] However, the above-mentioned conventional technology has a problem in that it requires emotional voice as input to express emotions, making it difficult to generate facial animation with expressions that evoke the emotions intended by the user from, for example, a neutral (unemotional) tone of voice.

[0006] In other words, the above-mentioned conventional technology does not have a mechanism for generating facial animation that matches emotions specified by the user other than voice. On the other hand, if emotions specified by the user other than voice could be reflected in the facial animation, it would be possible to easily generate more realistic facial animation.

[0007] Therefore, the present disclosure proposes a generation device, a learning device, a generation method, and a learning method that can realize the generation of more realistic facial animations that correspond not only to voice but also to emotions arbitrarily specified by the user.

[0008] To solve the above problems, a generation device according to one embodiment of the present disclosure includes a memory unit and a control unit. The memory unit stores network data including a multilayer network trained using training speech data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training speech data. The control unit acquires input speech data having the first dimension and input emotion vector data having the second dimension, and generates facial animation with lip synchronization according to the input speech data by inputting the input speech data and the input emotion vector data to the multilayer network.

[0009] FIG. 1 is a diagram illustrating an overview of a training method and a generation method according to an embodiment of the present disclosure. FIG. 2 is a supplementary explanatory diagram of FIG. 1. FIG. 3 is a block diagram illustrating an example of a configuration of a training device according to an embodiment of the present disclosure. FIG. 4 is a supplementary explanatory diagram of FIG. 3. FIG. 5 is a diagram illustrating an example of 3D face model sequence data for training according to an embodiment of the present disclosure. FIG. 6 is a schematic diagram illustrating an example of a configuration of a training network according to an embodiment of the present disclosure. FIG. 7 is a detailed diagram illustrating an example of a configuration of a training network according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a configuration of a self-attention mechanism of a "Transformer" layer. FIG. 9 is a diagram illustrating a processing flow of a training process performed by a training device. FIG. 10 is a block diagram illustrating a processing flow of an inference process performed by a generation device. FIG. 11 is a diagram illustrating an effect (part 1) when an embodiment of the present disclosure is applied. FIG. 12 is a diagram illustrating an effect (part 2) when an embodiment of the present disclosure is applied. FIG. 13 is a diagram illustrating an effect (part 3) when an embodiment of the present disclosure is applied. FIG. 14 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of a generation device.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] In the following description, a learning device according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment") is assumed to be the learning device 10 shown in Figure 3 and subsequent figures. A generating device according to the present embodiment is assumed to be the generating device 20 shown in Figure 10 and subsequent figures.

[0012] In the following description, the learning method according to this embodiment is assumed to be a learning method for the learning network 11c (see FIG. 3 and subsequent figures) executed by the learning device 10. The generation method according to this embodiment is assumed to be a facial animation generation method executed by the generation device 20.

[0013] Furthermore, various terms used in this embodiment are defined as follows: "3D face model" refers to a 3D model of a face that is learned or generated in this embodiment. The 3D face model is expressed as a polygon mesh, which is a collection of points, edges, and faces that define the shape of the face, or a point cloud.

[0014] Furthermore, "3D face model sequence data" refers to a collection of 3D face models that are continuous in time. The 3D face model sequence data is data in which each 3D face model is assigned a consecutive number, and by displaying the data in order of the consecutive numbers, it creates the illusion of continuous movement of the 3D face model. Note that the 3D face model sequence data may be read as "4D data" in the sense that the dimension of time is added to the 3D face model. Furthermore, the 3D face model sequence may be read as "facial animation." In this embodiment, the 3D face sequence data generated by the generation device 20 is specifically referred to as "facial animation."

[0015] Additionally, the term "neural network" refers to a mathematical model that mimics the neural network of the human brain and is central to deep learning algorithms. Neural networks are also known as artificial neural networks (ANN) or spiking neural networks (SNN). A neural network is composed of multiple structural layers, with the output of each layer serving as input for the next layer. Each layer performs a mathematical transformation on the given input and outputs the result. There are various types of layers, each with its own mathematical transformation. Examples include convolutional layers, transformer layers, and fully connected layers.

[0016] The Transformer layer is also a type of neural network. It is based on the Transformer architecture introduced in the well-known paper "Attention is All You Need."

[0017] This architecture is widely used in natural language processing and other time series data processing tasks, and its distinctive feature is its ability to capture long-range dependencies in the data using attention mechanisms. Its main components include self-attention mechanisms and feedforward neural networks.

[0018] Although the detailed configuration will be described later, the learning network 11c according to this embodiment refers to the entire network including these multiple neural networks in multiple layers. The learning network 11c is an example of a "multilayer network." The learning network 11c can also be considered a neural network.

[0019] In the following, a feature vector representing the characteristics of an emotion will be referred to as an "emotion vector" where appropriate. Similarly, a feature vector representing the characteristics of voice data will be referred to as an "voice vector" where appropriate. Similarly, a feature vector representing the characteristics of a face shape corresponding to an expression in a 3D face model will be referred to as a "face shape vector" where appropriate.

[0020] The present disclosure will be described in the following order: 1. Overview 2. Learning Device 2-1. Example Configuration of Learning Device 2-2. Example Configuration of Learning Network 2-3. Processing Flow of Learning Process 3. Generation Device 3-1. Example Configuration of Generation Device 3-2. Processing Flow of Inference Process 4. Effects 5. Modifications 6. Hardware Configuration 7. Conclusion

[0021] <<1. Overview>> Fig. 1 is a diagram illustrating an overview of a generation method and a learning method according to an embodiment of the present disclosure. Fig. 2 is a diagram supplementing Fig. 1 .

[0022] As shown in Figure 1, the generation method according to this embodiment involves inputting user-specified emotion vector data and speech data of arbitrary utterances into a pre-trained training network 11c, and using the training network 11c to generate facial animation with expressions that express at least the emotion corresponding to the emotion vector data specified by the user. Note that the user-specified emotion vector data input here is an example of "input emotion vector data," and the speech data of the arbitrary utterance is an example of "input speech data."

[0023] Specifically, in the generation method according to this embodiment, the user can specify emotion vector data that represents an emotion label. As shown in FIG. 1, the emotion label can specify at least one of eight emotions: neutral, happiness, disgust, surprise, anger, contempt, fear, and sadness, along with an emotion level indicating the degree of each emotion, expressed as a value between 0 and 1. In other words, the emotion vector data has eight dimensions (equivalent to an example of a "second dimension").

[0024] Meanwhile, speech data is input to the training network 11c as a 16 kHz monaural, one-dimensional (equivalent to an example of the "first dimension") array. The emotion vector data and speech data are then "standardized" in the training network 11c so that they become feature vectors with the same dimensions. In this embodiment, the emotion vector data and speech data are converted so that they become the same 128-dimensional (equivalent to an example of the "third dimension") feature vector. Standardization in this disclosure may be considered a type of normalization and may be referred to as "z-score normalization."

[0025] Then, in the learning network 11c, a "voice-emotion mixing" process is performed to mix the standardized voice vector and emotion vector, and a "face shape vector calculation" process is performed based on the mixed feature vector.Facial animation is then generated based on the calculated face shape vector.

[0026] In other words, according to the generation method of this embodiment, if any voice data and emotion vector data based on emotion labels specifying at least one emotion are input, it is possible to output high-quality 3D face model sequence data as facial animation. For example, as shown in Figure 2, if a user specifies different emotions, such as "neutral," "happiness," and "surprise," it is possible to output facial animation accompanied by lip-synching with expressions corresponding to the specified emotions at the same positions (here, the positions of the underlined "mu") in the input voice data.

[0027] To achieve this, the training network 11c is trained in advance by the training method according to the present embodiment. As shown in Fig. 1, in the training method according to the present embodiment, the training network 11c is trained so as to be capable of generating voice-driven facial animation using a training dataset 11a including previously acquired 3D face model sequence data corresponding to each of the above-mentioned eight types of emotions (shown here as "Emotions 1 to 8") and voice data synchronized therewith.

[0028] The 3D face model sequence data included in the training data set 11a is an example of "training 3D face model sequence data," and the voice data is an example of "training voice data." Furthermore, the eight types of emotions corresponding to these are each an example of "training emotion vector data."

[0029] This embodiment is characterized by the configuration of the learning network 11c, which is a multi-layer network having neural networks for an emotion encoder, an audio encoder, a mixture layer, and a 3D face model decoder.

[0030] The emotion encoder estimates emotion vectors from emotion labels associated with the training data. The audio encoder estimates speech vectors from the speech data in the training data. The mixing layer mixes two feature vectors, the estimated emotion vector and the speech vector. The 3D face model decoder estimates and generates 3D face model sequence data from the mixed feature vector.

[0031] In addition, in the learning method according to this embodiment, the 3D face model sequence data M estimated by the learning network 11c during learning is ｉ is compared with the 3D face model sequence data of the training data (i.e., ground truth). Then, the parameters of the training network 11c are updated and optimized so as to minimize the difference in the comparison. The configuration of the training network 11c will be described in detail later with reference to FIGS. 6 to 8.

[0032] Hereinafter, examples of the configuration of the learning device 10 to which the learning method according to this embodiment is applied and the generation device 20 to which the generation method according to this embodiment is applied will be described in more detail.

[0033] <<2. Learning Device>> <2-1. Configuration Example of Learning Device> Fig. 3 is a block diagram showing a configuration example of a learning device 10 according to an embodiment of the present disclosure. Fig. 4 is a supplementary explanatory diagram for Fig. 3. Note that Figs. 3 and 4 and Figs. 10 and 11 shown later depict only components necessary for explaining the features of the embodiment of the present disclosure, and descriptions of general components are omitted.

[0034] In other words, the components shown in Figures 3, 4, 10, and 11 are conceptual functional components and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0035] In addition, in the description using FIG. 3, FIG. 4, FIG. 10, or FIG. 11, the description of components that have already been described may be simplified or omitted.

[0036] As shown in FIG. 3 , the learning device 10 includes a storage unit 11 and a control unit 12. The learning device 10 is also connected to a learning data generation device 3. The learning data generation device 3 is a device that generates the above-mentioned learning dataset. The learning device 10 is connected to the learning data generation device 3 so as to be able to communicate directly with the learning data generation device 3 via wired communication or wireless communication. Alternatively, the learning device 10 is indirectly connected to the learning data generation device 3 via a recording medium or the like.

[0037] The storage unit 11 is realized by a storage device such as a random access memory (RAM), a read only memory (ROM), a flash memory, a hard disk drive (HDD), etc. In the example of Fig. 3, the storage unit 11 stores a training dataset 11a, network configuration information 11b, and a training network 11c.

[0038] The training data set 11a is acquired from the training data generation device 3 by the acquisition unit 12a (described later) and stored. The network configuration information 11b is definition information regarding the configuration of the training network 11c trained by the training unit 12b (described later). The training network 11c is a neural network trained by the training unit 12b so as to be capable of generating facial animations driven by voice. The network configuration information 11b and the training network 11c correspond to examples of "network data."

[0039] The control unit 12 corresponds to a so-called processor or controller. The control unit 12 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), etc. The control unit 12 executes a program according to this embodiment (not shown) stored in the storage unit 11, using RAM as a work area. The control unit 12 can also be realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0040] The control unit 12 has an acquisition unit 12a and a learning unit 12b, and realizes or executes the functions and actions of information processing described below.

[0041] The acquisition unit 12a acquires a training dataset from the training data generation device 3 and stores it as a training dataset 11a in the storage unit 11. The learning unit 12b executes a learning process to train the training network 11c based on the network configuration information 11b, the 3D face data sequence data, the voice data, and the emotion vector data included in the training dataset 11a.

[0042] Specifically, as shown in FIG. 4, the learning data generating device 3 includes a clock generator 3a, a sound recording device 3b, a face photographing device 3c, and a 3D scan model generating unit 3d.

[0043] The clock generator 3 a provides a unified clock signal for synchronization to each of the sound recording device 3 b and the face photographing device 3 c. The sound recording device 3 b records audio data to be used as training data. In this embodiment, the audio data is recorded at a sampling rate of 16 kHz and is monaural 16-bit linear PCM (Pulse Code Modulation) data saved in a WAV (RIFF Waveform Audio Format) file format.

[0044] The face photographing device 3c photographs source images of 3D face model sequence data that will be used as learning data. The audio data and face images are synchronized by the clock generator 3a.

[0045] The 3D scan model generation unit 3 d generates 3D face model sequence data, which serves as learning data, from the source video captured by the face photographing device 3 c. The 3D scan model generation unit 3 d is realized, for example, by application software having a 3D scan function.

[0046] In this embodiment, when generating a training dataset, a predetermined scenario covering various phonemes and their combinations is prepared. Then, the actor who is the subject of the source video is asked to read the scenario with facial expressions and tones corresponding to each of the eight emotions described above. The recording device 3b records the audio data at this time. The face photographing device 3c photographs the actor's face at this time. The 3D scan model generation unit 3d then 3D scans the video photographed by the face photographing device 3c and generates 3D face model sequence data. An example of the 3D face model sequence data generated in this manner is shown in FIG. 5.

[0047] 5 is a diagram illustrating an example of 3D face model sequence data for training according to an embodiment of the present disclosure. Each face image enclosed in a rectangle in FIG. 5 represents one frame of a source video corresponding to one of the seven emotions (happiness, sadness, anger, contempt, surprise, fear, and disgust) of the eight emotions described above, excluding neutral. A 3D face model corresponding to each frame is depicted on the right side of each face image.

[0048] Returning to the explanation of Fig. 4, the learning data generation device 3 generates a learning data set including such a set of 3D face model sequence data and voice data synchronized therewith for each of eight types of emotions 1 to 8.

[0049] The learning device 10 acquires this from the learning data generation device 3 and stores it as a learning dataset 11a. Then, the learning unit 12b generates a learning network 11c based on the configuration defined in the network configuration information 11b. The learning unit 12b also executes a learning process using the learning dataset 11a to update the parameters of the learning network 11c and optimize the learning network 11c.

[0050] <2-2. Example Configuration of Learning Network> As described above, this embodiment is characterized by the configuration of the learning network 11c. Next, an example configuration of this learning network 11c will be described with reference to FIGS. 6 to 8. FIG. 6 is a schematic diagram illustrating an example configuration of the learning network 11c according to an embodiment of the present disclosure. FIG. 7 is a detailed diagram illustrating an example configuration of the learning network 11c according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example configuration of a self-attention mechanism in the "Transformer" layer.

[0051] The configuration of the learning network 11c shown in FIGS. 6 to 8 is defined in the network configuration information 11b.

[0052] As shown in Figure 6, the training network 11c is a multi-layer network including neural networks: an emotion encoder L1, an audio encoder L2, a mixing layer L3, and a 3D face model decoder L4. The emotion encoder L1 estimates emotion vectors from emotion labels associated with the training data. The audio encoder L2 estimates speech vectors from the speech data of the training data.

[0053] The blending layer L3 blends two feature vectors, the estimated emotion vector and the speech vector, and the 3D face model decoder L4 generates 3D face model sequence data from the blended feature vectors.

[0054] To be more specific, the emotion encoder L1 has a "Linear Projection" layer (fully connected layer). As mentioned above, the user can specify at least one of the eight-dimensional emotion vectors as an emotion label, and the element value of each dimension, i.e., the emotional degree, can be set to a value between 0 and 1. The emotion encoder L1 converts this eight-dimensional emotion vector specified by the user into a 128-dimensional emotion vector f using the "Linear Projection" layer. Ｅ and the emotion vector f Ｅ The number of dimensions of the speech vector is set to be the same as the number of dimensions of the speech vector.

[0055] Next, the audio encoder L2 calculates the speech vectors using a neural network called Wav2Vec2, which has been trained in advance using a large amount of speech data. As shown in Fig. 7, Wav2Vec2 has 12 transformer layers consisting of a TCN (Temporal Convolution) layer, a "Linear Interpolation" layer, a "Multi-head Self Attention" layer, and a "Feed Forward" layer, and a "Linear Projection" layer. The TCN layer is a so-called one-dimensional convolutional neural network.

[0056] The audio data input to the audio encoder L2 is a 16 kHz mono, one-dimensional array. The audio encoder L2 divides the input audio data into 25-millisecond segments, each 20 milliseconds long. The audio encoder L2 then passes each segment through a TCN layer, a Linear Interpolation layer, 12 transformer layers, and a Linear Projection layer, generating a 128-dimensional audio vector a ｉ In this embodiment, the audio encoder L2 outputs the voice vector a ｉ is downsampled to 30 Hz.

[0057] Next, the mixed layer L3 has an "Audio-Emotion Entanglement" layer L31 as shown in Fig. 6. The "Audio-Emotion Entanglement" layer L31 has an "Audio Standardization" layer L311 and an "Emotion Entanglement" layer L312 as shown in Fig. 7.

[0058] The “Audio Standardization” layer L311 reduces the internal emotions contained in the audio data by using the audio vector a ｉ The "Audio Standardization" layer L311 standardizes the standardized audio vector a ｉ Ｓ The mean μ in formula (a) is calculated by the following formula (b), and the standard deviation σ is calculated by the following formula (c).

[0059]

[0060]

[0061]

[0062] The “Emotion Entanglement” layer L312 calculates the standardized speech vector a ｉ Ｓ For emotion vector f Ｅ is added as an offset, and the speech-emotion mixed vector a ｉ Ｅ As a result, the emotion vector specified by the user is mixed with the voice vector in the latent space (feature space) of the learning network 11c. ｉ Ｅ corresponds to an example of the "first mixed vector."

[0063]

[0064] Next, the 3D face model decoder L4 has a "Transformer" layer L41, an "Output-Emotion Entanglement" layer L42, and a "Linear Projection" layer, as shown in FIG.

[0065] Specifically, as shown in FIG. 7, the "Transformer" layer L41 includes a "Periodic Positional Encoding (PPE)" layer, a "Multi-head Self Attention" layer L411, an "Encoder-Decoder Multi-head Attention" layer L412, and a "Feed Forward" layer.

[0066] The speech-emotion mixed vector a obtained from the mixing layer L3 ｉ Ｅ , i∈{1, 2, 3...} is a sequence of 128-dimensional feature vectors in a time series at 30 Hz. Since facial expressions in emotional expressions are not always fixed but depend on the content of the voice and their position in the sequence, the 3D face model decoder L4 uses a "Transformer" layer L41.

[0067] In addition, the speech-emotion mixed vector a ｉ Ｅ does not contain position information, the "Transformer" layer L41 uses the "Periodic Positional Encoding" layer to generate the speech-emotion mixed vector a ｉ Ｅ Then, the self-attention mechanism "Multi-head Self Attention" layer L411 and "Encoder-Decoder Multi-head Attention" layer L412 generate the speech-emotion mixed vector a with location information. ｉ Ｅ and emotion vector f Ｅ and how to calculate the face shape vector f ｉ Calculate how to convert to.

[0068] As shown in Figure 8, the "Multi-head Self Attention" layer L411 and the "Encoder-Decoder Multi-head Attention" layer L412 receive two time-series sequences as input, transform each of them using learnable parameters, and calculate a sequence of mixed feature vectors by multiplying each element of the first sequence by each element of the second sequence.

[0069] In the example of Figure 8, the first sequence is a speech-emotion mixed vector a ｉ Ｅ and the second sequence is a fixed emotion vector f Ｅ The transformation matrices are Q (Query) and K (Key), and the output features are the face shape vector f ｉ Q and K are parameters estimated from the learning data.

[0070] Returning to the explanation of Fig. 7, the "Output-Emotion Entanglement" layer L42 includes an "Emotion Entanglement" layer L421. The "Emotion Entanglement" layer L421 calculates the facial shape vector f ｉ For emotion vector f Ｅ is added as an offset, and the facial shape-emotion mixed vector f ｉ Ｍ As a result, the emotion vector designated by the user is mixed with the facial shape vector in the latent space of the learning network 11c. Note that the facial shape-emotion mixed vector f ｉ Ｍ corresponds to an example of a "second mixed vector."

[0071]

[0072] Finally, the 3D face model decoder L4 generates the face shape-emotion mixture vector f ｉ Ｍ 3D face model sequence data M ｉThe parameters of the neural networks of the emotion encoder L1, audio encoder L2, "Emotion Entanglement" layer L312, "Multi-head Self Attention" layer L411, "Encoder-Decoder Multi-head Attention" layer L412, "Emotion Entanglement" layer L421, and the "Linear Projection" layer of the 3D face model decoder L4 are estimated from training data.

[0073] The 3D face model sequence data M estimated by the learning network 11c ｉ is compared with the 3D face model sequence data of the learning data (i.e., the above-mentioned ground truth), and a parameter search is performed so as to minimize the difference. For this parameter search, for example, a commonly used optimization algorithm such as the Gauss-Newton method or an extended version of it, such as Adam, can be used.

[0074] 2-3. Processing Flow of Learning Processing> Next, a description will be given of the processing flow of the learning processing executed by the learning device 10. FIG.

[0075] First, the learning unit 12b generates a learning network 11c based on the network configuration information 11b in the storage unit 11, and inputs the 3D face model sequence data from the learning dataset 11a to the learning network 11c, as shown in FIG. 9 (step S101).

[0076] The learning unit 12b also inputs emotion vector data associated with the 3D face model sequence data from the training data set 11a to the training network 11c (step S102), and inputs audio data synchronized with the 3D face model sequence data from the training data set 11a to the training network 11c (step S103).

[0077] Then, the learning unit 12b generates an emotion vector f by an emotion encoder L1 based on the emotion vector data. Ｅ(Step S104). The learning unit 12b also calculates the voice vector a by using the audio encoder L2 based on the voice data. ｉ is calculated (step S105).

[0078] Next, the learning unit 12b performs standardization processing using the above formula (a) in the "Audio Standardization" layer L311 (step S106), and obtains the audio vector a ｉ Ｓ Furthermore, the learning unit 12b calculates the speech vector a by using the above formula (d) in the "Emotion Entanglement" layer L312. ｉ Ｓ and emotion vector f Ｅ The speech-emotion mixing process is performed to mix the speech-emotion mixed vector a ｉ Ｅ is calculated (step S107).

[0079] Then, the learning unit 12b calculates the speech-emotion mixed vector a ｉ Ｅ and emotion vector f Ｅ Based on this, the "Transformer" layer L41 generates a face shape vector f ｉ (Step S108). In addition, the learning unit 12b calculates the facial shape vector f ｉ and emotion vector f Ｅ A facial shape-emotion mixing process is performed to mix the facial shape-emotion mixed vector f ｉ Ｍ is calculated (step S109).

[0080] Then, the learning unit 12b outputs the facial shape-emotion mixed vector f ｉ Ｍ A 3D face shape is calculated from the calculated 3D face shape (step S110), and 3D face model sequence data is generated and output based on the calculated 3D face shape (step S111).

[0081] Then, the learning unit 12b compares the 3D face model sequence data of the learning data set 11a with the 3D face model sequence data output in step S111, and updates the parameters of the learning network 11c based on the difference (step S112).

[0082] <<3. Generation Device>> <3-1. Configuration Example of Generation Device> Next, a configuration example of the generation device 20 according to the present embodiment will be described. Fig. 10 is a block diagram showing a configuration example of the generation device 20 according to an embodiment of the present disclosure. Fig. 11 is a supplementary explanatory diagram of Fig. 10.

[0083] 10 , the generation device 20 includes a storage unit 21 and a control unit 22. The generation device 20 is also connected to the learning device 10. The generation device 20 is connected to the learning device 10 so as to be able to communicate directly with the learning device 10 via wired communication or wireless communication. Alternatively, the generation device 20 is indirectly connected to the learning device 10 via a recording medium or the like. The generation device 20 may be integrated with the learning device 10 to form an integrated structure.

[0084] The generating device 20 is also connected to a sound recording device 5, an input device 7, and a display device 9. The sound recording device 5 records any audio data that triggers the generation of facial animation. The input device 7 allows the user to specify emotion vector data corresponding to the emotion that the user wants to express in the facial expression of the generated facial animation. The input device 7 is realized by a keyboard, a mouse, other pointing device, etc. The display device 9 displays the generated facial animation. The display device 9 is realized by a display, etc.

[0085] The storage unit 21 is realized by a storage device such as a RAM, a ROM, a flash memory, a HDD, etc. In the example of Fig. 10, the storage unit 21 stores network configuration information 11b, a training network 11c, and a facial animation result 21a.

[0086] The network configuration information 11b and the learning network 11c are acquired from the learning device 10 by an acquisition unit 22a (described later) and stored. The facial animation result 21a stores facial animation generated by an inference unit 22b (described later).

[0087] The control unit 22 corresponds to a so-called processor or controller. The control unit 22 is realized by, for example, a CPU, an MPU, a GPU, etc. The control unit 22 executes a program according to this embodiment (not shown) stored in the storage unit 21, using RAM as a work area. The control unit 22 can also be realized by, for example, an integrated circuit such as an ASIC or an FPGA.

[0088] The control unit 22 has an acquisition unit 22a, an inference unit 22b, and an output control unit 22c, and realizes or executes the functions and actions of information processing described below.

[0089] The acquisition unit 22a acquires the network configuration information 11b from the learning device 10 and stores it in the storage unit 21. The acquisition unit 22a also acquires the learned learning network 11c from the learning device 10 and stores it in the storage unit 21.

[0090] The acquisition unit 22a also acquires any audio data that will be used as a trigger for generating facial animation from the recording device 5. The acquisition unit 22a also acquires emotion vector data designated by the user via the input device 7.

[0091] The inference unit 22b executes an inference process to generate a facial animation using the voice data, emotion vector beta, and learning network 11c acquired by the acquisition unit 22a.

[0092] 11, the inference unit 22b executes an inference process to generate facial animation using the voice data acquired by the acquisition unit 22a from the recording device 5, the emotion vector data from the input device 7, and the trained learning network 11c. The inference unit 22b outputs a facial animation result 21a as a result of the executed inference process. The facial animation result 21a is displayed on the display device 9.

[0093] Returning to the description of Fig. 10, the output control unit 22c controls the output to display the facial animation result 21a on the display device 9.

[0094] 3-2. Processing Flow of Inference Processing> Next, a description will be given of the processing flow of the inference processing executed by the generating device 20. FIG.

[0095] The processing flow of the inference process is roughly the same as the processing flow of the learning process shown in Fig. 9, except that steps S101 and S112 are removed. Steps S202 to S211 in Fig. 12 correspond to steps S102 to S111 in Fig. 9.

[0096] First, the inference unit 22b reads the trained training network 11c from the storage unit 21, and inputs any emotion vector data designated by the user via the input device 7 to the training network 11c (step S202), as shown in Fig. 12. The inference unit 22b also inputs any voice data from the recording device 5 to the training network 11c (step S203).

[0097] Then, the inference unit 22b calculates an emotion vector f by an emotion encoder L1 based on the emotion vector data. Ｅ (Step S204). The inference unit 22b also calculates the voice vector a by the audio encoder L2 based on the voice data. ｉ is calculated (step S205).

[0098] Next, the inference unit 22b performs standardization processing using the above formula (a) in the "Audio Standardization" layer L311 (step S206), and obtains the audio vector a ｉ Ｓ Furthermore, the inference unit 22b calculates the speech vector a ｉ Ｓ and emotion vector f Ｅ The speech-emotion mixing process is performed to mix the speech-emotion mixed vector a ｉ Ｅ is calculated (step S207).

[0099] Then, the inference unit 22b calculates the voice-emotion mixed vector a ｉ Ｅ and emotion vector f Ｅ Based on this, the "Transformer" layer L41 generates a face shape vector f ｉ (Step S208). In addition, the inference unit 22b calculates the face shape vector f ｉ and emotion vector f Ｅ A facial shape-emotion mixing process is performed to mix the facial shape-emotion mixed vector f ｉ Ｍ is calculated (step S209).

[0100] Then, the inference unit 22b outputs the facial shape-emotion mixed vector f through the "Linear Projection" layer of the 3D face model decoder L4. ｉ Ｍ A 3D face shape is calculated from the calculated 3D face shape (step S210), and a face animation result 21a, which is 3D face model sequence data based on the calculated 3D face shape, is generated and output (step S211).

[0101] <<4. Effects>> Next, effects when the above-described embodiments of the present disclosure are applied will be described with reference to FIGS. 13 to 15. FIG. 13 is a diagram (part 1) showing the effects when the embodiments of the present disclosure are applied. FIG. 14 is a diagram (part 2) showing the effects when the embodiments of the present disclosure are applied. FIG. 15 is a diagram (part 3) showing the effects when the embodiments of the present disclosure are applied.

[0102] 13, for comparison, the left side shows the case where "this embodiment is not applied," and the right side shows the case where "this embodiment is applied." In each case, as shown in the top row, it is assumed that the voice that triggers the generation of facial animation includes the emotion of "surprise," and the emotion vector data specified by the user is "neutral."

[0103] When this embodiment is not applied, the facial animation generated will contain a lot of "surprised" expressions, whereas when this embodiment is applied, the facial animation generated will contain a lot of "unemotional" expressions.

[0104] In contrast, as shown in the lower part, the voice that triggers the generation of the facial animation is "unemotional," while the emotion vector data designated by the user is "surprise."

[0105] When this embodiment is not applied, the facial animation generated will contain many expressions of "neutral," whereas when this embodiment is applied, the facial animation generated will contain many expressions of "surprise." In other words, according to this embodiment, the emotional vector data specified by the user can be used to easily reflect the emotional expression intended by the user in the facial animation that is generated.

[0106] 14 shows the effect when the user specifies multiple emotion vector data. As shown in FIG. 14, it is assumed that the user simultaneously specifies the labels "happiness" and "surprise" as emotion vector data.

[0107] In this case, a facial animation that simultaneously includes expressions of "happiness" and "surprise" can be generated for one piece of voice data. In other words, according to this embodiment, the user can easily generate facial animation that simultaneously includes multiple emotions by specifying multiple pieces of emotion vector data.

[0108] 14 shows an example in which the emotional values ​​of "happiness" and "surprise" are each set to 1, but as mentioned above, the user can individually set values ​​between 0 and 1 for each emotional level. Regarding this specification of emotional levels, an example in which the emotional level of "surprise" is changed is shown in FIG.

[0109] As shown in Figure 15, suppose that the user specifies three examples of the emotional level of "surprise": "0," "0.5," and "1." In this case, the facial animation generated includes more of the "surprised" expression when the emotional level is "0.5" than when it is "0." Also, the facial animation generated includes more of the "surprised" expression when the emotional level is "1" than when it is "0.5."

[0110] In other words, according to this embodiment, the user can easily reflect the degree of emotion in the facial expression of the facial animation by changing the emotional intensity of each emotion between 0 and 1. Furthermore, by individually adjusting the emotional intensity of each emotion specifying the multiple emotional vector data shown in Fig. 14, the facial expression of the generated facial animation can be easily and finely adjusted.

[0111] <<5. Modifications>> Furthermore, several modifications can be made to the above-described embodiments of the present disclosure. For example, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0112] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0113] For example, as described above, the generation device 20 may be integrated with the learning device 10 to form an integrated configuration. Furthermore, for example, the training data generation device 3 may be integrated with the learning device 10 to form an integrated configuration.

[0114] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any contradiction in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0115] <<6. Hardware Configuration>> The learning device 10 and the generation device 20 according to the above-described embodiments of the present disclosure are realized by, for example, a computer 1000 configured as shown in FIG. 16 . The generation device 20 will be described as an example. FIG. 16 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the generation device 20. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected via a bus 1050.

[0116] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0117] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0118] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the programs according to this embodiment.

[0119] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0120] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0121] For example, when computer 1000 functions as generation device 20, CPU 1100 of computer 1000 executes a program loaded onto RAM 1200 to realize the functions of control unit 22. Furthermore, secondary storage device 1400 stores programs according to the present disclosure and data in storage unit 21. Note that CPU 1100 reads and executes program data 1450 from secondary storage device 1400, but as another example, these programs may be obtained from another device via external network 1550.

[0122] Furthermore, functions realized by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs, conventional circuits, and / or combinations thereof, programmed to realize the described functions, without being limited to the hardware configuration described with reference to FIG. 16 . A processor includes transistors and other circuits and is considered to be a circuit or processing circuitry. A processor may be a programmed processor that executes a program stored in a memory. The processor, for example, realizes the functions of the control unit 22 or the control unit 12 as a programmed processor.

[0123] In this specification, a circuit arrangement, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0124] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or processor.

[0125] <<7. Conclusion>> As described above, according to one embodiment of the present disclosure, the generation device 20 includes a memory unit 21 and a control unit 22. The memory unit 21 stores network data including training voice data having one dimension (corresponding to an example of a "first dimension"), training emotion vector data representing specific emotion labels having eight dimensions (corresponding to an example of a "second dimension"), and a training network 11c (corresponding to an example of a "multilayer network") trained using training 3D face model sequence data operable in synchronization with the training voice data. The control unit 22 acquires input voice data having one dimension and input emotion vector data having eight dimensions, and inputs the input voice data and the input emotion vector data to the training network 11c to generate facial animation with lip synchronization corresponding to the input voice data. This makes it possible to generate more realistic facial animation that corresponds not only to voice but also to emotions arbitrarily specified by the user.

[0126] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0127] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0128] The present technology may also be configured as follows: (1) A generation device comprising: a storage unit that stores network data including a multilayer network trained using training audio data having a first dimension, training emotion vector data representing a specific emotion label having a second dimension, and training 3D face model sequence data operable in synchronization with the training audio data; and a control unit that acquires input audio data having the first dimension, acquires input emotion vector data having the second dimension, and generates facial animation with lip synchronization according to the input audio data by inputting the input audio data and the input emotion vector data to the multilayer network. (2) The generation device described in (1), wherein the multilayer network includes neural networks for an emotion encoder, an audio encoder, a mixing layer, and a 3D face model decoder. (3) The generation device described in (2), wherein the emotion encoder converts the input emotion vector data into an emotion vector having a third dimension higher than the second dimension. (4) The generation device according to (3), wherein the audio encoder converts the input audio data into an audio vector having the third dimension by dividing the input audio data into a plurality of segments and passing each segment through a convolutional layer, a transformer layer, and a fully connected layer. (5) The generation device according to (4), wherein the mixing layer standardizes the audio vector. (6) The generation device according to (5), wherein the mixing layer standardizes the audio vector by subtracting the mean value of the audio vector from the audio vector and dividing the result by the standard deviation of the audio vector. (7) The generation device according to (5) or (6), wherein the mixing layer mixes the audio vector and the emotion vector by adding the emotion vector as an offset to the standardized audio vector. (8) The generation device according to (7), wherein the 3D face model decoder has a transformer layer configured to output a first mixed vector obtained by mixing the audio vector and the emotion vector in the mixing layer and a face shape vector by receiving the emotion vector as input.(9) The generation device according to (8), wherein the 3D face model decoder mixes the facial shape vector and the emotion vector by adding the emotion vector as an offset to the facial shape vector output from the Transformer layer. (10) The generation device according to (9), wherein the 3D face model decoder estimates the facial animation by passing a second mixed vector, which is a mixture of the facial shape vector and the emotion vector, through a fully connected layer. (11) The generation device according to any one of (1) to (10), wherein the second dimension is eight dimensions, and the input emotion vector data can specify at least one of eight emotions: neutral, happiness, disgust, surprise, anger, contempt, fear, and sadness. (12) The generation device according to (11), wherein the input emotion vector data can further individually specify an emotionality level indicating the degree of each specified emotion. (13) A generation method including: storing network data including a multilayer network trained using training speech data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training speech data; acquiring input speech data having the first dimension; acquiring input emotion vector data having the second dimension; and generating facial animation with lip synchronization according to the input speech data by inputting the input speech data and the input emotion vector data to the multilayer network.(14) A learning device comprising: acquiring training voice data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training voice data; converting the training voice data and the training emotion vector data into a voice vector and an emotion vector, respectively, having a third dimension; estimating 3D face model sequence data in which the emotion vector is mixed with the voice vector by inputting the voice vector and the emotion vector to a neural network configured to generate facial animation by voice activation; and a control unit that updates parameters of the neural network based on a comparison between the estimated 3D face model sequence data and the training 3D face model sequence data. (15) The learning device according to (14), wherein the control unit updates parameters of the neural network so as to minimize a difference in the comparison between the estimated 3D face model sequence data and the training 3D face model sequence data. (16) A training method including: acquiring training voice data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training voice data; converting the training voice data and the training emotion vector data into a voice vector and an emotion vector, respectively, having a third dimension; estimating 3D face model sequence data in which the emotion vector is mixed with the voice vector by inputting the voice vector and the emotion vector into a neural network configured to generate facial animation by voice activation; and updating parameters of the neural network based on a comparison between the estimated 3D face model sequence data and the training 3D face model sequence data.

[0129] 3 Learning data generation device 3a Clock generator 3b Sound recording device 3c Face photographing device 3d 3D scan model generation unit 5 Sound recording device 7 Input device 9 Display device 10 Learning device 11 Memory unit 11a Learning data set 11b Network configuration information 11c Learning network 12 Control unit 12a Acquisition unit 12b Learning unit 20 Generation device 21 Memory unit 21a Facial animation result 22 Control unit 22a Acquisition unit 22b Inference unit 22c Output control unit L1 Emotion encoder L2 Audio encoder L3 Mixing layer L4 3D face model decoder

Claims

1. A generation device comprising: a memory unit that stores network data including a multilayer network trained using training speech data having a first dimension, training emotion vector data having a second dimension representing a specific emotion label, and training 3D face model sequence data that can operate in synchronization with the training speech data; and a control unit that acquires input speech data having the first dimension, acquires input emotion vector data having the second dimension, and generates facial animation with lip synchronization in accordance with the input speech data by inputting the input speech data and the input emotion vector data into the multilayer network.

2. The generating device of claim 1, wherein the multilayer network includes neural networks for an emotion encoder, an audio encoder, a mixing layer, and a 3D face model decoder.

3. The generating device according to claim 2, wherein the emotion encoder converts the input emotion vector data into an emotion vector having a third dimension that is higher than the second dimension.

4. The generation device according to claim 3, wherein the audio encoder converts the input audio data into an audio vector having the third dimension by dividing the input audio data into a plurality of segments and passing each segment through a convolutional layer, a transformer layer, and a fully connected layer, respectively.

5. The generation device according to claim 4, wherein the mixing layer normalizes the speech vectors.

6. The generation device according to claim 5, wherein the mixing layer performs the normalization by subtracting the mean value of the speech vector from the speech vector and dividing the result by the standard deviation of the speech vector.

7. The generation device according to claim 5, wherein the mixing layer mixes the voice vector and the emotion vector by adding the emotion vector as an offset to the voice vector after the standardization.

8. The generation device according to claim 7, wherein the 3D face model decoder has a transformer layer configured to receive a first mixed vector obtained by mixing the voice vector and the emotion vector in the mixing layer, and to output a face shape vector in response to the emotion vector.

9. The generation device of claim 8, wherein the 3D face model decoder blends the face shape vector and the emotion vector by adding the emotion vector as an offset to the face shape vector output from the transformer layer.

10. The generation device according to claim 9, wherein the 3D face model decoder estimates the facial animation by passing a second mixed vector, which is a mixture of the face shape vector and the emotion vector, through a fully connected layer.

11. The generation device of claim 1, wherein the second dimension is eight dimensions, and the input emotion vector data can specify at least one of eight emotions: neutral, happiness, disgust, surprise, anger, contempt, fear, and sadness.

12. The generation device according to claim 11, wherein the input emotion vector data further allows for individual designation of emotion levels indicating the degree of each designated emotion.

13. A generation method comprising: storing network data including a multilayer network trained using training speech data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training speech data; acquiring input speech data having the first dimension; acquiring input emotion vector data having the second dimension; and generating facial animation with lip synchronization according to the input speech data by inputting the input speech data and the input emotion vector data into the multilayer network.

14. A learning device comprising: a control unit that acquires training voice data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training voice data; converts the training voice data and the training emotion vector data into a voice vector and an emotion vector, respectively, having a third dimension; estimates 3D face model sequence data in which the emotion vector is mixed with the voice vector by inputting the voice vector and the emotion vector into a neural network configured to generate facial animation driven by voice; and updates parameters of the neural network based on a comparison between the estimated 3D face model sequence data and the training 3D face model sequence data.

15. The learning device according to claim 14, wherein the control unit updates the parameters of the neural network so as to minimize a difference in a comparison between the estimated 3D face model sequence data and the training 3D face model sequence data.

16. A training method comprising: obtaining training speech data having a first dimension, training emotion vector data representing a specific emotion label and having a second dimension, and training 3D face model sequence data operable in synchronization with the training speech data; converting the training speech data and the training emotion vector data into speech vectors and emotion vectors, respectively, having a third dimension; estimating 3D face model sequence data in which the emotion vectors are mixed with the speech vectors by inputting the speech vectors and the emotion vectors into a neural network configured to generate speech-driven facial animation; and updating parameters of the neural network based on a comparison between the estimated 3D face model sequence data and the training 3D face model sequence data.

Images