Apparatus and method for generating speech synthesis image

Abstract

An apparatus and a method for generating a speech synthesis image are provided. The apparatus for generating a speech synthesis image, according to one embodiment, comprises: a head movement prediction module, which estimates three-dimensional geometric transformation of a head movement of a person between a first image and a second image of the person, calculates, on the basis of a pinhole camera model, a plurality of transformation matrices for applying the three-dimensional geometric transformation at a plurality of preset depths, and predicts the head movement of the person between the first image and the second image by using the plurality of transformation matrices; a facial expression movement prediction module for predicting a facial expression movement of the person between the first image and the second image on the basis of a pre-trained facial expression movement prediction model; and a movement generation module for transforming the first image on the basis of the predicted head movement and facial expression movement.

Description

Device and method for generating speech synthesis images

[0001] The disclosed embodiments relate to a technology for generating synthetic speech images.

[0002] When the difference in the position, posture, and facial expression of a specific person in a pair of images (or video frames) containing the person is defined as "movement," the person's movement can be estimated by separating it from the person's identity, and tasks such as image transformation and video synthesis can be performed.

[0003] In particular, when performing such tasks, if the movement of the person is controlled by separating it into the movement of the entire head (global movement) and the movement of facial expressions (local movement), the degree of freedom of the synthesis method can be increased and more natural synthesis results can be achieved.

[0004] This task can be accomplished using a machine learning model. The training process for this model is as follows: First, one of a pair of images containing the same person is designated as the source and the other as the driving image. Next, the machine learning model estimates the motion between the source and driving images and transforms the source image into the driving image based on the estimated motion. The transformed image is then compared to the actual driving image, thereby completing the training process.

[0005] Motion between source and driving can be expressed as a 2-dimensional geometric transformation, including translation, rotation, scaling, and tilting. Conventionally, motion within an image has been estimated by applying multiple 2-dimensional geometric transformations in parallel and combining them. However, these conventional 2-dimensional geometric transformation methods have limitations in capturing object motion in 3D space, which can also degrade the quality of the synthesized image.

[0006] Meanwhile, local motions associated with facial expressions, such as mouth movements and eye blinks in images of a specific individual, are dependent on global motions associated with the upper body, including the head and chest. This characteristic makes it difficult to exclude the influence of global motion within the image when predicting local motion from an image.

[0007] The disclosed embodiments are intended to provide a technical means for effectively generating a speech synthesis image by separating head movements and facial expression movements of a person in an image and predicting each movement in generating a speech synthesis image.

[0008] According to an exemplary embodiment, a speech synthesis image generation device is provided, including a head movement prediction module for estimating a three-dimensional geometric transformation of a head movement of a person between a first image and a second image of the person, calculating a plurality of transformation matrices for applying the three-dimensional geometric transformation at a plurality of preset depths based on a pinhole camera model, and predicting the head movement of the person between the first image and the second image using the plurality of transformation matrices; an expression movement prediction module for predicting the expression movement of the person between the first image and the second image based on a pre-learned expression movement prediction model; and a motion generation module for transforming the first image based on the predicted head movement and expression movement.

[0009] In one embodiment, the head movement prediction module can predict a first head pose and a second head pose based on 6 degrees of freedom from each of the first image and the second image based on a pre-learned head pose prediction model, and estimate a 3D geometric transformation using the first head pose and the second head pose.

[0010] In one embodiment, the head movement prediction module can obtain transformation matrix selection weights for applying a combination of multiple transformation matrices to each pixel of the first image using a pre-learned depth-based optical flow prediction model.

[0011] In one embodiment, the head motion prediction module can compute an optical flow for transforming a head pose of a first image from a plurality of transformation matrices and transformation matrix selection weights.

[0012] In one embodiment, the optical flow prediction model can be trained by applying image warping to a first image to deform the first image and comparing the deformed first image with a second image.

[0013] In one embodiment, the person in the first image and the person in the second image may be different from each other.

[0014] In one embodiment, the facial expression movement prediction model may be a model trained using one or more augmented images that have the head movement of the training images transformed using a head movement prediction module.

[0015] In one embodiment, the augmented image can be generated by predicting a 6-degree-of-freedom-based head pose from a training image based on a pre-trained head pose prediction model, adding random noise to the head pose to generate an augmented head pose, and calculating optical flow to deform the training image to correspond to the augmented head pose using a head motion prediction module.

[0016] In one embodiment, the head movement prediction module can predict a first head pose and a second head pose based on 6 degrees of freedom from each of the first image and the second image based on a pre-learned head pose prediction model, and estimate a 3D geometric transformation using the first head pose and the second head pose.

[0017] In one embodiment, the head movement prediction module can obtain transformation matrix selection weights for applying a combination of multiple transformation matrices to each pixel of the first image using a pre-learned depth-based optical flow prediction model.

[0018] In one embodiment, the head motion prediction module can compute an optical flow for transforming a head pose of a first image from a plurality of transformation matrices and transformation matrix selection weights.

[0019] In one embodiment, the optical flow prediction model can be trained by applying image warping to a first image to deform the first image and comparing the deformed first image with a second image.

[0020] In one embodiment, the person in the first image and the person in the second image may be different from each other.

[0021] In one embodiment, the facial expression movement prediction model may be a model trained using one or more augmented images that have the head movement of the training images transformed using a head movement prediction module.

[0022] In one embodiment, the augmented image can be generated by predicting a 6-degree-of-freedom-based head pose from a training image based on a pre-trained head pose prediction model, adding random noise to the head pose to generate an augmented head pose, and calculating optical flow to deform the training image to correspond to the augmented head pose using a head motion prediction module.

[0023] According to the disclosed embodiments, a more natural speech synthesis video can be generated compared to the prior art by separating the head movement and facial expression movement of a person in an image and predicting each movement.

[0024] Figure 1 is a block diagram for explaining a speech synthesis image generation device (100) according to one embodiment.

[0025] Figure 2 is a block diagram for explaining the detailed configuration of a head movement prediction module (102) according to one embodiment.

[0026] Figure 3 is a flowchart for explaining the process of performing learning of a facial expression movement prediction model in the facial expression movement prediction module (104).

[0027] Figure 4 is a flowchart for explaining a method (400) for generating a synthetic speech image according to one embodiment.

[0028] FIG. 5 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments.

[0029] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the present invention is not limited thereto.

[0030] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing embodiments of the present invention and should not be limited in any way. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0031] Additionally, while terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0032] In the disclosed embodiment, a "synthesized speech video" is a video synthesized using a machine learning model of a specific person speaking, and may also be referred to as a lip-sync video. The synthesized speech video may be an upper body video including, but not limited to, the head and chest of the specific person.

[0033] FIG. 1 is a block diagram illustrating a speech synthesis image generating device (100) according to one embodiment. The illustrated speech synthesis image generating device (100) means a device that receives a source image and a driving image, predicts head movement and facial expression movement between the two images, and generates or synthesizes the source image into a driving movement (head movement and facial expression movement) based on the predicted head movement and facial expression movement. In the disclosed embodiments, the speech synthesis image generating device (100) is configured to predict head movement between the two images by using distance (depth) information from a pinhole and a subject in a 3D pinhole camera model. In addition, in the disclosed embodiments, the speech synthesis image generating device (100) is configured to improve the performance of facial expression movement prediction by separating only facial expression movements independent of head movement from the images by utilizing a head movement prediction technique between the two images.

[0034] As illustrated in FIG. 1, a speech synthesis image generation device (100) according to one embodiment includes a head movement prediction module (102), a facial expression movement prediction module (104), and a movement generation module (106).

[0035] The head movement prediction module (102) receives a source image (first image) and a driving image (second image) as input and predicts head movement between two images of a person included in the two images. In the disclosed embodiments, the image may be a still image such as a photograph, or a specific frame within a video. In one embodiment, the source image and the driving image may be images taken of a scene in which the same person speaks. In another embodiment, the source image and the driving image may be images taken of scenes in which different people speak, respectively.

[0036] The facial expression movement prediction module (104) predicts facial expression movements between two images of a person included in an input source image and a driving image. In the disclosed embodiments, facial expression movements may refer to movements of the eyes, eyebrows, mouth, and jaw, etc., that appear on the face of a person in the image when the person speaks.

[0037] The motion generation module (106) generates a synthetic image by transforming a source image by combining the head movement predicted by the head movement prediction module (102) and the facial expression movement predicted by the facial expression movement prediction module (104).

[0038] Below, the detailed configuration of the speech synthesis image generation device (100) configured as described above is described.

[0039] Head movement prediction

[0040] In one embodiment, the head movement prediction module (102) is configured to estimate a three-dimensional geometric transformation for the head movement of a person between source and driving images, calculate a plurality of transformation matrices for applying the three-dimensional geometric transformation at a plurality of preset depths based on a pinhole camera model, and calculate an optical flow for the head movement between the source and driving images using the plurality of transformation matrices.

[0041] FIG. 2 is a block diagram illustrating a detailed configuration of a head movement prediction module (102) according to one embodiment. As illustrated, the head movement prediction module (102) according to one embodiment includes a head pose prediction module (202), a geometric transformation module (204), a transformation matrix calculation module (206), a weight calculation module (208), and an optical flow calculation module (210).

[0042] The head pose prediction module (202) predicts the head pose of a person included in each image from each of the source image and the driving image. In one embodiment, the head pose of the person included in the image may be a head pose based on 6 degrees of freedom including the center coordinates of the head and the rotation angle of the head. In addition, in one embodiment, the head pose prediction module (202) may predict the head pose based on a previously learned head pose prediction model. The head pose prediction model may be a model learned to receive an image as input and predict a 3D head pose based on 6 degrees of freedom therefrom. In the following description, the head pose estimated from the source image is referred to as a first head pose, and the head pose estimated from the driving image is referred to as a second head pose.

[0043] When inputting source and driving images into a head pose prediction model, the 3D head pose of the person can be estimated as follows.

[0044] [Mathematical Formula 1]

[0045]

[0046] Here, I is the input image, F head is a head pose prediction model, x, y, z represent the center coordinates of the head, and φ, θ, ψ represent the rotation angles of the head (roll, pitch, yaw), respectively.

[0047] The geometric transformation module (204) estimates a three-dimensional geometric transformation for head movement between the source and driving images using the first and second head poses calculated by the head pose prediction module (202). In one embodiment, the geometric transformation module (204) can calculate a three-dimensional geometric transformation for head movement between the source and driving images using the following mathematical equation.

[0048] [Equation 2]

[0049]

[0050] In the above mathematical formula, A S←D A 3D geometric transformation of the head pose between source and driving, S←R and A D←R are 3D geometric transformations for the head poses of the source and driving, respectively, which can be expressed as a 3x4 transformation matrix or a 4x4 transformation matrix in homogeneous coordinates. Here, R is a reference coordinate, which can be, for example, the case where the center coordinate and rotation angle of the head are both 0, or the average value of the head poses included in the training data.

[0051] The transformation matrix calculation module (206) calculates multiple transformation matrices for applying a three-dimensional geometric transformation between the source and driving images at multiple preset depths based on a pinhole camera model. This is described in more detail as follows.

[0052] 3D geometric transformation between source and driving (A S←R ) is applied to the 3D projection method of the camera pinhole model, the correspondence between the pixel coordinates of the source and driving images can be calculated as follows. This can be expressed mathematically as follows.

[0053] [Equation 3]

[0054]

[0055] In the above mathematical formula, p D is the pixel coordinate value of the driving image, p S is the pixel coordinate value of the corresponding source image, K is the intrinsic parameter matrix of the pinhole camera model, K -1 is the inverse of K, D(p D ) represents the distance (depth) between the pinhole and the subject in the pinhole camera model. As can be seen from the mathematical formula above, p D is the inverse matrix of K and pD Depth value at location (D(p) D )) can be multiplied to project it into a 3D coordinate value, and then A S←D It can be converted to a 3D source coordinate value by multiplying by K, and then projected to the source image pixel coordinate by multiplying by K again.

[0056] The transformation matrix calculation module (206) calculates the preset depth values ​​D = {D1, ..., D N A set of transformation matrices T = {T1, ..., T} N} can be calculated using the following mathematical formula.

[0057] [Equation 4]

[0058]

[0059] ...

[0060]

[0061] In the above mathematical formula, the range and number of depth values ​​D can be appropriately set according to the characteristics of the learning data or learned through a machine learning model.

[0062] The weight calculation module (208) calculates transformation matrix selection weights for applying the transformation matrix set T calculated by the transformation matrix calculation module (206) to each pixel of the source image. In one embodiment, the weight calculation module (208) may calculate transformation matrix selection weights for applying a combination of multiple transformation matrices to each pixel of the source image using a pre-learned depth-based optical flow prediction model (212). The optical flow prediction model may be a model that is learned by applying image warping to the source image to transform the source image and comparing the transformed source image with the driving image.

[0063] The transformation matrix selection weight for applying the transformation matrix T to each pixel of the source image can be calculated by the following mathematical formula.

[0064] [Equation 5]

[0065]

[0066] Here, F flow is a depth-based optical flow prediction model (212), S is the source image, T is the transformation matrix, W = {W1, ..., W N} are {T1, 쪋 T respectively N} is a set of weights to be applied in combination to each pixel of the source image.

[0067] The optical flow calculation module (210) calculates an optical flow for transforming the head pose of the source image into the head pose of the driving image from the transformation matrix and the transformation matrix selection weights. In one embodiment, the optical flow can be calculated by the following mathematical equation.

[0068] [Equation 6]

[0069]

[0070] At this time, f S←D is the optical flow for transforming the source image into the head pose of the driving image, and Wi and Ti are the ith element of the set of transformation matrices and the set of weights, respectively.

[0071] The facial expression movement prediction module (104) can transform the source image into the head pose of the driving image through an image warping method using the calculated optical flow.

[0072] Facial expression movement prediction

[0073] The facial expression movement prediction module (104) predicts facial expression movement between two images of a person included in an input source image and a driving image. In the disclosed embodiments, the facial expression movement prediction module (104) may be configured to predict facial expression movement between two images of a person included in a source image and a driving image using a previously learned facial expression movement prediction model. In one embodiment, the facial expression movement prediction model may be a model learned using one or more augmented images that transform the head movement of a learning image using the head movement prediction module (102) described above. This will be described in more detail as follows.

[0074] The facial expression movement prediction module (104) can induce the facial expression movement prediction model to learn facial expression movements regardless of the head pose by using the head movement prediction module (102) to randomly change the head pose of the input image through random pose augmentation. This data augmentation method can be applied only during the learning of the facial expression movement prediction model, and once learning is complete, only facial expression movements can be predicted from the input image without data augmentation.

[0075] Figure 3 is a flowchart for explaining a process (300) of performing learning of a facial expression movement prediction model in a facial expression movement prediction module (104).

[0076] In step 302, the facial expression movement prediction module (104) predicts a head pose based on six degrees of freedom from a learning image (including a source image or a driving image). In one embodiment, the facial expression movement prediction module (104) may be configured to predict a head pose based on six degrees of freedom from the learning image based on the head pose prediction model of the head movement prediction module (102) described above.

[0077] Let the training image be I, and the head pose θ predicted from it Ican be calculated as follows:

[0078] [Equation 7]

[0079]

[0080] In step 304, the facial expression movement prediction module (104) adds random noise to the head pose calculated in step 302 to generate one or more augmented head poses. Specifically, the augmented head pose θ I aug can be calculated as follows:

[0081] [Equation 8]

[0082]

[0083] At this time, N aug is a random noise value for the 6-degree-of-freedom pose.

[0084] In step 306, the facial expression movement prediction module (104) calculates an optical flow to transform the head pose of the learning image using the augmented head pose calculated in step 304. The facial expression movement prediction module (104) calculates θ I and θ I aug Using the mathematical equations 2 to 6 described above as input, the head posture of I is N aug Head motion optical flow f to transform as much as I←aug can be calculated.

[0085] In one embodiment, the head pose of image I can be transformed using image warping as follows, where warp is an image backward mapping function.

[0086] [Equation 9]

[0087]

[0088] Data augmentation for the input image of the facial expression movement prediction module (104) using the head movement prediction module (102) may be performed separately after the training of the head movement prediction module (102) is completed, or may be performed simultaneously with the training of the head movement prediction module (102). In addition, when the movement generation module (106) is trained simultaneously with the head movement prediction module (102) or the facial expression movement prediction module (104), the quality of the speech synthesis image can be improved.

[0089] FIG. 4 is a flowchart illustrating a method (400) for generating a speech synthesis image according to one embodiment. The illustrated method may be performed in a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors, for example, a speech synthesis image generating device (100) according to one embodiment. In the illustrated flowchart, the method or process is described by dividing it into a plurality of steps, but at least some of the steps may be performed in a changed order, combined with other steps and performed together, omitted, divided into detailed steps and performed, or one or more steps not illustrated may be added and performed.

[0090] In step 402, the head movement prediction module (102) receives a source image and a driving image as input and predicts head movement between two images of a person included in the two images.

[0091] In step 404, the facial expression movement prediction module (104) predicts facial expression movements between two images of a person included in the input source image and the driving image.

[0092] In step 406, the motion generation module (106) generates a synthetic image by transforming the source image using a motion that combines the head motion predicted by the head motion prediction module (102) and the facial expression motion predicted by the facial expression motion prediction module (104).

[0093] FIG. 5 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

[0094] The illustrated computing environment (10) includes a computing device (12). In some embodiments, the computing device (12) may be one of a data collection device (102) and an image recording device (104) constituting a real-time collection and linkage system (100) for autonomous vehicle data and image data. The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may cause the computing device (12) to operate according to the exemplary embodiments described above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, which, when executed by the processor (14), may cause the computing device (12) to perform operations according to the exemplary embodiments.

[0095] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that can be accessed by the computing device (12) and store desired information, or a suitable combination thereof.

[0096] A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

[0097] The computing device (12) may also include one or more input / output interfaces (22) that provide interfaces for one or more input / output devices (24) and one or more network communication interfaces (26). The input / output interfaces (22) and the network communication interfaces (26) are connected to the communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) via the input / output interfaces (22). Exemplary input / output devices (24) may include input devices such as pointing devices (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and / or photographing devices, and / or output devices such as display devices, printers, speakers and / or network cards. The exemplary input / output devices (24) may be included within the computing device (12) as a component constituting the computing device (12), or may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0098] While representative embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims set forth below but also by equivalents thereof.

[0099] [Explanation of symbols]

[0100] 100: Speech Synthesis Image Generation Device

[0101] 102: Head Movement Prediction Module

[0102] 104: Facial Expression Movement Prediction Module

[0103] 106: Motion Generation Module

[0104] 202: Head Pose Prediction Module

[0105] 204: Geometric Transformation Module

[0106] 206: Transformation Matrix Calculation Module

[0107] 208: Weight Calculation Module

[0108] 210: Optical Flow Computation Module

[0109] 212: Depth-based optical flow prediction model

Claims

1. A head movement prediction module that estimates a three-dimensional geometric transformation for head movement of a person between a first image and a second image of the person, calculates a plurality of transformation matrices for applying the three-dimensional geometric transformation at a plurality of preset depths based on a pinhole camera model, and predicts head movement of the person between the first image and the second image using the plurality of transformation matrices; An expression movement prediction module that predicts the expression movement of the person between the first image and the second image based on a previously learned expression movement prediction model; and A speech synthesis image generation device including a motion generation module that transforms the first image based on the predicted head movement and the facial expression movement.

2. In claim 1, The above head movement prediction module, Predicting a first head pose and a second head pose based on 6 degrees of freedom from each of the first image and the second image based on a previously learned head pose prediction model, A speech synthesis image generation device that estimates the three-dimensional geometric transformation using the first head posture and the second head posture.

3. In claim 1, The above head movement prediction module, A speech synthesis image generation device that obtains a transformation matrix selection weight for applying a combination of the plurality of transformation matrices to each pixel of the first image using a previously learned depth-based optical flow prediction model.

4. In claim 3, The above head movement prediction module, A speech synthesis image generation device that calculates an optical flow for transforming the head pose of the first image from the plurality of transformation matrices and the transformation matrix selection weights.

5. In claim 3, The above optical flow prediction model is, A speech synthesis image generation device that learns by applying image warping to the first image to transform the first image and comparing the transformed first image with the second image.

6. In claim 1, A speech synthesis image generation device in which the person in the first image and the person in the second image are different from each other.

7. In claim 1, A speech synthesis image generation device, wherein the above facial expression movement prediction model is a model learned using one or more augmented images that have transformed the head movement of a learning image using the above head movement prediction module.

8. In claim 7, The above augmented image is, Predicting a head pose based on 6 degrees of freedom from the above learning image based on a previously learned head pose prediction model, Generate an augmented head pose by adding random noise to the above head pose, A speech learning image generation device, which is generated by calculating an optical flow for transforming the learning image to correspond to the augmented head posture using the head movement prediction module.

9. In a head movement prediction module, a step of estimating a three-dimensional geometric transformation for head movement of a person between a first image and a second image of the person, calculating a plurality of transformation matrices for applying the three-dimensional geometric transformation at a plurality of preset depths based on a pinhole camera model, and predicting head movement of the person between the first image and the second image using the plurality of transformation matrices; In the facial expression movement prediction module, a step of predicting the facial expression movement of the person between the first image and the second image based on a previously learned facial expression movement prediction model; and A method for generating a speech synthesis image, comprising a step of deforming the first image based on the predicted head movement and the facial expression movement in a motion generation module.

10. In claim 9, The above head movement prediction module, Predicting a first head pose and a second head pose based on 6 degrees of freedom from each of the first image and the second image based on a previously learned head pose prediction model, A method for generating a speech synthesis image, wherein the three-dimensional geometric transformation is estimated using the first head posture and the second head posture.

11. In claim 9, The above head movement prediction module, A method for generating a speech synthesis image, wherein a transformation matrix selection weight is obtained for applying a combination of the plurality of transformation matrices to each pixel of the first image using a depth-based optical flow prediction model that has been previously learned.

12. In claim 11, The above head movement prediction module, A method for generating a speech synthesis image, wherein an optical flow for transforming a head pose of the first image is calculated from the plurality of transformation matrices and the transformation matrix selection weights.

13. In claim 11, The above optical flow prediction model is, A method for generating a speech synthesis image, wherein the first image is deformed by applying image warping to the first image, and the deformed first image is compared with the second image, thereby learning.

14. In claim 9, A method for generating a speech synthesis image, wherein the person in the first image and the person in the second image are different from each other.

15. In claim 9, A method for generating a speech synthesis image, wherein the above facial expression movement prediction model is a model learned using one or more augmented images in which the head movement of the learning image is transformed using the above head movement prediction module.

16. Claim 15, The above augmented image is, Predicting a head pose based on 6 degrees of freedom from the above learning image based on a previously learned head pose prediction model, Generate an augmented head pose by adding random noise to the above head pose, A method for generating a speech learning image, wherein the method is generated by calculating an optical flow for transforming the learning image to correspond to the augmented head posture using the head movement prediction module.

Images