Electronic device for generating talking head and operation method thereof

Abstract

Disclosed is an electronic device for generating a talking head. The electronic device for generating a talking head may comprise a storage unit and a processor, wherein the processor may: acquire a style intermediate feature and a style feature from a reference frame by using an encoder; acquire an audio feature from audio data; and acquire a content feature from a target frame. The processor may: acquire a first decoder intermediate feature from the style feature, the audio feature, and the content feature by using a style-based decoder including an attention-block; acquire a second decoder intermediate feature by performing an attention mechanism on the first decoder intermediate feature and the style intermediate feature; and generate a talking head frame from the second decoder intermediate feature.

Description

Electronic device for generating a talking head and method of operating the same

[0001] The present invention relates to an electronic device for generating a talking head and a method of operating the same. A talking head may have the same meaning as a dubbed video, and may include the head or face of a speaker who is speaking (or is speaking).

[0002] Audio-driven talking head generation is the process of synthesizing a talking face using lip movements synchronized with driven audio. This process has attracted significant attention due to its wide-ranging applications in the media industry, including virtual human animation, audiovisual dubbing, and video content creation. One well-known method is the one-shot audio-driven method, which generates a talking head using only a single reference frame (reference image). However, this method tends to overfit to a single reference frame. In other words, it struggles to generate facial poses and / or mouth shapes that deviate from the reference frame. Scenarios that require talking heads with significantly different facial poses and mouth shapes compared to the reference frame are the most common scenarios encountered in real life, yet research on these scenarios is limited. Furthermore, given the sensitivity of one-shot audio-driven methods to the selection of a reference frame, it is impractical and inconvenient for designers to manually select an appropriate reference frame.

[0003] An object of the present invention is to provide an electronic device for generating a talking head that is insensitive to the selection of a reference frame and a method of operating the same.

[0004] An object of the present invention is to provide an electronic device and an operating method thereof for generating a talking head that is not affected by the facial pose, mouth shape, or facial expression of a reference person (a person in a reference frame).

[0005] An object of the present invention is to provide an electronic device for generating a talking head capable of consistently generating a high-quality talking head by obtaining the identity of a reference person (target identity) from a reference frame of one sheet, and an operating method thereof.

[0006] An object of the present invention is to provide an electronic device for generating a talking head capable of consistently generating a high-quality talking head reflecting detailed information of a mouth area of ​​a reference person from a single reference frame, and an operating method thereof.

[0007] One aspect of the present invention provides an electronic device for generating a talking head, which may include a storage unit and a processor, wherein the processor may obtain a style intermediate feature and a style feature from a reference frame using an encoder, obtain an audio feature from audio data, and obtain a content feature from a target frame, obtain a first decoder intermediate feature from the style feature, the audio feature, and the content feature using a style-based decoder including an attention-block, obtain a second decoder intermediate feature by performing an attention mechanism on the first decoder intermediate feature and the style intermediate feature, and generate a talking head frame from the second decoder intermediate feature.

[0008] In one embodiment, the processor can obtain a value and a key from the mouth region of the style intermediate feature, obtain a query from the mouth region of the first decoder intermediate feature, and obtain an attention score from the value, the key, and the query.

[0009] In one embodiment, the processor can obtain a second decoder intermediate feature by concatenating the attention score with the first decoder intermediate feature.

[0010] In one embodiment, the processor may generate a style by concatenating style features and audio features, and generate a talking head frame from the second decoder intermediate features through style-based modulation. In one embodiment, the style features may be sparse features.

[0011] In one embodiment, the processor can further obtain content intermediate features from the target frame and add the content intermediate features to the second decoder intermediate features to generate the talking head frame.

[0012] Another aspect of the present invention is a method of operating an electronic device for generating a talking head, which may include the steps of (a) using an encoder to (a1) obtain a style intermediate feature and a style feature from a reference frame, (a2) obtaining an audio feature from audio data, (a3) ​​obtaining a content feature from a target frame, (b) using a style-based decoder including an attention-block to (b1) obtain a first decoder intermediate feature from the style feature, the audio feature, and the content feature, (b2) performing an attention mechanism on the first decoder intermediate feature and the style intermediate feature to obtain a second decoder intermediate feature, and (b3) generating a talking head frame from the second decoder intermediate feature.

[0013] In one embodiment, step (b2) may include step (b2-1) obtaining a value and a key from the mouth region of the style intermediate feature, step (b2-2) obtaining a query from the mouth region of the first decoder intermediate feature, and step (b2-3) obtaining an attention score from the value, the key, and the query.

[0014] In one embodiment, step (b2) may further include the step of obtaining a second decoder intermediate feature by concatenating the attention score (b2-4) with the first decoder intermediate feature.

[0015] In one embodiment, step (b1) may include generating a style by concatenating style features and audio features, and step (b3) may include generating a talking head frame from the second decoder intermediate features through style-based modulation. In one embodiment, the style features may be sparse features.

[0016] In one embodiment, step (a3) ​​may further include obtaining content intermediate features from the target frame, and step (b3) may include generating a talking head frame by adding the content intermediate features to the second decoder intermediate features.

[0017] According to the present invention, an electronic device for generating a talking head that is insensitive to the selection of a reference frame and an operating method thereof can be provided.

[0018] According to the present invention, an electronic device and an operating method thereof for generating a talking head that is not restricted by the facial pose, mouth shape, or facial expression of a reference person can be provided.

[0019] According to the present invention, an electronic device for generating a talking head and an operating method thereof can be provided, which can consistently generate a high-quality talking head by obtaining the identity characteristics of a reference person from a single reference frame.

[0020] According to the present invention, an electronic device for generating a talking head and an operating method thereof can be provided that can consistently generate a high-quality talking head reflecting detailed information of a mouth area of ​​a reference person from a single reference frame.

[0021] According to the present invention, an electronic device for generating a talking head and an operating method thereof can be provided that can consistently generate a high-quality talking head even in harsh scenarios where the reference frame significantly deviates from the ground truth.

[0022] FIG. 1 is a schematic block diagram of an electronic device for generating a talking head according to one embodiment of the present invention.

[0023] Figure 2 is a conceptual diagram of a talking head generation model according to one embodiment of the present invention.

[0024] FIG. 3 is a schematic block diagram of a style encoder according to one embodiment of the present invention.

[0025] FIG. 4 is a schematic block diagram of an audio encoder according to one embodiment of the present invention.

[0026] FIG. 5 is a schematic block diagram of a content encoder according to one embodiment of the present invention.

[0027] FIG. 6 is a schematic block diagram of a residual down-block according to one embodiment of the present invention.

[0028] Figure 7 is a schematic block diagram of an attention block according to one embodiment of the present invention.

[0029] Figure 8 is a schematic block diagram of a style modulation block according to one embodiment of the present invention.

[0030] Figure 9 shows frames generated by the first to sixth comparative examples and the embodiment, respectively.

[0031] Figure 10 shows frames generated from three different reference frames according to an embodiment of the present invention.

[0032] FIG. 11 illustrates an attention map that highlights important regions within a reference frame corresponding to a patch of a frame generated by an embodiment of the present invention.

[0033] [Explanation of symbols]

[0034] 100: Electronic device for generating talking heads

[0035] 110: Storage

[0036] 120: Processor

[0037] MD: Talking Head Generation Model

[0038] Hereinafter, the present invention will be described in detail. However, the present invention is not limited or restricted by the exemplary embodiments. The objectives and effects of the present invention can be naturally understood or made more clear through the following description, and the objectives and effects of the present invention are not limited by the following description alone. Furthermore, in explaining the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, such a detailed description will be omitted.

[0039] FIG. 1 is a schematic block diagram of an electronic device for generating a talking head according to an embodiment of the present invention. Referring to FIG. 1, an electronic device (100) for generating a talking head may include a storage unit (110) and a processor (120). The storage unit (110) may store reference frames, audio data, target frames, etc., and may store a talking head generation model (MD, see FIG. 2). The storage unit (110) may include at least one of a volatile memory and a non-volatile memory. The processor (120) may control the overall operation of the electronic device (100) for generating a talking head, and may control the storage unit (110). The processor (120) may include at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0040] Fig. 2 is a conceptual diagram of a talking head generation model according to an embodiment of the present invention. Referring to Fig. 2, the talking head generation model (MD) includes a style encoder (E s ), audio encoder (E a ), content encoder (E c ) and a decoder (G). First, the input data and output data of the talking head generation model (MD) are described.

[0041] Training phase

[0042] The talking head generation model (MD) can receive at least one reference frame as input, and can also receive multiple reference frames for easy learning. Each reference frame can be randomly selected, and the person in each reference frame can be the same.

[0043] The talking head generation model (MD) can receive at least one target frame as input, and can also receive multiple target frames for easy learning. The multiple target frames can be sequentially sequential frames. The mouth area of ​​each target frame can be masked.

[0044] The number of reference frames and target frames input to the talking head generation model (MD) can be the same. The person in each reference frame and the person in each target frame can be the same.

[0045] The talking head generation model (MD) can receive at least one audio data input, and can also receive multiple audio data inputs for easy learning. The audio data may be a mel-spectrogram. Each audio data input to the talking head generation model (MD) can correspond to and be aligned with each target frame input to the talking head generation model (MD).

[0046] A talking head generation model (MD) can generate a talking head. The talking head can include at least one, for example, a plurality of talking head frames. The talking head generation model (MD) can generate a talking head frame by inpainting a masked mouth region of a target frame. The talking head generation model (MD) can extract an identity from a person in a reference frame and pay attention to the mouth region of the reference frame to inpaint the masked mouth region of the target frame. The number of talking head frames generated by the talking head generation model (MD) can be the same as the number of target frames input to the talking head generation model (MD). Even if the talking head generation model (MD) receives a plurality of reference frames, a plurality of audio data, and a plurality of target frames as input, the talking head generation model (MD) can individually perform inpainting of the target frames, i.e., generation of the talking head frames.

[0047] Inference stage

[0048] The talking head generation model (MD) can receive one reference frame and one target frame as input and generate one talking head frame.

[0049] Style Encoder (E s )

[0050] Fig. 3 is a schematic block diagram of a style encoder according to an embodiment of the present invention. Referring to Fig. 2 together with Fig. 3, the style encoder (E s ) may contain multiple blocks, for example, L blocks. Style encoder (E s ) may include at least one residual down-block and at least one convolutional block. More specifically, the style encoder (E s ) may contain L-2 residual down-blocks and two convolution blocks. For example, the style encoder (E s) can contain six blocks, including four residual down-blocks and two convolution blocks. The style encoder (E s ) may further include at least one fully-connected layer. For example, a style encoder (E s ) may include one more fully-connected layer. Style encoder (E s ) style encoder excluding fully-connected layers (E s ) and the number of blocks in the content encoder (E c ) can have the same number of blocks as L.

[0051] Style Encoder (E s ) is at least one style intermediate feature (f) from the reference frame. s ) can be obtained, and style features (F s ) can be obtained. More specifically, the style encoder (E s ) are the first to fourth style intermediate features (f) from each residual down-block. s,1 , f s,2 , f s,3 , f s,4 ) can be obtained respectively, and the fifth and sixth style intermediate features (f) can be obtained from each convolution block. s,5 , f s,6 ) can be obtained respectively. The intermediate features of the 1st to 6th styles (f s,1 Inland f s,6 ) can be used in the attention mechanism of the decoder (G) described below. For example, the first and second style intermediate features (f s,1 , f s,2 ) can be used in the attention mechanism of the decoder (G). The style encoder (E s ) is a style feature (F) from a fully-connected layer. s ) can be obtained. Style features (F s ) may be a sparse feature (e.g. F s ∈ R 512 ). Style Encoder (Es ) through a fully-connected layer to the sixth style intermediate feature (f s,6 ) can be sparse, allowing the decoder (G) to capture general information (attributes) of the reference frame while ignoring structural details of the reference frame, such as facial pose, mouth shape, etc.

[0052] Audio Encoder (E a )

[0053] Fig. 4 is a schematic block diagram of an audio encoder according to an embodiment of the present invention. Referring to Fig. 4, an audio encoder (E a ) may include at least one convolutional layer, at least one residual block, at least one self-attention pooling layer, and at least one fully-connected layer. For example, an audio encoder (E a ) may include one convolutional layer, four residual blocks, one self-attention pooling layer, and one fully-connected layer. The four residual blocks may include one residual block that does not include an additional convolutional layer and three residual blocks that include an additional convolutional layer. Each convolutional layer illustrated in FIG. 4 may be a convolutional 2D layer. The self-attention pooling layer may include, for example, two consecutive convolutional layers, and has the advantage of focusing on frame-level features.

[0054] Audio Encoder (E a ) is an audio feature (F) from a fully-connected layer. a ) can be obtained (e.g. F a ∈ R 512 ) For example, a self-attention pooling layer can obtain attention weights (W) from intermediate features (X) obtained by the last residual block, and an audio encoder (E a) can obtain the intermediate feature (X') by multiplying the intermediate feature (X) and the attention weight (W) element by element. The fully-connected layer can obtain the audio feature (F) from the intermediate feature (X'). a ) can be obtained.

[0055] Create a style

[0056] The talking head generative model (MD) is a style feature (F s ) and audio features (F a ) to connect the W space features (F W ) can be obtained (e.g. F W ∈ R 1024 ), W spatial features (F W ) can be used to generate styles by affine transformation. The affine transformation is a W space feature (F W ) can mean obtaining scale and shift from.

[0057] Content Encoder (E c )

[0058] FIG. 5 is a schematic block diagram of a content encoder according to an embodiment of the present invention. Referring to FIG. 2 together with FIG. 5, the content encoder (E c ) may contain multiple blocks, for example, L blocks. Content encoder (E c ) may include at least one residual down-block and at least one convolution block. More specifically, the content encoder (E c ) may contain L-2 residual down-blocks and two convolution blocks. For example, the content encoder (E c ) can contain six blocks, including four residual down-blocks and two convolution blocks.

[0059] Content Encoder (E c ) is at least one content intermediate feature (f) from the target frame. c ) can be obtained, and content features (F c ) can be obtained (e.g. F c∈ R 12×12×512 ) More specifically, the content encoder (E c ) is the first to fourth content intermediate feature (f) from each residual down-block. c,1 , f c,2 , f c,3 , f c,4 ) can be obtained respectively, and the fifth and sixth content intermediate features (f) from each convolution block c,5 , f c,6 ) can be obtained respectively. 6th content feature (f c,6 ) is a content feature (F c ) may be identical to the content encoder (E c ) may include a global average pooling layer. The content encoder (E c ) is a content feature (F) through a global average pooling layer. c ) can be obtained. The first to fifth content intermediate features (f c,1 Inland f c,5 ) can be used for residual mapping of the decoder (G) described below. For example, the first to fourth content intermediate features (f c,1 Inland f c,4 ) can be used for residual mapping of the decoder (G). Content features (F c ) can be used as input to the decoder (G).

[0060] Residual down-block

[0061] Fig. 6 is a schematic block diagram of a residual down-block according to an embodiment of the present invention. The residual down-block of Fig. 6 includes a style encoder (E s ) and content encoder (E c ) can be used in at least one of the following. Referring to FIG. 6, the residual down-block can include multiple convolutional layers and at least one activation layer.

[0062] In one embodiment, the input of the first convolutional layer (#1) can be a reference frame or a target frame. In another embodiment, the input of the first convolutional layer (#1) can be a style intermediate feature (f) obtained by the previous residual down-block. s ) or content mid-feature (f c ) may be.

[0063] The first convolutional layer (#1) can obtain a first intermediate feature (f') from the input. The second convolutional layer (#2) can obtain a second intermediate feature (f'') from the first intermediate feature (f'). The third convolutional layer (#3) can obtain a third intermediate feature (f''') from the input.

[0064] The input of the third convolutional layer (#3) may be the same as the input of the first convolutional layer (#1). In other words, the input of the third convolutional layer (#3) may be a reference frame or a target frame in one embodiment, and may be a style intermediate feature (f) obtained by the immediately preceding residual down-block in another embodiment. s ) or content mid-feature (f c ) may be.

[0065] The activation layer can receive the second intermediate feature (f'') from the second convolutional layer (#2) through a skip-connection structure, the third intermediate feature (f''') from the third convolutional layer (#3), and the style intermediate feature (f) from the second intermediate feature (f'') and the third intermediate feature (f'''). s ) or content mid-feature (f c ) can be obtained.

[0066] In one embodiment, the first convolution layer (#1) and the second convolution layer (#2) may have the same kernel size, and the third convolution layer (#3) may have a kernel size smaller than the kernel sizes of the first convolution layer (#1) and the second convolution layer (#2). For example, the kernel sizes of the first convolution layer (#1) and the second convolution layer (#2) may each be 3, and the kernel size of the third convolution layer (#3) may be 1. In one embodiment, the activation layer may be LeakyReLU.

[0067] Decoder (G)

[0068] The decoder (G) can be a style-based decoder comprising multiple, for example, L, hierarchical blocks. For example, the style-based decoder can be a decoder of StyleGAN2 (Tero Karras et al., "Analyzing and improving the image quality of StyleGAN", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020). Each block of the decoder (G) can perform style modulation.

[0069] The decoder (G) may include at least one block including an attention block. The attention block may be a self-attention block. For example, the attention block may be a Vision Transformer (ViT, Dosovitskiy et al., "An image is worth 16 x 16 words: Transformers for image recognition at scale", 2021). The decoder (G) may include a style feature (F) which is a sparse feature. s) to generate a talking head, it may not be able to faithfully reconstruct details of the mouth area of ​​the target identity, such as fine textures and / or features. The decoder (G) can faithfully reconstruct details of the mouth area of ​​the target identity by including an attention block that can understand meaningful patterns and relationships between global image patches.

[0070] The decoder (G) may include at least one block that does not include an attention block and at least one block that includes an attention block. The block that does not include an attention block may be referred to as a style modulation block. The block that includes an attention block may include both an attention block and a style modulation block. The style modulation block may be located on the output side of the attention block. For example, the decoder (G) may include six blocks, including four blocks that do not include an attention block and two blocks that include an attention block. The first to fourth blocks that do not include an attention block may be located consecutively, and the fifth and sixth blocks that include an attention block may also be located consecutively, and the fifth block may be located on the output side of the fourth block. The attention block may include at least one attention layer. For example, an attention block may include two consecutive attention layers. In one embodiment, the attention layers may be multi-head attention layers. For example, the number of heads may be eight.

[0071] The block of the decoder (G) to which the attention block is applied can be determined by considering whether it can deliver semantically interpretable results based on the size of the patch. For example, the resolution of the decoder intermediate features of the fourth block of the decoder (G) may be 12 Х 12, which may be small to split these decoder intermediate features into patches. Meanwhile, in terms of visual fidelity and lip synchronization, it may be advantageous for one attention block to include two attention layers rather than one attention block to include one attention layer. In terms of visual fidelity and lip synchronization, it may be advantageous for one attention block to include two attention layers rather than one attention block to include one attention layer. In terms of visual fidelity and lip synchronization, it may be advantageous for one decoder block to include two attention blocks rather than one attention block to apply an attention block to each of the two blocks.

[0072] Fig. 7 is a schematic block diagram of an attention block according to an embodiment of the present invention. Referring to Fig. 7, the attention block includes a style intermediate feature (f s ) can obtain the value (V) and key (K) from the input region, and the decoder intermediate feature (f d ) can obtain a query (Q) from the mouth region. The mouth region may be an area including the lips and their surroundings. For example, the style intermediate feature (f s ) is the mouth area of ​​the style intermediate feature (f s ) may be the lower half region of the decoder middle feature (f d ) is the decoder intermediate feature (f d ) may be the lower half area of ​​the target frame (note that the masked mouth area of ​​the target frame may also be the lower half area of ​​the target frame).

[0073]

[0074] The attention layer uses the value (V), key (K) and query (Q) to calculate the attention score (f') according to [Mathematical Formula 1]. d) can be obtained. In [Mathematical Formula 1], the value (V), key (K), and query (Q) may be layer-normalized corresponding intermediate features. d k is the dimension of the key (K), and · represents the inner product. The attention-block is the attention score (f' d ) to the decoder intermediate feature (f d ) of the decoder intermediate features (f) in the remaining regions except the mouth region d ) and the decoder intermediate features (f'') d ) can be obtained (in the present invention, calculating the style intermediate feature and the decoder intermediate feature can be referred to as fidelity mapping). As illustrated in Fig. 7, the attention block can preserve detailed information of the mouth area of ​​the target identity by focusing on the mouth area.

[0075] In one embodiment, the attention-block uses style intermediate features (f) to obtain values ​​(V) and keys (K). s ) can perform patch embedding and position embedding, and add the patch embedding value and the position embedding value. In one embodiment, the attention-block obtains the query (Q) by using the decoder intermediate features (f d ) can perform patch embedding and position embedding, and add the patch embedding value and the position embedding value. The patch size can be as follows for the i-th block of the decoder (G): H i / 32 × W i / 32. Here, H i Wow W i refers to the height and width of the intermediate feature that the i-th block of the decoder (G) receives as input.

[0076] Style Encoder (E s ) and the number of blocks of the decoder (G) can be equal to L. At least one of the L blocks of the decoder (G) has at least one style intermediate feature (f s ) can be input. Style middle feature (f s) input to the decoder (G) with the block index (1, 2, ..., L) and the corresponding style intermediate feature (f s ) to obtain a style encoder (E s ) can be equal to L+1 depending on the encoder-decoder structure. The style intermediate feature (f) received by the block of the decoder (G) s ) may be the same as the resolution of the decoder intermediate features that the block of the decoder (G) receives from the previous block. For example, the style encoder (E s ) and the decoder (G) each contain six blocks, the sixth block of the decoder (G) contains the first style intermediate feature (f s,1 ) can be input, and the fifth block of the decoder (G) can input the second style intermediate feature (f s,2 ) can be input. The first style intermediate feature (f s,1 ) and the second style intermediate features (f s,2 ) may be the same as the resolution of the decoder intermediate feature input from the 5th block and the 4th block of the decoder (G) to the 6th block and the 5th block, respectively.

[0077] FIG. 8 is a schematic block diagram of a style modulation block according to an embodiment of the present invention. Referring to FIG. 8, the style modulation block may include a plurality of convolution layers and at least one upsampling layer. For example, the style modulation block may include two convolution layers (#1, #2) and one upsampling layer positioned between the two convolution layers. For example, the first convolution layer (#1) of the first block may include a content feature (F c ) can be input, and the first convolutional layer (#1) of each of the second to fourth blocks can receive the decoder intermediate features (f) acquired by each of the first to third blocks. d) can be input, and the first convolutional layer (#1) of each of the fifth and sixth blocks can receive the decoder intermediate features (f'') acquired by the attention-block of each of the fifth and sixth blocks. d , see Fig. 7) can be input (note that the attention block of the 5th block can input the decoder intermediate features acquired by the 4th block, and the attention block of the 6th block can input the decoder intermediate features acquired by the style modulation block of the 5th block). Hereinafter, for convenience of explanation, the features input to the first convolutional layer (#1) are collectively referred to as convolution inputs.

[0078] The first convolutional layer (#1) performs style modulation on the convolutional input to produce the first intermediate feature (f (1) ) can be obtained. Style modulation may mean, for example, performing a convolution operation on the convolution input after modulating the convolution kernel weights to obtain the convolution operation value, and then normalizing the convolution operation value (more specifically, the value obtained by dividing the convolution operation value by its standard deviation) by a scale and adding a shift. The upsampling layer obtains the first intermediate feature (f (1) ) by upsampling the second intermediate feature (f (2) ) can be obtained. The second convolutional layer (#2) obtains the second intermediate feature (f (2) ) to perform style modulation on the third intermediate feature (f (3) ) can be obtained. The style modulation block obtains the third intermediate feature (f (3) ) and content intermediate features (f s ) and add features (f (4) ) can be obtained. For example, feature (f (4) ) is the first to fifth block of the decoder (G), the feature (f (4) ) is the decoder intermediate feature (f d ) can be, and features (f (4) ) is the 6th block of the decoder (G), the feature (f (4)) may be a talking head frame.

[0079] Content Encoder (E c ) and the number of blocks of the decoder (G) can be equal to L. At least one of the L blocks of the decoder (G) has at least one content intermediate feature (f c ) can be input. Content middle feature (f c ) of the decoder (G) receiving the block index (1, 2, ..., L) and the corresponding content intermediate feature (f c ) to obtain the content encoder (E c ) can be equal to L+1 depending on the encoder-decoder structure. The intermediate features (f) of the content input to the block of the decoder (G) c ) is the third intermediate feature (f) acquired by the block of the corresponding decoder (G). (3) , see Fig. 8) may be the same as the resolution of the content encoder (E c ) and the decoder (G) each contain 6 blocks, the third block of the decoder (G) contains the fourth content intermediate feature (f c,4 ) can be input, and the fourth block of the decoder (G) can input the third content intermediate feature (f c,3 ) can be input, and the fifth block of the decoder (G) can input the second content intermediate feature (f c,2 ) can be input, and the sixth block of the decoder (G) can input the first content intermediate feature (f c,1 ) can be input. The fourth to first content intermediate features (f c,4 , f c,3 , f c,2 , f c,1 ) is the third intermediate feature (f) acquired by each of the third to sixth blocks of the decoder (G). (3) , may be the same as the resolution of Fig. 8).

[0080] loss function

[0081]

[0082] [Mathematical formula 2] is the reconstruction loss function (L rec ) is the reconstruction loss function (L rec ) can be composed of an L1 pixel loss function and a perceptual loss function. In [Mathematical Formula 2], φ i is the i-th layer of the VGG network, and L is the number of VGG layers. The weights λ for each layer i can be different from each other, and the weights can increase as the layers get deeper.

[0083]

[0084] [Mathematical formula 3] is the adversarial loss function (L adv ) is the adversarial loss function (L adv ) is a loss function commonly used in generative models to maintain high fidelity of images. In [Equation 3], D is the discriminator of StyleGAN2 and is trained together with the decoder (G).

[0085]

[0086]

[0087] [Equation 4] is the cosine similarity. In [Equation 4], p, I, and A represent probability, image, and audio, respectively. [Equation 5] is the synchronization loss function. In [Equation 5], y i indicates whether it is synchronized. The talking head generation model (MD) may include a gray-scale synchronization discriminator. The gray-scale synchronization discriminator may include an audio encoder (see Sa in [Equation 4]) that receives a mel-spectrogram as input and a vision encoder (see Sv in [Equation 4]) that receives a face image as input. The gray-scale synchronization discriminator determines whether it is synchronized (y i= 1) It can be trained with binary-cross entropy loss ([Equation 5]) to increase the cosine similarity ([Equation 4]) of visual and audio features of five consecutive frames. Unsynchronized (y i = 0) frame, the opposite is true. For the detailed configuration of the gray-scale sync discriminator, refer to "A lip sync expert is all you need for speech to lip generation in the wild (KR Prajwal et al., In Proceedings of the 28th ACM International Conference on Multimedia, pages 484-492, 2020)". For the detailed configuration of the audio encoder, refer to "In defense of metric learning for speaker recognition (Joon Son Chung et al., In Proc. Interspeech, 2020)".

[0088]

[0089] [Equation 6] is the total training loss function (L total ) is the total training loss function (L total ) is the reconstruction loss function (L rec ), adversarial loss function (L adv ) and synchronization loss function (L sync ) is the sum of λ adv and λ sync are the adversarial loss functions (L, respectively). adv ) and synchronization loss function (L sync ) is the equilibrium constant.

[0090] Hereinafter, experimental examples of the present invention will be described.

[0091] Dataset and Preprocessing

[0092] 1) Training dataset: LRW (A. Zisserman JS Chung, "Lip reading in the wild", In Asian Conference on Computer Vision (ACCV), 2016). The LRW dataset is an audio-visual dataset containing 1,000 utterances consisting of 500 different words.

[0093] 2) Evaluation dataset:

[0094] - HDTF (Zhimeng Zhang et al., "Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset", In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 3661-3670, 2021).

[0095] - VoxCeleb2 (A. Nagrani JS Chung et al., "Voxceleb2: Deep speaker recognition", In INTERSPEECH, 2018).

[0096] The HDTF dataset primarily consists of videos with frontal head poses, while the VoxCeleb2 dataset consists of videos with a wide range of head poses. Fifty videos were randomly selected from each dataset, and all videos were resampled to a frame rate of 25 FPS.

[0097] 3) Image preprocessing: After FFHQ alignment (Tero Karras et al., "A style-based generator architecture for generative adversarial networks", In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (CVPR), pages 4401-4410, 2019), the image was cropped and the resolution of the image was adjusted to 192 Х 192. In the preprocessed image, both the mouth and nose are vertically centered and cover the entire lower half of the face.

[0098] 4) Each encoder was configured identically to Figs. 3, 4, and 5. The decoder was configured with six blocks (i.e., L = 6) for resolution matching. An attention block was applied to each of the fifth and sixth blocks of the decoder, and the attention block consisted of two consecutive attention layers.

[0099] baseline

[0100] First comparative example: ATVGNet (Lele Chen et al., "Hierarchical cross-modal talking face generation with dynamic pixel-wise loss", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7832-7841, 2019).

[0101] Second comparative example: MakeItTalk (Yang Zhou et al., "Makeittalk: speaker-aware talking-head animation", ACM Transactions on Graphics (TOG), 39(6):1-15, 2020).

[0102] 제3 비교예: PC-AVS(Hang Zhou et al., "Pose-controllable talking face generation by implicitly modularized audio-visual representation", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021).

[0103] 제4 비교예: Wav2Lip(K R Prajwal et al., "A lip sync expert is all you need for speech to lip generation in the wild", In Proceedings of the 28th ACM International Conference on Multimedia, page 484-492, 2020).

[0104] 제5 비교예: DINet(Zhimeng Zhang et al., "Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video", In AAAI, 2023).

[0105] 제6 비교예: IP-LAP(Weizhi Zhong et al., "Identity-preserving talking face generation with landmark and appearance priors", In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2023).

[0106] Note that Comparative Examples 1, 2, and 3 generate a full head driven by audio, and thus the alignment of the synthesized face differs from the correct answer. Comparative Examples 4, 5, and 6, as well as the example, correspond to inpainting-based methods with slightly different masked areas around the mouth. The authors of Comparative Examples 5 and 6 used multiple reference frames for better identity extraction. In this experiment, for fair comparison, all baselines used only the first frame of the video as a reference frame.

[0107] Qualitative evaluation

[0108] Figure 9 shows frames generated by Comparative Examples 1 to 6 and the embodiment, respectively. For the HDTF video clip on the left, given a reference frame with a closed mouth, each Comparative Example and the embodiment should generate a frame similar to the correct answer with a wide open mouth. For the VoxCeleb2 video clip on the right, given a reference frame of a left-facing face, each Comparative Example and the embodiment should generate a right-facing face. For a clear comparison, except for Comparative Example 1, the inventors aligned all generated frames using FFHQ alignment. The embodiment generated a face very similar to the correct answer in terms of visual fidelity and lip synchronization compared to all other Comparative Examples. Comparative Examples 1, 2, and 3, which synthesized the entire head, failed to preserve target identity due to insufficient resources for driving poses, and their alignment also deviated significantly from the correct answer. Comparative Example 4 generated a blurry lower face with unclear mouth properties. Comparative Examples 5 and 6 exhibited poor performance, with the generated mouth region more similar to the reference frame's mouth region than the correct mouth region. Comparative Example 6 failed to generate an open mouth and generated defects that deviated from pose alignment.

[0109] An experiment in resilience

[0110] Figure 10 illustrates frames generated from three different reference frames according to an embodiment of the present invention. The target frames are identical. Each reference frame is a front-facing image (A), a right-facing image with a closed mouth (B), and a left-facing image with an open mouth (C). An ideal scenario is to consistently generate accurate mouths (lips) regardless of various poses and mouth shapes. As shown in Figure 10, the embodiment exhibited nearly zero sensitivity to reference frames. In other words, the embodiment demonstrated robustness in generating lip-synchronized content by consistently generating accurately dubbed videos regardless of changes in the reference frame. All generated frames were very similar to the correct answer, and the mouth shapes of the generated frames were almost identical to the correct answer. Due to this robustness, the present invention does not require a process for selecting reference frames.

[0111] quantitative evaluation

[0112] The reconstruction quality and lip synchronization quality of each comparative example and example were measured.

[0113] - PSNR, MS-SSIM (Z. Wang et al., "Multiscale structural similarity for image quality assessment", In The Thirty-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pages 1398-1402 Vol.2, 2003), and LPIPS (Richard Zhang et al., "The unreasonable effectiveness of deep features as a perceptual metric", In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018) measure the pixel-by-pixel and feature-by-feature similarity between the generated frame and the correct answer.

[0114] The confidence score Sync-C and distance score Sync-D of SyncNet (JS Chung et al., "Out of time: automated lip sync in the wild," In Workshop on Multi-view Lip-reading, ACCV, 2016) measure lip synchronization quality. For a fair comparison, the inventors used the officially released SyncNet1 version.

[0115] - LMD (lip landmark distance) measures the normalized mouth landmark distance between the image generated for lip synchronization evaluation and the correct answer.

[0116] Evaluating raw images using measurement methods designed for reconstruction quality can be unfair, as the size of the generated region varies across the comparative examples and embodiments. For example, Comparative Example 5 synthesizes only a small area around the mouth using its own cropping algorithm, while Comparative Example 4 inpaints the lower half of a tightly cropped face, and inpaints the lower half of an FFHQ-aligned face that includes more background for generation. Since the final dubbed video only needs to contain a tightly zoomed face, the inventors used a cropping method that zooms in the face at the same ratio and then resizes the face to the same resolution for evaluation.

[0117]

[0118]

[0119]

[0120]

[0121] Tables 1 and 2 show quantitative comparisons between each comparative example and the embodiment. In Tables 1 and 2, the highest score is indicated in bold, the next highest score is underlined, and the Sync-C / D score closest to the correct answer is indicated with an *. The embodiment achieved the best performance in all visual quality measures (PSNR, MS-SSIM, LPIPS). Comparative Example 6 achieved significant visual quality compared to the embodiment, but fell short in lip synchronization, as did Sync-C / D and LMD. This is because Comparative Example 6 focuses on personalized facial features and uses only landmark priors for lip synchronization, which can be quite inaccurate in single-image tasks. With regard to lip synchronization, the embodiment achieved the best performance in LMD. Although the fourth comparative example achieved the best performance on Sync-C / D, this result is because it used a SyncNet pre-trained on the same LRS2 (T. Afouras et al., "Deep audio-visual speech recognition", In arXiv:1809.02108, 2018) dataset as the official SyncNet used for evaluation. The third comparative example achieved a high score on Sync-C / D, but generated irregular lip movements for the audio, which resulted in a lower LMD score. Although the embodiment achieved the third best performance on Sync-C / D, it is noteworthy that it was closest to the correct result. The embodiment generated the most natural synchronized lip movements, as supported by the qualitative evaluation. The embodiment can be said to be the only method that can robustly provide both fidelity preservation and synchronization compared to the comparative examples.

[0122] Ablation study of attention-blocks

[0123] Figure 11 illustrates an attention map highlighting significant regions within a reference frame corresponding to patches of a frame generated by an embodiment of the present invention. The attention map is an attention map of the second attention layer located in the attention-block of the fifth block of the decoder. An arbitrary identity is indicated in the first column of each row. The patch locations are indicated in green in the upper half of the generated frame, and the corresponding attention maps are indicated in the lower half, on the reference frame. The attention layer of the embodiment consistently focused on globally relevant regions within the reference frame for each local patch. The embodiment demonstrates that, despite the generated mouth shape being different compared to the reference frame, patches around the mouth successfully attended to significant features such as the mouth location and cheeks, as shown in the second and third columns. In contrast, patches unrelated to the details of the mouth primarily attended more to the corresponding regions on the reference face, and less toward the mouth, as shown in the fourth and fifth columns. This phenomenon indicates that the attention-block successfully attended to detailed information in the oral region by utilizing global contextual understanding and semantic knowledge.

[0124]

[0125]

[0126] Table 3 shows the results of a quantitative evaluation of the performance of the talking head generation model by changing each configuration on the LRW validation dataset. PSNR and LPIPS were used to evaluate the perceptual quality compared to the correct answer. To evaluate lip-synchronization accuracy, the similarity of [Equation 4] was calculated using SyncNet, which was pre-trained on the LRW training dataset. Each comparative example and embodiment was trained for 210K iterations with a batch size of 16, and the resolution was reduced to 96 х 96 for resource efficiency.

[0127] The seventh comparative example is a model that directly injects information of a reference frame into a decoder that does not include an attention block as a baseline. The seventh comparative example uses the connected target frame and reference frame as inputs to a visual encoder (refer to the input design of the fourth comparative example). The seventh comparative example uses information of a reference frame as input to a decoder, and the decoder modulates the decoder input using only audio features (i.e., F w = F a ∈ R 512 ) The quantitative results indicate that the generated frames are easily influenced by the reference frame, resulting in a deterioration in synchronization quality.

[0128] Comparative Example 8 obtains style features and content features from reference frames and target frames, respectively, using a style encoder and a content encoder. Comparative Example 8 constructs W spatial features by connecting style features and audio features (i.e., F W = {F a , F s} ∈ R 1024 ) Comparative Example 8 can improve synchronization quality by capturing less structural information from the reference frame and thus reducing the dependence on the mouth shape of the reference frame.

[0129] The ninth comparative example is based on the eighth comparative example. The ninth comparative example adds the style intermediate features of the reference frame to the block of the decoder, and adds the style intermediate features and the decoder intermediate features (i.e., f s + f d ) Even with skip connections of these reference frames, improvements were observed across the perceptual quality measures. However, this simple addition ultimately degraded synchronization quality due to sensitivity to the reference frame.

[0130] The embodiment demonstrated the greatest improvement over the seventh comparative example by combining style modulation and fidelity mapping via attention blocks. The embodiment demonstrated the best quantitative performance by simultaneously learning high-fidelity detail information and maintaining high synchronization.

[0131] While the present invention has been described in detail through representative examples above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims described below but also by all changes or modifications derived from the claims and their equivalent concepts.

Claims

1. Includes storage and processor, The above processor, Using an encoder, Obtain style intermediate features and style features from the reference frame, Obtain audio features from audio data, Obtain content features from the target frame, Using a style-based decoder with attention blocks, Obtaining a first decoder intermediate feature from the above style feature, the above audio feature, and the above content feature, Obtaining a second decoder intermediate feature by performing an attention mechanism on the first decoder intermediate feature and the style intermediate feature, An electronic device for generating a talking head frame from the intermediate features of the second decoder.

2. In paragraph 1, The above processor, Obtain the value and key from the mouth area of ​​the above style middle feature, Obtain a query from the mouth region of the first decoder intermediate feature, An electronic device for generating a talking head, which obtains an attention score from the above value, the above key and the above query.

3. In paragraph 2, The above processor, An electronic device for generating a talking head, which obtains the second decoder intermediate feature by connecting the above attention score with the first decoder intermediate feature.

4. In paragraph 1, The above processor, Generate a style by connecting the above style features and the above audio features, An electronic device for generating a talking head frame from the second decoder intermediate feature through the above style-based modulation.

5. In paragraph 4, The above style features an electronic device for generating talking heads, which is a rare feature.

6. In paragraph 1, The above processor, Obtaining additional content intermediate features from the above target frame, An electronic device for generating a talking head frame by adding the above content intermediate feature to the second decoder intermediate feature. 7.(a) Using an encoder, (a1) a step of obtaining style intermediate features and style features from a reference frame; (a2) a step of obtaining audio features from audio data; (a3) A step of obtaining content features from a target frame; (b) Using a style-based decoder that includes attention blocks, (b1) a step of obtaining a first decoder intermediate feature from the style feature, the audio feature, and the content feature; (b2) a step of obtaining a second decoder intermediate feature by performing an attention mechanism on the first decoder intermediate feature and the style intermediate feature; and (b3) A method of operating an electronic device for generating a talking head, comprising the step of generating a talking head frame from the intermediate feature of the second decoder.

8. In paragraph 7, Step (b2) above, (b2-1) A step of obtaining values ​​and keys from the mouth area of ​​the above style intermediate features; (b2-2) a step of obtaining a query from the mouth region of the first decoder intermediate feature; and (b2-3) A method of operating an electronic device for generating a talking head, comprising the step of obtaining an attention score from the above value, the above key, and the above query.

9. In paragraph 8, Step (b2) above, (b2-4) A method of operating an electronic device for generating a talking head, further comprising the step of obtaining the second decoder intermediate feature by connecting the attention score with the first decoder intermediate feature.

10. In paragraph 7, The above step (b1) is: A step of generating a style by connecting the above style features and the above audio features, Step (b3) above, A method of operating an electronic device for generating a talking head, comprising the step of generating the talking head frame from the second decoder intermediate feature through the style-based modulation.

11. In paragraph 10, A method of operating an electronic device for generating a talking head, wherein the above style feature is a rare feature.

12. In paragraph 7, In the above step (a3), Obtaining additional content intermediate features from the above target frame, Step (b3) above, A method of operating an electronic device for generating a talking head, comprising the step of generating the talking head frame by adding the content intermediate feature to the second decoder intermediate feature.

Images