Computing device for head swapping

Abstract

A computing device for head swapping comprises a storage unit and a processor, wherein the processor: generates, by means of an input editing module, an edited image including a source head and a chroma-key region for neck reconstruction; encodes, by means of an encoder, the edited image into a first latent feature; performs, by means of a conditional colorization module, cross-attention on the first latent feature and a color value of a target head so as to generate a second latent feature in which the source head and the chroma-key region are colorized; and decodes, by means of a decoder, the second latent feature so as to output a head swapped image.

Description

Computing device for head swapping

[0001] The present invention relates to a computing device for head swapping. The present invention also relates to a head swapping method performed by the computing device. More specifically, the present invention relates to the generation of a neck and the coloring of a source head during a head swapping operation.

[0002] Deep learning-based face swapping, which replaces a target face with a source face, can generate photorealistic facial images with diverse identities (hereinafter referred to as "IDs") and poses. However, face swapping struggles to generate the source person, as hairstyle and overall head appearance play a crucial role in determining a person's ID.

[0003] The first study to propose a head-swapping task is HeSer (abbreviation for Head Swapper, hereinafter referred to as "HESER") (Changyong Shu et al. Few-shot head swapping in the wild. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 10789-10798, 2022). The head-swapping task aims to seamlessly transplant the entire source head, including the face and hair of the source person, onto the target body while fully preserving the source ID.

[0004] In head-swapping tasks, one of the key challenges is to smoothly restore the region resulting from the difference between the source and target heads (see (A) in Figure 1). Meanwhile, to generate realistic head-swapping results, the color of the source head must be colorized based on the color of the target head. While such conditional colorization is actively being studied in the field of makeup transfer, its application to head-swapping is challenging. This is because makeup transfer alone cannot cover the large pose differences between the source and target heads.

[0005]

[0006] An object of the present invention is to provide a computing device capable of generating high-fidelity and natural head-swapped images.

[0007]

[0008] The present invention is based on a framework, also known as Chroma-HS (HS: Head Swapping), that utilizes chroma-keying during head swapping operations. In other words, the present invention transfers the head swapping operation to a chroma-key setting. Chroma-keying is an image and / or video editing technique that replaces a captured, monochrome (usually green) video background with a desired background.

[0009] The head swapping operation of the present invention includes the operation of generating a background area of ​​a target image through chroma-keying and the operation of generating a foreground area. The foreground area refers to the area where the head-swapped person is located. The operation of generating the foreground area includes (1) the operation of generating a neck that smoothly connects the source head and the target body, and (2) the operation of transferring the color of the target head to the source head and colorizing the neck with reference to the color of the target head.

[0010] The present invention is based on a novel input editing method for tree generation and a conditional coloring method for coloring gray-scaled source heads.

[0011] - The present invention includes an input manipulation module that simulates neck generation. Since the outline of the source head is different from the outline of the target head, the neck between the source head and the target body may appear in various ways. The input manipulation module simulates mismatches that may occur between the source head and the target body by amplifying the neck generation area using image amplification techniques. In addition, the input manipulation module focuses on the neck generation task by filling the background of the target image with the color of the chroma-key screen image.

[0012] - The present invention includes a conditional colorization module that performs a conditional colorization task based on a cross-attention mechanism. Instead of transferring a target image in a pure color space, the conditional colorization module captures cross-attention values ​​between a gray-scaled source head and a target head, thereby colorizing the gray-scaled source head and, further, the neck.

[0013]

[0014] The present invention provides a computing device capable of generating high-fidelity, natural-looking head-swapping images. By combining the present invention with a state-of-the-art facial reconstructor network, the present invention can generate head-swapping results not only from images but also from various green screen videos.

[0015]

[0016] Figure 1 shows the result of head swapping in the chroma-key setting of HESER.

[0017] Figure 2 illustrates head swapping results according to one embodiment of the present invention.

[0018] Figure 3 illustrates the configuration of Chroma-HS according to one embodiment of the present invention.

[0019] Figure 4 visualizes a method for training the generation of input X according to one embodiment of the present invention.

[0020] Figure 5 shows the head swapping results of HESER and the head swapping results of Chroma-HS of the present invention.

[0021] Figure 6 shows the qualitative results of the etching study.

[0022] [Explanation of symbols]

[0023] 10: Chroma-HS

[0024] 100: Input Editing Module

[0025] 200: Encoder

[0026] 300: Conditional Colorization Module

[0027] 400: Decoder

[0028]

[0029] Hereinafter, the present invention will be described in detail. However, the present invention is not limited or restricted by the exemplary embodiments. The objectives and effects of the present invention can be naturally understood or made more clear through the following description, and the objectives and effects of the present invention are not limited solely by the following description. Furthermore, in describing the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, such a detailed description will be omitted.

[0030] The present invention relates to a computing device for head swapping. The present invention also relates to a head swapping method performed by the computing device. The computing device includes a storage unit and a processor. The storage unit stores data required for head swapping, Chroma-HS, and the like. The storage unit includes at least one of volatile memory and non-volatile memory. The processor controls the overall operation of the computing device for head swapping and the storage unit. The processor includes at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0031] The present invention transfers the head swapping operation to a chroma-key configuration. Applying HESER directly to a chroma-key configuration (e.g., a green screen image) presents two problems. Figure 1 illustrates the head swapping result in a chroma-key configuration using HESER.

[0032] - Referring to (C) of Fig. 1, HESER produces results that are indistinguishable from the target and background. Since HESER aims to perform background restoration and target generation simultaneously using the Reference Color Creation Module, the commands for background restoration and target generation are not clear.

[0033] - Referring to (B) of Fig. 1, HESER produces low-fidelity results in the face region. This is because HESER aims to colorize gray-scale input by conveying RGB color values ​​of semantically related regions in the target image using a domain correspondence matrix.

[0034] The present invention aims to solve the incomplete reconstruction problem of HESER illustrated in (A) of Fig. 1 and the low-fidelity problems of HESER illustrated in (B) and (C) of Fig. 1. (D), (E) and (F) of Fig. 1 illustrate high-fidelity results of the present invention corresponding to (A), (B) and (C) of Fig. 1, respectively. Additionally, Fig. 2 illustrates head-swapping results according to an embodiment of the present invention. Fig. 2 (A) illustrates source images, target images and high-fidelity head-swapped images randomly sampled from benchmark data sets (VoxCeleb1, VoxCeleb2 and HDTF described below), and Fig. 2 (B) illustrates the scalability of the present invention for industrial applications combined with a state-of-the-art face reconstruction network.

[0035] Fig. 3 illustrates the configuration of Chroma-HS according to one embodiment of the present invention. Referring to Fig. 3, Chroma-HS (10) includes an input editing module (100), an encoder (200), a conditional colorization module (300), and a decoder (400) as a head swapping framework. Although not shown in Fig. 3, Chroma-HS (10) further includes a final image generation unit and a loss function calculation unit. The input editing module (100) generates an input X from an input image. The input image is a source image I S and target image I T . The encoder (200) encodes the input X into latent features z (dimensions: B × C × h × w) (B: ​​batch size, C: number of channels, h: height, w: width). The conditional colorization module (300) colorizes the source head and neck regions of the input X to form latent representation z. hat generates a latent representation z. The decoder (400) hat Decode the head-swapped image Y hat . The final image generation unit generates the head-swapped image Y hat and target image I TGenerates the final head-swapped image from .

[0036] The input editing module (100) is a source image I S , target image I T And generates an input X, i.e., an edited image, using a chroma-key screen image. The input X includes a first chroma-key region for neck restoration. The input X further includes a second chroma-key region for target body restoration, particularly, target upper body and / or target background restoration. The input editing module (100) generates the second chroma-key region to focus on generating the first chroma-key region for seamlessly connecting the source head and the target body. The first and second chroma-key regions are generated from the chroma-key screen image. Since the chroma-key screen image is composed of a single color, the color of the first and second chroma-key regions is the same as the color of the chroma-key screen image. In one embodiment, the chroma-key screen image is a green screen image, and the first and second chroma-key regions are green.

[0037] The input editing module (100) utilizes various image amplification techniques (e.g., color jitter, affine transformation, and flipping) to mimic mismatches between source heads and target bodies that may occur during head swapping operations during the training phase. In one embodiment, the input editing module (100) uses a random color jitter function C(x) and a random horizontal flipping function T(x) accompanied by a random affine transformation.

[0038] Referring to Fig. 4, a method for training the generation of an input X (dimension: B × 3 × H × W) according to an embodiment of the present invention is described. Fig. 4 visualizes a method for training the generation of an input X according to an embodiment of the present invention. In the dimension of the input X, B, 3, H, and W represent the size of the batch, the number of channels, the height, and the width, respectively, and B, 3, H, and W described below also have the same meaning.

[0039] The generation training method of input X includes a total of four steps. Prior to this, the input editing module (100) generates a source head mask M from the source image using a pre-trained face analysis (Parsing) network (Changqian Yu et al., Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018). S,head Wow Source Upper Body Mask M S,body Extract each of the source upper body mask M S,body is the source head mask M S,head Includes.

[0040]

[0041] The first step is explained with reference to [Mathematical Equation 1] and Eq. (1) of Fig. 4. The input editing module (100) receives the source image I S Training target image I from (dimensions: B × 3 × H × W) T Generate (dimensions: B × 3 × H × W) target image I for training. T It consists of an upper body and a background. The upper body includes the head. The background is green. Here, the background is the training target image I T Refers to the remaining area excluding the upper body. The input editing module (100) is a source image I S The random color jittered source image C(I) is input to the random color jitter function C(x). S ), the randomly color-jittered source image C(I S ) and Source Upper Body Mask M S,body The input editing module (100) generates a randomly colored jittered source upper body image by multiplying the pixels by the green screen image I green(Dimensions: B × 3 × H × W) and inverted mask 1-M of the source upper body mask S,body The input editing module (100) multiplies the randomly color-jittered source upper body image and the green background image to generate a training target image I. T Creates.

[0042]

[0043] The second step is described with reference to [Mathematical Equation 2] and Eq. (2) of Fig. 4. The input editing module (100) generates a gray-scaled source head image I S,gray (Dimension: B × 1 × H × W)(1: number of channels) is generated. The input editing module (100) generates the source image I S Wow Source Head Mask M S,head Multiply the source head image by pixel to generate the source head image, and input the source head image to the gray-scaling function G(x) to generate the gray-scaled source head image I. S,gray Creates.

[0044]

[0045] The third step is explained with reference to [Mathematical Equation 3] and Eq. (3) of Fig. 4. The input editing module (100) inputs a training target upper body image I T,body (Dimension: B × 3 × H × W) is generated. The input editing module (100) generates a training target image I T and the inverse mask 1-d(M) of the extended composite head mask Union,head ) multiplied pixel by pixel to obtain the training target upper body image I T,body Generate a training target upper body image I T,body does not include the head.

[0046] The input editing module (100) is a source head mask M S,head Input the source head mask M into the T(x) function S,headBy amplifying the source target mask M T,head It creates M. That is, M T,head = T(M S,head ) is. The input editing module (100) is a source head mask M S,head Wow Target Head Mask M T,head Composite head mask M by combining Union,head It generates . That is, it is as follows: . The input editing module (100) generates a source head mask M through the T(x) function. S,head Wow Target Head Mask M T,head From this, various composite head masks can be generated, and various first chroma-key regions can be generated in a self-supervised manner via the T(x) function.

[0047] The input editing module (100) is a composite head mask M Union,head The dilated composite head mask d(M) is input into the dilation function d(x). Union,head ) is generated. The extension function is a composite head mask M Union,head Expands the area where 1 is assigned as the mask value within. Therefore, the extended composite head mask d(M Union,head ) The area where the mask value is assigned 1 is the composite head mask M Union,head It includes at least a portion of the area where the mask value is assigned 1 and the surrounding area of ​​that area. The range of the mask extended by the dilation function can be set more specifically by adjusting the size of the Gaussian kernel using the Gaussian kernel. The size of the Gaussian kernel is a hyperparameter.

[0048]

[0049] The fourth step is described with reference to [Mathematical Equation 4] and Eq. (4) of Fig. 4. The input editing module (100) adds three images by channel to generate input X. The three images are gray-scaled and the duplicated source head image I S,gray,hat(Dimensions: B × 3 × H × W), Training target upper body image I T,body and green images for neck restoration, which are referred to as the first image, the second image, and the third image, respectively, hereinafter.

[0050] The input editing module (100) is configured to add the gray-scaled source head image I before channel-by-channel summation. S,gray The first image is generated by duplicating pixel values ​​for each channel by inputting them into the channel-specific replication function R(x). Gray-scaled source head image I S,gray This is to match the number of channels between the first, second, and third images, since the first image has one channel, while the second and third images each have three channels.

[0051] The input editing module (100) excludes the source head mask from the extended composite head mask to create a neck restoration mask M. ip and generate (i.e., M ip = d(M Union,head ) - M S,head ), green screen image I green Fruit Restoration Mask M ip A third image is generated by multiplying pixel by pixel. The third image includes the first chroma-key region of the input X. The input X includes the first chroma-key region and the second chroma-key region as described above. The first chroma-key region of the input X is derived from the chroma-key region of the third image, and the second chroma-key region of the input X is derived from the chroma-key region of the second image and the chroma-key region of the third image. Since the first chroma-key region of the input X is a region for neck restoration as described above, the generation process of the input X can be referred to as a neck generation process.

[0052] The process of generating input X in the inference step will be described. However, the difference from the process of generating input X in the training step will be mainly described. The input editing module (100) receives both a source image and a target image, and the pre-trained face analysis network extracts a target head mask from the input target image. In other words, unlike the training step, the input editing module (100) does not generate a target image from a source image, and does not generate a target head mask from a source head mask. The input editing module (100) generates input X by summing, channel by channel, the gray-scaled and replicated source head image, the target upper body image, and the green image for neck restoration, similar to the fourth step. Here, the target upper body image is generated by multiplying the target image and the inverted mask of the extended composite head mask pixel by pixel, wherein the target image is the input target image, and the target head mask of the composite head mask is the target head mask extracted from the input target image. In addition, the green image for neck restoration is generated by multiplying the green screen image and the neck restoration mask pixel by pixel, and the target head mask required for generating the neck restoration mask is also the target head mask extracted from the input target image.

[0053] The conditional colorization module (300) colorizes the source head and the first chroma-key region based on cross-attention. In particular, the conditional colorization module (300) can colorize the target head and the first chroma-key region with natural color values ​​by referencing the color values ​​of the target head. The configuration of the conditional colorization module (300) will be described in detail below.

[0054] The conditional coloring module (300) includes at least one conditional coloring block. In one embodiment, the conditional coloring module (300) includes a plurality of conditional coloring blocks. The plurality of conditional coloring blocks constitute a sequence. Each conditional coloring block has a sequence number. The sequence number is an index indicating the position or order of a specific conditional coloring block within the sequence.

[0055] The conditional colorization block includes at least one cross-attention layer that performs cross-attention (operation). In one embodiment, the conditional colorization block includes multiple cross-attention layers. The multiple cross-attention layers form a multi-head cross-attention layer that performs cross-attention operations in parallel.

[0056]

[0057] Refer to [Equation 5]. First, in [Equation 5], "·" means dot-production. The ith conditional coloring block (i is a sequence number) is the ith intermediate smoothed feature θ. i (z) (see Fig. 3) and the color reference matrix Φ(I) of the target head T,color ) is input. The cross-attention layer of the i-th conditional colorization block is θ i (z) from the i-th query matrix Q i (Dimension: N × d k ) and extract Φ(I T,color ) from the i-th key matrix K i (Dimension: d × d k ) and the i-th value matrix V i (Dimension: d × d v ) is extracted. The i-th query matrix Q i The dimension of the i-th key matrix K i The dimension of is d k is the same as the i-th value matrix V i dimension d vis d k It is desirable that it is the same as .

[0058] i-th query matrix Q i , the i-th key matrix K i , i-th value matrix V i Each of them is a learnable projection parameter weight matrix (Ashish Vaswani et al., Attention is all you need. Advances in neural information processing systems, 30, 2017) W i,Q (Dimension: d × d φ ), W i,K (Dimension: d × d φ ), W i,V (Dimension: d × d θ ) is used to extract. If the conditional coloring module (300) is composed of multiple conditional coloring blocks, W between the conditional coloring blocks i,Q , W i,K , W i,V can be different. On the other hand, if the conditional colorization block consists of one cross-attention layer, d θ , d φ , d k , d v It is desirable that they are identical.

[0059] - θ i (z) is described in detail. First, the subscript i denotes the sequence number of the conditional coloring block. The conditional coloring module (300) includes a patch embedding unit (not shown). The patch embedding unit divides the latent feature z into multiple patches and flattens each patch to obtain a first intermediate flattened feature θ1(z) (dimension: N Х d) of the latent feature z. θ ) is generated. N and d θrepresents the number of patches and the dimension of each patch, respectively. The first conditional colorization block receives θ1(z) as input, and the source head and the first chroma-key region are the colorized latent representation z 1,hat (= Attention(Q1, K1, V1), [see Equation 7]) is output. The second conditional coloring block outputs the latent representation z 1,hat The intermediate flattened feature θ2(z 1,hat ) is input, and z 2,hat Outputs (= Attention(Q2, K2, V2)). In this way, the ith conditional coloring block is θ i (z) is input, z i,hat (= Attention(Q i , K i , V i )) outputs z, which is output by the i-th conditional coloring block. i,hat The i+1 conditional coloring block receives input θ i+1 (z i,hat ) may be the same.

[0060] - Φ(I T,color ) is described in detail. The conditional colorization module (300) includes a conditional projection embedding unit (Φ) (see FIG. 3). The conditional projection embedding unit (Φ) is a color reference image I of the target head. T,color (Dimension: B × 3 × H × W) is generated and projected onto the latent space to generate the color reference value of the target head. The color reference value of the target head is the color reference matrix Φ(I) of the target head. T,color )(Dimension: d × d φ ) is. d φ is d θ It is desirable that the same is true. Referring to [Mathematical Equation 6] and Eq. (5) of Fig. 4, the color reference image I of the target head T,color Describes the creation process.

[0061]

[0062] Conditional colorization module (300) trains target image I for training in the training phase T The amplified training target image I is input into the T(x) function. T,hat More specifically, we generate a randomly affine transformed and randomly horizontally flipped training target image, i.e., I T,hat = T(I T ) is. The pre-trained face analysis network is amplified with target training images I T,hat Target Head Mask M from T,head The conditional colorization module (300) extracts the amplified training target image I T,hat and target head mask M T,head Multiply by pixel to get the color reference image I of the target head T,color In the inference step, the input target image and the target head mask M are generated. T,head Multiply by pixel to get the color reference image I of the target head T,color Creates.

[0063]

[0064] [Mathematical formula 7] is described in detail. The conditional coloring block obtains an attention value matrix based on [Mathematical formula 7]. The i-th query matrix Q i and the transposed i-th key matrix K i,T Multiply the i-th attention score matrix (dimension: N × d) by the i-th attention score matrix, and multiply each value of the i-th attention score matrix by the dimension of the query matrix and the key matrix, d. k Divide by the square root of the i-th attention score matrix to scale it, and obtain the scaled i-th attention score matrix (dimension: N × d). Input the scaled i-th attention score matrix into the softmax function to obtain the i-th attention distribution matrix (dimension: N × d), and the i-th attention distribution matrix and the i-th value matrix V i Multiply by the i-th attention value matrix (dimension: N × d v) is obtained. For more specific attention operations of the conditional colorization block and patch embeddings for it, please refer to "Ashish Vaswani et al., Attention is all you need. Advances in neural information processing systems, 30, 2017" and "Alexey Dosovitskiy et al., An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020".

[0065]

[0066]

[0067] The decoder (400) outputs the latent representation z output by the last conditional coloring block of the conditional coloring module (300). hat Input the head-swapped image Y hat Generates a head-swapped image Y (see [Equation 8]). hat Silver I S,gray Contains spatial information about the colorized source head and neck.

[0068] The final image generation unit generates the final head-swapped image Y (see [Equation 9]). The final image generation unit generates the head-swapped image Y. hat and extended composite head mask d(M Union,head ) is multiplied pixel by pixel to create a foreground (person) image, and the target image I T and the inverse mask 1-d(M) of the extended composite head mask Union,head ) is multiplied pixel by pixel to generate a background image, and then the final head-swapped image Y is generated by adding the foreground image and the background image.

[0069] The loss function calculation part is the reconstruction loss L rec , perceptual loss Lper and oppositional loss L adv It utilizes the loss function calculation part, which is the discriminator D. I To regularize, hinge loss is used in adversarial learning. The corresponding objective functions are as in [Equations 10] and [Equations 11].

[0070]

[0071]

[0072] Here, Φ is the pre-trained VGG19 network.

[0073]

[0074]

[0075] The overall objective function is as shown in [Equation 14].

[0076]

[0077] Here, λ rec = 10, λ per = 1, λ adv = 1.

[0078] Experimental example

[0079] data set

[0080] (1) The inventors combined the three benchmark data sets below for training and testing of Chroma-HS and HESER, downloaded the original videos, re-cropped and filtered each data set, and resized each image to 256 × 256 pixels.

[0081] 1) VoxCeleb1: A. Nagrani et al., Voxceleb: a large-scale speaker identification dataset. In INTERSPEECH, 2017.

[0082] 2) VoxCeleb2: JS Chung et al, Voxceleb2: Deep speaker recognition. In INTERSPEECH, 2018.

[0083] 3) HDTF: Zhimeng Zhang et al., Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 3661-3670, 2021.

[0084] (2) The inventors used MIT Indoor Scenes as a background for chroma-key processing, selected 6 categories among 67 different indoor categories - auditorium, cloister, library, museum, office, and TV studio - and randomly sampled indoor images for each category (MIT Indoor Scenes: Ariadna Quattoni et al, Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pages 413-420. IEEE, 2009).

[0085] Implementation details

[0086] The Adam optimizer (Diederik P Kingma et al., Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014) with hyperparameters β1=0.9 and β2=0.999 was used for all models, and unbalanced learning rates of 1e-4 and 4e-4 were used for the generator and discriminator, respectively. Spectral normalization (Takeru Miyato et al., Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018) was applied to all models for robust adversarial training. Chroma-HS was trained on four NVIDIA RTX 3090 (24GB) GPUs. Twenty-five different IDs were randomly selected from the test group as source images, and ten different IDs were randomly selected as target images. For the self-head swapping described below, this experimental example followed the training method of the present invention. This enabled the computation of evaluation methods requiring ground truth. To ensure a fair comparison, the inventors retrained HESER, applying the same boosting techniques to both models during training.

[0087] Qualitative comparison

[0088] Figure 5 illustrates the head swapping results of HESER and the head swapping results of Chroma-HS of the present invention. For a fair comparison, we retrained HESER in both the in-the-wild scenario (see the third row) and the green screen scenario (see the fourth row). The head swapping results of HESER and Chroma-HS trained in the green screen scenario (see the fourth and fifth rows) were obtained by testing each model on a green screen. Referring to Figure 5, Chroma-HS of the present invention produced high-fidelity head swapping results with high-quality backgrounds. For HESER, pixel-by-pixel distortion and low-fidelity head colorization were observed in the green screen scenario. HESER produced blurriness and artifacts in the real-world scenario. Furthermore, HESER produced low-fidelity results in the facial region, including inconsistent skin tones, artifacts on glasses, and hair affected by background colors.

[0089] quantitative comparison

[0090] Quantitative Evaluation Methods for Self-Head Swapping

[0091] (1) SSIM (Zhou Wang et al., Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004): Measures the structural similarity between the generated image and the correct image (Ground Truth).

[0092] (2) PSNR: Peak signal-to-noise ratio used to calculate image reconstruction quality.

[0093] (3) LPIPS (Richard Zhang et al., The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018): Perceptual based on AlexNet (Alex Krizhevsky et al., Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017) Measurement method.

[0094] (4) L1: Average L1 distance across pixel-wise comparisons.

[0095] (5) FPS: Frames per second.

[0096] (6) MAC(Multiply-Accumulate Operation): Computational complexity.

[0097] (7) Number of parameters.

[0098]

[0099]

[0100] [Table 1] quantitatively compares the performance of HESER and Chroma-HS. In this experimental example, HESER was trained in both in-the-wild and green screen scenarios, and each model was tested on a green screen. The results are presented in the first and second rows of [Table 1]. Chroma-HS achieved a SSIM score of 0.967 and a PSNR score of 31.00, representing an 8% improvement (SSIM) and a 46% improvement (PSNR) compared to HESER, respectively. These results demonstrate the effectiveness of Chroma-HS in generating high-quality results that are structurally similar to the target image. For LPIPS and L1, this means that Chroma-HS reconstructed the target image. Furthermore, Chroma-HS achieved a speedup of 1.7 times that of HESER, with 51% less computational cost and 27% fewer parameters. These results demonstrate the practical feasibility of Chroma-HS for industrial applications.

[0101]

[0102]

[0103] [Table 2] shows the results of a user study conducted by developers on cross-head swapping (swapping the heads of different IDs). The number of evaluators was 22, and the evaluation scores ranged from 0 to 2. In [Table 2], Config. stands for Configuration, and IW and GS represent the In-the-Wild and Green Screen scenarios, respectively. In [Table 2], Head, Background, and Holistic represent skin color fidelity, background area naturalness, and overall perceptual quality, respectively. In a blind test between HESER and Chroma-HS, most users gave higher scores to the head-swapped results of Chroma-HS.

[0104] Ablation Study

[0105] [Table 3] and Fig. 6 show the results of the etching study performed to evaluate the contributions of the input editing module and the conditional colorization module of Chroma-HS. [Table 3] shows the quantitative results of the etching study. Fig. 6 shows the qualitative results of the etching study. Comparative Examples 1 to 5 are models in which one of the components of Chroma-HS was removed or replaced with a simpler component. Comparative Examples 1 and 2 are models modified to evaluate the contributions of the input editing module, and Comparative Examples 3, 4, and 5 are models modified to evaluate the contributions of the conditional colorization module.

[0106] - Comparative Example 1: X = I S,gray,hat + I T,body It is a model trained with . That is, the input X of Comparative Example 1 is a green screen image I green Does not include.

[0107] - Comparative Example 2 is a model in which T(x) is partially removed from the input editing module.

[0108] - Comparative Example 3 is a model of a simple encoder-decoder structure that does not have both an input editing module and a conditional colorization module.

[0109] - Comparative Example 4 is a model that uses HESER's reference color generation module as an alternative to the conditional colorization module.

[0110] - Comparative Example 5 is a model in which cross-attention is replaced with self-attention in the conditional colorization module.

[0111]

[0112]

[0113] Referring to [Table 3], the embodiment of the present invention achieved the highest performance in all evaluation indicators. Referring to the results of Comparative Example 2, it can be seen that the amplification of the restoration region of the input editing module is a crucial component in simulating mismatches between the source head and the target body. Referring to the results of Comparative Examples 4 and 5, it can be seen that the cross-attention of the conditional colorization module is an essential component in obtaining high-fidelity head-swapped results.

[0114] While the present invention has been described in detail through representative examples above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims described below but also by all changes or modifications derived from the claims and their equivalent concepts.

Claims

1. Includes storage and processor, The above processor, Through the input editing module, an edited image containing the chroma-key area for source head and neck restoration is generated, Through the encoder, the above edited image is encoded as a first latent feature, Through a conditional colorization module, cross-attention is performed on the color values ​​of the first latent feature and the target head to generate a second latent feature in which the source head and the chroma-key region are colorized, A computing device for head swapping, which decodes the second latent feature through a decoder and outputs a head-swapped image.

2. In paragraph 1, The above processor, A computing device for head swapping, which extracts a person image from the head-swapped image through a final image generation unit, extracts a background image from the target image, and then adds the person image and the background image to generate a final head-swapped image.

3. In paragraph 1, The above input editing module, in the training phase, A computing device for head swapping that simulates a mismatch that may occur between the source head and the target upper body by amplifying the chroma-key region by deforming the source head mask.

4. In paragraph 3, The above input editing module, in the training phase, A computing device for head swapping, wherein the source head mask is deformed by applying a random horizontal flip accompanied by a random affine transformation to the source head mask.

5. In paragraph 3, The above input editing module, in the training phase, Amplify the source image, and multiply the amplified source image and the source upper body mask pixel by pixel to generate an amplified source upper body image, A chroma-key background image is generated by multiplying the chroma-key screen image and the inverse mask of the source upper body mask pixel by pixel, A computing device for head swapping, which generates a training target image by adding the amplified source upper body image and the chroma-key background image.

6. In paragraph 5, The above input editing module, in the training phase, Multiplying the source image and the source head mask pixel by pixel to generate a source head image, and gray-scaling the source head image to generate a gray-scaled source head image, Amplifying the source head mask to generate a target head mask, adding and expanding the source head mask and the target head mask to generate an expanded composite head mask, and then multiplying the training target image and the inverted mask of the expanded composite head mask pixel by pixel to generate a training target upper body image. A computing device for head swapping, which generates a neck restoration mask by excluding the source head mask from the extended composite head mask, multiplies the chroma-key screen image and the neck restoration mask pixel by pixel to generate a chroma-key image for neck restoration including the chroma-key area, and adds the chroma-key image for neck restoration, the gray-scaled source head image, and the training target upper body image channel by channel to generate the edited image.

7. In paragraph 1, A computing device for head swapping, wherein the conditional coloring module comprises a plurality of conditional coloring blocks.

8. In paragraph 7, A computing device for head swapping, wherein each of the plurality of conditional colorization blocks performs multi-cross-attention.

Images