Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation

Abstract

GAN-based generators are useful for performing image-to-image transformations. GAN models have large storage size and resource usage requirements, making them too large to deploy directly on mobile devices. The system and method define a student GAN model by using knowledge distillation to refine the student generator, which is scaled downward from the teacher GAN model (and generator). Semantic relational knowledge distillation loss is used to transfer semantic knowledge from the teacher's intermediate layer to the student's intermediate layer. The student generator thus defined is stored and executed by mobile devices such as smartphones and laptops to provide an augmented reality experience. Effects on the image are simulated, including makeup, hair, nails, and age-simulation effects.

Description

cross reference

[0001] This application claims priority to U.S. Provisional Application No. 63 / 045,291, filed June 29, 2020, entitled "Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation," and to French Patent Application No. FR 2009478, filed September 18, 2020, entitled "Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation," the entire contents of each of which are incorporated herein by reference where permitted. [Technical Field]

[0002] This application relates to image processing using machine learning and to creating augmented reality through image processing. [Background technology]

[0003] Generative adversarial networks (GANs) have shown significant potential in modeling high-dimensional distributions of image data, especially in image-to-image translation tasks. However, due to the complexity of these tasks, state-of-the-art models often contain a huge amount of parameters, resulting in large model sizes and long inference times.

[0004] Real-world consumer (user) applications incorporating image-to-image transformation tasks are desirable and popular. As an example, real-time virtual makeup applications provide users with an interactive experience in augmented reality with high identity preservation and detailed realism. Ideally, users would like to preview makeup products (e.g., lipstick and eye shadow) without actually trying them on. In addition, virtual try-ons give users the freedom to adjust relevant product attributes (e.g., color and gloss) according to their personal preferences.

[0005] Therefore, it is desirable to provide a GAN-based model for use on typical user devices such as smartphones and tablets to meet user demands and improve the user experience. Summary of the Invention

[0006] In one embodiment, a novel approach applies knowledge distillation with the distillation of a semantic relation preserving matrix. In one embodiment, this matrix, derived from the encoding of teacher features, helps the student model learn better semantic relations. In contrast to existing compression approaches designed for classification tasks, in one embodiment, the method herein is well suited for image-to-image translation tasks on GANs by stabilizing training with guidance.

[0007] In one embodiment, GAN-based generators perform image-to-image translation. GAN models have large storage size and resource usage requirements, making them too large to deploy directly on mobile devices. The system and method define a student GAN model with a student generator that is scaled downward from the teacher GAN model (and generator) using knowledge distillation through conditioning. In one embodiment, a semantic relation knowledge distillation loss is used to transfer semantic knowledge from the teacher's intermediate layer (e.g., the last layer of the teacher's encoder component) to the student's intermediate layer (e.g., the last layer of the student's encoder component). In one embodiment, the student generators thus defined are stored and executed by mobile devices such as smartphones and laptops to provide an augmented reality experience. In one embodiment, simulated effects on the image include makeup, hair, nails, age simulation effects, etc. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is an illustration of the current approach taught herein, with embodiments contrasted with known approaches. [Figure 2] FIG. 1 illustrates an overview of a training pipeline according to one embodiment. [Figure 3] 1 is a graphical representation of the semantic similarity matrix for five samples. [Figure 4] 10 is an array of images for the Horse <- Zebra task to contrast the outputs of the teacher model, student model, prior model, and current model using selected input examples according to the present disclosure. [Figure 5] 10 is an array of images for the Apples⇔Oranges task contrasting the outputs of the teacher model, student model, prior model, and current model according to the present disclosure using selected input examples. [Figure 6]1 is a row of images of the Horse <- Zebra task for contrasting the outputs of the teacher model, student model, prior model, and current model according to the present disclosure using selected input examples that represent common fault cases. [Figure 7] 1 is an array of images for the Horse <- Zebra task to contrast the outputs of the teacher, student, prior, and current models according to the present disclosure using one example input. The array includes pairings of each image with its enlarged portion showing details. [Figure 8] 1 is an array of images for four different datasets showing an ablation study for the contrast output of a baseline model and a current model, according to one or more embodiments. [Figure 9] Arrays of images from each ablation study according to the specific network architecture experiment. [Figure 10] Arrays of images from each ablation study according to the specific network architecture experiment. [Figure 11] FIG. 1 is a block diagram of a computer system according to one embodiment. [Figure 12] 1 is a flowchart illustrating operations in accordance with the present disclosure.

[0009]

[0010]

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] DETAILED DESCRIPTION OF THE INVENTION

[0019] The present concepts are best explained through specific embodiments thereof, which are described herein with reference to the accompanying drawings, wherein like reference numerals refer to like features throughout. It should be understood that the term "invention," as used herein, is intended to refer to the inventive concepts underlying the embodiments described below, and not simply to the embodiments themselves. Furthermore, it should be understood that the general inventive concepts are not limited to the exemplary embodiments described below, and the following description should be read in that light. 1. Introduction

[0020] Briefly, generative adversarial networks (GANs) [8] are useful for a variety of image tasks, including image-to-image translation, but challenges exist in using GANs on target devices such as mobile devices (e.g., consumer user devices such as smartphones or tablets).

[0021] First, all GAN ​​models are generally too large to be directly deployed on mobile devices with limited storage and computational capabilities. Second, applying existing compression methods designed for regular discriminative convolutional neural networks on GANs does not work well because GANs are quite different from those models in structure and training scheme. Third, the compressed model must maintain a similar performance level compared to the original model in terms of fidelity of the generated images, and must be small enough and fast enough to run on the target device.

[0022] Knowledge distillation techniques are known to transfer knowledge from a teacher model to a student model, and the student model is often chosen to be a narrower or shallower version of the teacher model. Traditional (“vanilla”) knowledge distillation, especially for GANs, faces challenges. Traditional knowledge distillation transfers knowledge from the teacher model to an output, such as a probability distribution for classification. In the case of GANs, the generated image output cannot provide information for training the student model. Instead, the authors proposed transferring the teacher's intermediate representation to the student model at the end of the encoder component. Because the teacher and student models are trained separately, their intermediate representations are mostly in different latent spaces (i.e., they have different numbers of channels). Therefore, to improve traditional knowledge distillation for GANs, the challenge is to properly match the two latent spaces. However, methods such as projecting vectors from one space to the other are often difficult. Therefore, it is necessary to find an alternative method for comparing vectors from the teacher's latent space and the student's latent space.

[0023] More specifically, GANs have shown significant potential in modeling high-dimensional distributions of image data in various vision tasks. Many tasks, such as style transfer [31, 15] and super-resolution

[17] , can be considered image-to-image translation tasks, where a model is trained to map an image from one domain to another. (Meaningful features in an image define "attributes," such as hair color, age, gender, and facial expression, while "attribute values" are specific instances of the attribute's value (e.g., blonde / black / brown / red, 20 / 50, male / female, happy / sad / angry). A "domain" is a set of images that share the same attribute value.) The community has explored solutions for generating high-fidelity images [1, 26] and has shown success in handling unpaired data

[31] . The success of these studies has also led to a general trend in developing mobile applications based on generative models. However, little research has been done on making these models efficient on mobile devices. As a result, state-of-the-art GAN models are often large and slow for resource-limited edge devices. For example, the CycleGAN

[31] model requires 2.69 seconds to process one image with a resolution of 256x256 on a single CPU core of an Intel® Xeon® E5-2686. The model size is 44M.

[0024] Research on convolutional neural networks (CNNs) has led to numerous studies [9, 25, 14, 10, 13] on model compression to improve model efficiency in various computer vision tasks, including classification, object detection, and semantic segmentation. In 2016, Han et al. [9] proposed a three-stage pipeline that first prunes the model by discarding insignificant connections, then quantizes the weights and applies Huffman coding. They successfully reduced the performance of AlexNet

[16] and VGG-16

[28] from 35× to 49× on the ImageNet dataset [7]. This approach uses a complex training pipeline and requires significant manual effort at each stage. [25, 14] have attempted to improve model efficiency by redesigning convolutional layers into separable convolutional layers. Redesigning the network architecture often requires domain experts to explore a large design space and conduct a significant amount of experimentation.

[0025] Later works such as [10, 13] have utilized techniques that perform pruning and network design based on the predictions of trained agents to efficiently reduce the amount of manual effort in neural architecture search and reinforcement learning. After successfully compressing networks for classification tasks, research works [3, 24, 19] further extended these techniques to object detection and semantic segmentation.

[0026] However, the aforementioned solutions do not fit well with GANs, as they typically require an excessive amount of training process and manual design effort. Training generative adversarial networks is usually more difficult and unstable due to the design of an alternating training strategy for the discriminator and generator. Therefore, we explored ways to not only improve the model's efficiency but also provide guidance during training. Hinton et al.

[12] reinvented the concept of knowledge distillation to transfer dark knowledge from an ensemble of teacher models to a single student model and demonstrated the feasibility of using knowledge distillation in model compression. In this setting, implicit, intermediate information, such as probability distributions from the teacher network, can be utilized to guide the student during learning. Given the intuition of this concept, knowledge distillation naturally fits our goal of compressing a GAN generator using a guided training procedure.

[0027] In one embodiment, techniques and methods apply knowledge distillation to image-to-image translation tasks, employing a novel approach to distill information about semantic relationships from teacher to student. In one embodiment, our hypothesis is that, given a feature tensor, feature pixels of the same semantic class may have similar activation patterns, while feature pixels of different semantic classes may differ. Thus, referring to the visualization 100 in FIG. 1 , in one embodiment, in the top row 102, semantic relationships 102A (e.g., similarity, dissimilarity) learned in the teacher model 102B are transferred to the student model 102C using semantic relationship loss preservation 102D. In a high-dimensional space, feature encodings for pixels of the same semantic class may be closer together. The bottom row 104 shows how traditional knowledge distillation works for image-to-image (e.g., 104A to 104B) translation tasks, where traditional knowledge distillation is based on image loss preservation 104C.

[0028] For example, in the horse-to-zebra task, the horse feature tensor may be located close to, but far from, other background pixels such as sky and grass in high-dimensional space. A well-trained supervised model can better capture these correlations between different semantic pixels, both at the dataset and image level. We also demonstrate evidence supporting this intuition.

[0029] In one embodiment, a novel approach that applies knowledge distillation to compress GAN generators on image-to-image translation tasks involves distilling semantic relations. Pixel pairwise similarities of student models are trained in a supervised setting. The potential of this approach is further demonstrated experimentally on five different image-to-image translation benchmark datasets. Our results, both qualitative and quantitative, clearly demonstrate that our approach trains student models to perform as well as, and sometimes better than, the original teacher model. 2. Related Research 2.1 GANs for Image-to-Image Translation

[0030] Along with the success of GANs in modeling high-dimensional data, image-to-image translation tasks are nowadays dominated by GANs due to their superiority in generating high-fidelity and scalable images on different data domains. In

[15] , the authors proposed a model known as "Pix2Pix" that applies conditional GANs to paired image-to-image translation tasks, such as transferring sketches / semantic labels to photos. Tackling the task of unpaired image-to-image translation between two domains, a subsequent work on CycleGAN

[31] proposed constructing two generators that transfer images in both directions and implementing an additional cycle consistency loss during training. StarGAN [5] further extended the capabilities of CycleGAN to domain translation by adding domain-specific attribute vectors to the input while training the generators. 2.2 Semantic Relations that Maintain Knowledge Distillation

[0031] There have been many efforts dedicated to transferring knowledge from teacher models to student models. Hinton et al.

[12] reinvented the concept of knowledge distillation, in which a single student model learns knowledge from a collection of separately trained models. Compared to one-hot output, the information contained within the teacher's soft logits provides more specific knowledge and helps guide the training of the student model. In addition to classification tasks, this technique has also been widely adopted for numerous computer vision tasks, such as object detection and semantic segmentation [2, 20].

[0032] In recent years, learning class relationships has been observed to nontrivially improve model performance in various problems. Many studies [4, 23, 22, 29] have shown progress in applying similarity and relationship learning in knowledge distillation settings. In

[22] and

[23] , they both demonstrated that correlations between instances can be transferred and fully learned in a student model through geometric similarity learning of multiple instances. In

[29] , they empirically demonstrated that similar activation patterns appear on images of the same class (e.g., dogs). Based on this observation, they proposed guiding the student model with a similarity matrix of image instances, computed as the cross product of the teacher feature encodings of a particular layer. However, in image-to-image translation tasks, image-by-image relationships do not provide comprehensive information because they are typically images from the same class (e.g., horses and zebras). Do similar correlation patterns exist between semantic pixels? In this study, we explore the idea of ​​preserving pixel-by-pixel semantic relationships in a student model by transferring this knowledge from the teacher. 2.3 GAN model compression

[0033] Image-to-image translation tasks using generative models are fundamentally different from classification tasks using discriminative models. Traditional model compression approaches are designed for classification tasks that do not adapt well to GANs. Another study

[27] sought to compress GAN models through a co-evolutionary strategy of two generators in CycleGAN

[31] , resulting in a method to efficiently eliminate redundant convolutional filters.

[0034] However, it requires external efforts to maintain the quality of the generated images by controlling the model compression ratio and other hyper-parameters.

[0035] In this work, we aim to achieve better image quality by transferring semantic knowledge from a well-trained teacher model, while achieving effective compression and reducing the effort required for hyperparameter tuning. 3. Method

[0036] In one embodiment, the efficiency of GANs is improved by utilizing knowledge distillation in compressing generators. As discussed in the introduction, training GANs is difficult. In addition to the vanilla knowledge distillation loss, we separate each generator into one encoder and one decoder and formulate a semantic preserving loss based on the feature encoding produced by the encoder. Figure 2 shows a schematic diagram of the distillation strategy 200 in preserving semantic relationships.

[0037] FIG. 2 illustrates a pre-trained teacher model 204 (e.g., encoder Ê) for performing a transformation of an image from a first domain space to a second domain space (e.g., zebra 206 to horse 208). t 204A and generator G^ t Generator G with 204B t 202) to generate a single student model (e.g., encoder E^ S 202A and generator G^S Generator G with 202B S 202). In the CycleGANS framework, for example, a second student model and a corresponding pre-trained teacher model are also simultaneously trained (not shown) to transform images from the second domain space to the first domain space (e.g., from horses to zebras). Also not shown are the associated known functions (losses) for such image to image training (e.g., using CycleGAN or other architectures). For simplicity, only features of the knowledge distillation aspect are shown. For example, in the Pix2Pix framework, a single model pair (e.g., teacher and student models) with respective generators 202B, 204B of FIG. 2 are applicable. As explained below, the student feature encoding F̂ S is shown as 210 in FIG. 2, and the teacher feature coding F̂ t is indicated by 212.

[0038] It is understood that the student and teacher models 202 and 204 have the same general network architecture, with the student 202 scaled downward relative to the teacher 204. The teacher and student models follow a common (i.e., the same) network architecture and are defined in terms of components and connections. "Building blocks" refer to processing units, such as downsample blocks, residual blocks, and upsample blocks, used in the network. "Connections" refer to how each layer / component is connected / routed. The student model is a scaled-down version of the teacher model in terms of width, depth, or both width and depth. Width refers to the number of filters in each layer. Depth refers to the number of residual blocks in the network. For example, the teacher model ngf64, denoted as Resnet9, means that it has a residual depth of 9 and a filter width of 64. Thus, an exemplary student model can be represented as Resnet6, ngf16, where both width and depth are scaled. Another example is Resnet9, ngf32, where the width is preserved but the depth is scaled. Formally, scaling factors can be introduced to characterize two models S and T, where the width and depth numbers are usually constrained to be integers (e.g., positive integers) and can also have other constraints: S(w,d) = T(α*w, β*d), where both are applicable scaling factors.

[0039] In the intermediate layers, we represent semantic relations by computing pairwise activation similarities of pixels in the feature encodings and transferring knowledge via a distillation loss on similarity matrices. This loss can be added on top of the traditional distillation loss (line 214) on the final generated image. The semantic relation activation matrix is ​​computed as the cross product of the feature encodings. The distillation loss is used to compare the teacher's activation matrix with the student's. In this section, we discuss in detail how to apply vanilla knowledge distillation and semantic-preserving distillation to GANs. 3.1 Vanilla Knowledge Distillation of GANs

[0040] In traditional knowledge distillation (e.g., as used in discriminative rather than generative models), the task is formulated as follows:

number

[0041] In a generative adversarial training setting, an exemplary approach for applying knowledge distillation is to generate a trained image G t (x) and student image G S (x) would introduce another minimax game between

number

number

[0042] Previous studies [31, 15] have shown the benefits of blending the GAN goal with other traditional losses such as L1. Therefore, we applied vanilla knowledge distillation by computing a traditional reconstruction loss that compares the teacher output with the student output. For example, in CycleGAN

[31] , the original loss is weighted between two GAN losses and one cycle consistency loss. We add the distillation loss only to the cycle consistency loss, which is an L1 norm loss. Our vanilla knowledge distillation setting has the following goals:

number

[31] . We apply a similar setup in Pix2Pix

[15] training. 3.2 Semantic preservation loss

[0043] The generator G is considered to consist of two parts: an encoder E^ that encodes the input image, and a generator G^ that decodes and generates the output image. i is the ith input image x i is the output image of y i =G(x)=G^(E^(x i Note that it is expressed as

[0044] Semantic Relation Activation Matrix. Figure 2 shows the semantic relation activation matrix determination block 215, including the operations described herein. Tung and Mori

[29] showed interesting and distinct activation patterns between image instances of different classes relative to image instances of the same class. However, in image-to-image translation tasks, the correlation of instances usually contains less information than those from the same class (e.g., horse, orange). Our hypothesis is that similarities and dissimilarities may similarly exist in the feature encodings of different semantic pixels, which is also more informative for image-to-image translation tasks. A distillation loss can be introduced to penalize the difference between the teacher's and student's encoded similarities. We represent this activation matrix by the cross product of the feature encoding F1 (e.g., 212 and 214) similar to [30, 29]. Here, the feature encoding F (i) Define ,be the output matrix of the i-th image example in the last layer of the encoder E^:

number

number

[0045] Figure 3 provides evidence supporting the intuition of the semantic relationship activation matrix. Figure 3 is an array of graphical representations 300 to enhance the interpretability of semantic similarity mapping. Pixels are grouped and aligned together based on their semantic class. Brighter colors (lighter shades of grayscale) indicate higher correlation. The teacher model shows similarity for semantic pixels within the same semantic class (diagonal block matrices) and dissimilarity across different semantic classes (o-diagonal block matrices). This is consistent with our hypothesis that the teacher model displays clearer semantic relationships than the student model. Regarding Figure 3, we sample five horse and zebra images from the COCO dataset

[18] , which provides ground truth segmentation masks, and generate all corresponding teacher and student activation matrices A using Equation 7. To clearly show the different activation patterns, we group values ​​for pixels of the same semantic class. Clear blockwise patterns in the teacher model indicate that pixels of the same semantic class are much more similar compared to pixels of different classes. On the other hand, this pattern is less observable in the student model trained without distillation. This empirical finding strongly supports our hypothesis that there are certain relationship patterns that can be explicitly transferred from the teacher network to the student network. Second, the activation matrix A is independent of the number of channels in the features F, which is important when F is distributed across different feature spaces. t and F S This avoids the difficulty of introducing handcrafted feature losses to match

[0046] Meaning maintenance distillation loss L SP (line 220 in Figure 2) is defined as the L1 loss between two activation matrices:

number

[0047] In preliminary experiments, we also tried L2 loss when matching two matrices, but did not observe any significant improvement. At that time, our complete goal was

number

[0048] Setup. To illustrate the effectiveness of our method for GAN compression, we qualitatively and quantitatively evaluate it on five benchmark image-to-image translation datasets, including horse⇔zebra, summer⇔winter, apple⇔orange, tiger⇔leopard, and labeled photos of cityscapes.

[0049] We performed a fair comparison by following the CycleGAN implementation and setup from the official PyTorch implementation. (The official PyTorch implementation of CycleGAN is available at the URL: github.com / junyanz / PyTorch-CycleGAN-and-pix2pix.) Specifically, the supervised generator sequentially stacks one 7x7 stride-1 convolutional layer, two 3x3 stride-2 convolutional layers, six or nine residual blocks, two 3x3 stride-2 transposed convolutional layers, and one final 7x7 stride-1 convolutional layer. The student generator has the same architecture as the teacher generator, but is two or four times narrower for each layer, depending on the dataset it was trained on.

[0050] Because the teacher and student generators share the same structure in the downsampling and upsampling sections, we use the number of residual blocks and the number of filters in the first convolutional layer to specify the generator architecture. This convention defines both the depth and width of the model. Specifically, we use Resnet9, ngf64 and Resnet9, ngf16 (scaling factor of 4) as our primary teacher-student model pairs for all datasets except the horse⇔zebra dataset, where Resnet9, ngf32 (scaling factor of 2) is used for the student model. It should be understood that the "9" in Resnet9 refers to the residual block count (6 or 9) as described above, and the parameter ngfNN refers to the setting for the number of filters in the first layer of the generator. It should be understood that there is a balance between scaling and performance. Experiments can be conducted to scale down from the teacher model, monitor one or more performance measures for the student (e.g., perform a comparison between the student and the teacher), and select a scaling factor that produces the desired student performance measure.

[0051] Since the cityscape dataset is essentially a paired dataset of Street View photo images and their corresponding semantic segmentation labels, we also conducted experiments in the Pix2Pix setting. The Pix2Pix framework is similar to the CycleGAN framework in terms of the setting for knowledge distillation. The teacher and student generators in our Pix2Pix experiments have a UNet structure

[15] . The UNet structure has only one degree of freedom for shrinking the model (between teacher and student), which is the width. See also the supplementary notes.

[0052] In both the CycleGAN and Pix2Pix embodiments herein, the classifier network follows the PatchGAN classifier

[15] structure. For all datasets (each embodiment), the models were trained and evaluated on images with a resolution of 256x256.

[0053] It will be appreciated that it is not necessary to utilize an existing teacher model, such as a publicly or privately available model, but it can be advantageous, as these are generally pre-trained for the desired task.

[0054] Quantitative Evaluation Metrics. We employ the Fréchet Inception Distance (FID)

[11] for the horse vs. zebra, summer vs. winter, apple vs. orange, and tiger vs. leopard datasets. FID calculates the Wasserstein-2 distance between the feature maps extracted by the Inception Network from the generated image and the real image. As a distance measure, the higher the correlation between the synthetic and real images, the lower the score. For the cityscape labels vs. photos dataset [6], we followed the evaluation method used by Isola et al.

[15] and used the FCN score. This method uses a pre-trained semantic classifier, the FCN-8s network, to score synthetic images using standard segmentation evaluation metrics from cityscape benchmarks, including mean pixel precision, mean class precision, and mean class intersection over the union (IoU).

[0055] Quantitative Comparison. Table 1 lists experiments performed on four unpaired datasets trained with CycleGAN. We compare our results with two previous studies [27, 21] on pruning and different settings of our design. As a reference for the compression ratio, a table of the calculated model size, number of parameters, memory usage, and number of FLOPs is provided in Table 2. [Table 1]

[0056] In Table 1, the reference / baseline FID values ​​are shown in the top row, while the values ​​for our method variations are shown in the bottom row. Experiments were conducted on datasets ranging from horse to zebra (h → z, z → h), summer to winter (s → w, w → s), apple to orange (a → o, o → a), and tiger to leopard (t → l, l → t). Lower is better. Both Co-evolutionary

[27] and ThiNet

[27] apply pruning, but Co-evolutionary is specifically designed to compress CycleGANs, while ThiNet is a pruning method adapted from classification tasks. For a fair comparison with Co-evolutionary and ThiNet, the models compared above have similar model sizes and computational requirements (see Table 2). [Table 2]

[0057] In Table 2, the computation and memory results of the model for the main experiment are shown, where T is the teacher and S1 and S2 are the student references, respectively. Our model achieved superior performance in all tasks compared to Coevolution and ThiNet, with smaller / similar model sizes and computations. For the remaining datasets, we select S1 at hz and S2. The selection is based on the gap between the teacher and student baseline performance.

[0058] We present a variation of the CycleGAN method: 1) we introduce an intermediate distillation loss to the fake images generated by the first generator of the cycle and calculate the L1 norm difference between the teacher's generated images and the student's. This is denoted as intermediate KD. 2) We experimented with semantic relationship preserving losses in the two parts of the cycle. "Semantic preserving (SP)" indicates that we only apply the semantic distillation loss to the first generator of the cycle (i.e., γ2 = 0 in Eq. (9)). "Two-way SP" indicates that we apply the semantic distillation loss to both generators in the cycle. Also, "+" indicates that we add vanilla KD.

[0059] While all the compared models achieve similar performance on the s⇔w dataset, our method achieves significantly better performance than other methods on other datasets. By adding the proposed distillation loss to both generators, we significantly improved performance from vanilla knowledge distillation and even outperformed the original teacher model on several tasks. We further demonstrate visual evidence in the discussion below. However, no performance increase was observed for the summer-to-winter task (s⇔w), which we attribute to the baseline student model being barely numerically different from the teacher model. There is limited space and knowledge for improvement to occur. Furthermore, we conducted experiments on the cityscape dataset and present the FCN scores in Table 3. Interestingly, we notice a dramatic increase in FCN scores when applying the proposed method, yet only similar or slightly better image quality is observed compared to the original model (see Supplementary Note). We believe that the proposed semantic-preserving loss responds strongly to this semantic segmentation dataset by making pixels more recognizable in a semantic manner. [Table 3] Qualitative results

[0060] In this section, we present visual observations on images generated from the model and the reference model. To compare the results with

[27] , we generated images using the model on selected input images shown in Figure 4. Figure 4 therefore shows an array of images 400 for comparing the output of a teacher model, a prior model from

[27] , and the current model according to the present disclosure. The top row displays the input horse image and a zebra image generated using each model. The bottom row displays the input zebra image and a horse image generated using each model. Clearly, the images generated using the current model (the rightmost column in Figure 4) contain more realistic horses / zebras and minimize artifacts in the background.

[0061] In the previous discussion, we mentioned the possibility that adding the proposed semantic preserving loss could enable the student model to outperform the teacher model, using the numerical evidence in Table 1. The extra guidance signal from the teacher's pairwise semantic preserving activations encourages the student not only to learn more intra-pixel relationships within a particular image, but also to learn the semantic understanding of the entire training population. Furthermore, this technique accelerates the learning of the classifier toward capturing more details at an early stage. Incorporating both effects allows the student model to outperform the teacher model in certain cases.

[0062] Figure 5 shows an array of images 500 for contrast output. Figure 5 shows two important examples of the apples⇔oranges task, where the proposed method achieves exceptionally good results. Clear and realistic textures are generated using our method, even outperforming the supervised one.

[0063] Figure 6 shows an array of images 600 for contrast output. An interesting example shown in Figure 6 is from a common failure case of CycleGAN, where CycleGAN has multiple objects to transfer. However, our method provides significant improvement in converting more objects in the generated image, and more zebras are corrected as horses. Another interesting finding of the proposed method is the observation of significantly better detail and texture retention across different tasks.

[0064] Figure 7 is an array of images 700 with zoomed in sections for contrast output. In Figure 7, more detailed and realistic eyes are maintained from the input horse image to the generated zebra image. We also provide examples in Figure 8, an array of images 800 from different datasets, in an ablation study manner. Examples from multiple datasets compare results from baseline models with variations in our approach. 4.2 Different Architectures

[0065] We also provide evidence that our method is scalable to other types of network structures. For the horse-to-zebra task, we replace the Resnet9 generator with the Resnet6 generator for both the teacher and student models. The FID evaluation is shown in Table 4, where we show the FID values ​​of the Resnet6 generator on the horse-to-zebra dataset. Our method still achieves the greatest improvement over the others, but it is not as significant as in the case of Resnet9. We speculate that reducing the three residual blocks from the encoder impairs the expressive power of the teacher model to extract more semantic relationships, which limits the amount of knowledge that can be transferred during the training process. We also conducted additional experiments on paired data using UNet

[15] , a cityscape dataset mapping from semantic labels to Street View photos. The UNet generator produces an encoder-decoder structure that encodes the input through eight 4x4 stride-2 convolutional layers and decodes it through eight 4x4 stride-2 transposed convolutional layers. Skip-connections are used as in the regular UNet scheme. [Table 4] [Table 5]

[0066] In the case of the Resnet generator, the teacher encoding output from the bottleneck layer with a spatial resolution of 64x64 is distilled to guide student training. Because the UNet encoder downsamples the input to a spatial resolution of 1x1 in the bottleneck layer, the desired spatial semantic information is missing in the bottleneck layer. Therefore, we choose to distill the semantic relation activation matrices in layers 2 and 3, which have spatial dimensions of 64x64 and 32x32, respectively. We present the FCN score results for different models on the cityscape dataset through Pix2Pix training in Table 5. The feature encoding extracted from layer 2 of UNet256 has a spatial resolution of 64x64, and the feature encoding from layer 3 has a spatial resolution of 32x32.

[0067] In one embodiment, the highest mean pixel accuracy for distillation is found at layer 3, but similar results are obtained for the mean class IoU at both layers. Detailed model compression rates and visual results are provided in the Supplementary Notes. 5. Supplementary explanation 5.1 Knowledge distillation objective function for Pix2Pix experiments

[0068] Vanilla Knowledge Distillation. The CycleGAN framework includes two generators and a cycle consistency loss that is applied to the task. The Pix2Pix framework transforms in only one direction using a transformation trained with paired data in a supervised manner. By analogy with how vanilla knowledge distillation is applied to classification tasks, the objective function in the Pix2Pix framework has the following form:

number

[0069] Semantic Preserving Knowledge Distillation. Built on the vanilla knowledge distillation goal, the semantic preserving knowledge distillation loss is added directly to the above objective function:

number

[0070] The teacher and student models used in the Pix2Pix experiment with computation and memory statistics are shown in Table 6. In one embodiment, the selection is made based on the gap between the teacher's and student's baseline performance. [Table 6] 5.3 Qualitative results on cityscape

[0071] We conducted both paired and unpaired image translation experiments on the cityscape dataset via Pix2Pix and CycleGAN training, respectively. Synthetic Street View images transformed from semantic masks are displayed in Figures 9 and 10, along with the instance segmentation masks generated by FCN-8s. Figure 9 shows an array of 900 images from an ablation study showing Street View images generated using FCN-8s segmentation masks through CycleGAN training on the cityscape dataset. Images generated by our method (last column) significantly reduced artifacts compared to student-generated images. While the teacher generated more realistic images, we observed that our model preserved the semantic class of pixels with respect to the input mask. For example, in the upper right corner, the teacher-generated image only includes buildings within the specified region of trees. Figure 10 shows an array of 1000 images from an ablation study showing Street View images generated using FCN-8s segmentation masks through Pix2Pix training on the cityscape dataset. Among all the generated masks, the model (last column) shows the clearest segmentation mask with clear boundaries for each semantic class. For example, on the right side of the segmented mask, a significant improvement is observed at the boundary between the green belt and the sidewalk. 5.4 Experimental details

[0072] All models are trained on 256x256 input images with a batch size of 1 and optimized using Adam

[32] , an adaptive learning rate optimization algorithm for training deep neural networks. Other settings for GAN training are the same as those for CycleGAN and Pix2Pix.

[0073] The Horse⇔Zebra, Summer⇔Winter, and Apple⇔Orange datasets are downloaded using scripts provided by CycleGAN. The Horse⇔Zebra with segmentation mask sample images used to draw the semantic similarity matrix is ​​downloaded from COCO

[18] . The Tiger⇔Leopard dataset is obtained from ImageNet [7] using the keywords: Tiger⇔Leopard. The Cityscapes dataset is downloaded from the official website (URL: cityscapes-dataset.com).

[0074] The FID score implementation is adapted from the PyTorch port version of its official implementation (URL: github.com / mseitzer / PyTorch-fid). The FCN score calculation is provided in the official Pix2Pix Torch implementation (URL: github.com / phillipi / pix2pix).

[0075] In the vanilla knowledge distillation training, we set λ = 10 and α = 0.05 for all experiments. γ (γ1 = γ2) is set to 0.9 for horse⇔zebra, 0.5 for summer⇔winter, 0.8 for apple⇔orange, 0.2 for tiger⇔leopard, and 0.2 for cityscape in the unpaired transformation experiments. In the paired transformation experiments, γ is set to 1 and λ is set to 100. 6. Application

[0076] In each embodiment, the GAN-based model has a generator for image-to-image transformation to provide any of collection style transfer, object transformation, season transfer, photo enhancement, or other image processing effects. As an example, makeup, hair, nails, or other effects are applied to a source image to transform the source image from a first domain space (e.g., without the desired effect) to a second domain space with the desired effect. The image-to-image transformation can simulate other effects, such as through appropriate training. In one embodiment, such effects are applied to (human) skin and / or hair. In one example, an age simulation effect is applied, for example, to simulate age effects on an image of a person's face.

[0077] In one embodiment, the model with the generator for image-to-image transformation is incorporated into a computer-implemented method (e.g., application) or computing device or system to provide a virtual reality, augmented reality, and / or modified reality experience. In one embodiment, a user can take a selfie image (or video) using a camera-equipped smartphone or tablet, and the generator applies the desired effect in real time for playback or other presentation by the smartphone or tablet.

[0078] Models with generators compressed according to the teachings herein can be loaded and run on commonly available consumer smartphones or tablet devices (e.g., target devices). Experiments were conducted using the following device hardware specifications: Intel® Xeon® CPU E5-2686v4 @ 2.30GHz, profiled with only a single core and single thread. As mentioned above, the teacher model was defined using Resnet9 and ngf64 as the baseline. The new student model was defined using Resnet9 and ngf16. The teacher model size was 44MB, and the student model was 2.8MB. The teacher inference time was 2.69 seconds, and the student inference time was 0.43 seconds. More comparison results are shown in Table 2.

[0079] An example of the use of GANS is shown and described in Applicant's U.S. Patent Application No. 16 / 683,398, filed November 14, 2019, entitled "System and Method for Augmented Reality by Transforming Images Using Conditional Cycle-Matched Generative Adverse Networks (ccGans)," which is incorporated herein by reference.

[0080] In one embodiment, the disclosed techniques and methods include a developer-related method and system for defining (e.g., through training) a student model with a generator for image-to-image transformation that maintains pixel-by-pixel semantic knowledge determined by a teacher model configured for the same task. The student model is a compressed form of the teacher model, reducing the parameters, footprint, and inference time execution associated with the teacher model. Also shown are user-related methods and systems in which the student model (e.g., generator) is used at runtime to process images for image-to-image transformation.

[0081] Those skilled in the art will appreciate that in one embodiment, in addition to developer (e.g., used at training time) and target (e.g., used at inference time) computing device aspects, computer program product aspects are disclosed in which instructions are stored on a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) for configuring a computing device to perform any of the method aspects disclosed herein. Also shown and described are aspects related to an electronic commerce system. A user's computing device, in one embodiment, is configured as a client computing device with respect to the electronic commerce system, and the electronic commerce system stores, for example, computer programs for such client computing devices. Thus, the electronic commerce system has as its component a computer program product, which stores instructions that, when executed by a client computing device (e.g., a processing unit), configures such client computing device. These and other aspects will be apparent.

[0082] 11 is a block diagram of a computer system 1100. In one embodiment, the computer system 1100 comprises multiple computing devices, including, in one embodiment, a server, developer computers (PCs, laptops, etc.), and mobile devices such as smartphones and tablets. S 11. The network model training environment 1102 is shown, comprising hardware and software for defining and constructing a GAN-based student model 1104 having a GAN-based network. The student model 1104 (and student generator 1106) is trained by a teacher generator 1110 (generator G). T ) using knowledge distillation techniques. The teacher model (and student model) are configured for image processing tasks such as image-to-image translation.

[0083] In one embodiment, conditioning uses semantic knowledge preserving loss (a form of distillation loss added to the traditional distillation loss between the teacher and student outputs (images)) to transfer the semantic knowledge developed for the teacher model 1108 to the student model 1104. T ) is transferred to the hidden layers using semantic knowledge-preserving loss. The teacher model and student model are defined according to a common network architecture. The student model is scaled downward relative to the teacher model to reduce size and / or processing resource requirements to enable use of the generator on a target device, such as one of the mobile devices 1110, 1112 (e.g., 1110) of the system 1100.

[0084] In one embodiment, the network model training environment 1102 uses a teacher model that has been pre-trained for the imaging task. The teacher model 1108 is pre-trained, for example, by using an image dataset 1114 stored on a data server 1116. In one embodiment, the teacher model 1108 is a model developed "in-house." In one embodiment, the teacher model 1108 is publicly available, such as via an open source license. Datasets may be similarly developed and available. Depending on the imaging task and the type of network architecture (e.g., supervised), training is supervised and the dataset is annotated according to such training. In other scenarios, training is unsupervised and the data is defined accordingly.

[0085] In one embodiment, the student generator 1106 generates or incorporates an augmented reality (AR) application 1120. While not shown, in one embodiment, the application is developed using an application developer computing device for a particular target device having a particular hardware and software, particularly an operating system configuration. In one embodiment, the AR application 1120 is a native application configured for execution in a particular native environment, such as one defined for a particular operating system (and / or hardware). In one embodiment, the AR application 1120 takes the form of, for example, a browser-based application configured to run in a browser environment on the target device.

[0086] In one embodiment, the AR application 1120 is distributed (e.g., downloaded) by user devices such as mobile devices 1110 and 1112. Native applications are often distributed via an application distribution server 1122 (e.g., a "store" operated by a third-party service), although this is not required.

[0087] In one embodiment, the AR application 1120 is configured to provide an augmented reality experience to the user (e.g., via an interface). For example, effects are applied to images through processing by the student generator 1106. The mobile device includes a camera (not shown) for capturing images (e.g., still images or video images, whether selfies or not). Effects are applied to images, e.g., video images, in a real-time manner (and displayed on a display device of the mobile device) to simulate the effect on the user as the video is captured. As the camera position is changed, effects are applied in response to the captured video images to simulate augmented reality.

[0088] In one embodiment, the computing devices are coupled for communication over one or more networks (eg, 1122), including wireless networks or otherwise, public networks or otherwise.

[0089] By way of example and not limitation, the e-commerce system 1124 is web-based and provides a browser-based AR application 1120A as a component of the e-commerce service provided by the e-commerce system 1124. The e-commerce system 1124 comprises an configured computing device and a data store 1126 (e.g., a database or other configuration). The data store 1126 stores data related to products, services, and related information (e.g., techniques for applying the products). The data store 1126 or another data store (not shown) stores recommendation rules or other forms of product and / or service recommendations to assist a user in selecting from among available products and services. The products and services are presented via a user experience interface displayed on the user's (mobile) computing device. It will be understood that the e-commerce system 1124 may be simplified.

[0090] In one embodiment, browser-based AR application 1120A (or AR application 1120) provides an augmented reality customer experience, such as simulating a product, technique, or service offered or promoted by e-commerce system 1124. It will be appreciated that AR application 1120, in this embodiment, may also be configured to provide e-commerce services, such as via a connection to e-commerce service 1124.

[0091] By way of example, and without limitation, the products include cosmetic (e.g., makeup) products, anti-aging or rejuvenation products, and the services include cosmetic, anti-aging or rejuvenation services. The services include treatments or other procedures. The products or services relate to a part of the human body, such as the face, hair, or nails. In one embodiment, a computing device (e.g., mobile device 1112) configured in this manner provides a facial effects unit 1112A that includes processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the applied-effect source image (e.g., for display) on an e-commerce interface of the computing device facilitated by an e-commerce system. In one embodiment, the facial effects unit 112A uses a generative adversarial network (GAN)-based student generator (GAN) as described herein to generate the applied-effect source image. S In one embodiment, the computing device provides a user experience unit 112B including processing circuitry for determining at least one product or service from the data store 1126 and generating one or more virtual instances of recommendations on an e-commerce interface for purchasing the product or service. In one embodiment, at least one product is associated with a respective facial effect, and the facial effect unit applies the respective facial effect to provide a virtual try-on experience.

[0092] In one embodiment, the user experience unit 112B is configured to present a graphical user interface (e.g., browser-based or otherwise) for interacting with the computing device 1112 and the e-commerce system 1124. In one embodiment, the e-commerce system 1124 is configured to provide AR applications for execution by client computing devices, such as mobile devices (e.g., 1112), and is cooperatively configured to provide e-commerce services to the client computing devices to facilitate (product / service) recommendations and purchases for AR simulations via the client computing devices (e.g., 1112).

[0093] Accordingly, any of the computing devices, particularly mobile devices, may provide a computing device for transforming an image from a first domain space to a second domain space. The computing device may include a Generative Adversarial Network (GAN)-based student generator (GAN) configured to transform the image from the first domain space to the second domain space. S ) and a GAN-based teacher generator (G T ) with respect to the generator G S is scaled downwards and is configured to transform the image from the first domain space to the second domain space. S is the generator G T In one embodiment, the computing device includes a processing unit configured (e.g., via an AR application 1120) to: receive an image; and convert the image to a generator G to obtain a transformed image. S and provides the converted image for display.

[0094] Generator G S The configuration of the generator G T The semantic relation knowledge determined for S To transfer to the generator GT The generator G is trained using semantic relation knowledge distillation loss. S and generator G T is defined according to a common network architecture. S is the generator G T , where width refers to the number of filters in each layer and depth refers to the number of residual blocks in common network architectures.

[0095] In one embodiment, the processing unit is configured to provide the transformed image in an augmented reality interface to simulate an effect applied to the image. In one embodiment, the effect includes any of a makeup effect, a hair effect, a nail effect, and an age simulation effect applied to the image. For example, the image includes an applicable part (e.g., a face, hair, nails, or body part) of a subject, such as a user of the device.

[0096] In one embodiment, the semantic relation knowledge is generated by the generator G T From the intermediate layer of S The packet is forwarded to the intermediate layer.

[0097] In one embodiment, the generator G S The configuration is adjusted as follows: a. Generator G in each hidden layer during conditioning S and generator G T In each of these, the semantic relations are S and generator G T The feature encoding is represented by a similarity matrix defined by computing pairwise activation similarities for the pixels of each feature encoding performed by b. Knowledge is transferred via semantic relation knowledge distillation loss on the similarity matrix.

[0098] In one embodiment, the generator G S The configuration of the generator G Sand generator G T In addition to the traditional distillation loss on the final generated image of G, the semantic relation knowledge distillation loss is added. Each semantic relation activation matrix is ​​calculated as the cross product of each feature encoding, and the semantic relation knowledge distillation loss is calculated by adding the semantic relation knowledge distillation loss to the generator G. S and generator G T In one embodiment, a computing device, such as the mobile device 1110, is configured to perform a method according to the computing device aspects described above. Other aspects, such as computer program product aspects, will be apparent.

[0099] The network model training environment 1102 provides a computing device configured to perform a method, such as a method of configuring by training a GAN-based student generator. It will be understood that embodiments of the computing device aspect of the network model training environment 1102, and any related embodiments, e.g., of a student generator or model, may be applied to the training method aspect, with appropriate adaptations. FIG. 12 shows a flowchart of operations 1200 according to the present specification. In one embodiment, the operations provide a method, such as training (e.g., through conditioning). In step 1202, the operations train a generative adversarial network (GAN)-based student generator (GAN) to transform an image from a first domain space to a second domain space. S ) and the generator G S is a GAN-based supervised generator (G) configured to transform images from a first domain space to a second domain space. T ) and scaled downwards relative to the generator G S is the generator G T In step 1204, the operation is performed by generating a generator G for use by the computing device to transform the image. S Related computing device and computer program product aspects, as well as other aspects, will become apparent. 7. Conclusion

[0100] In various embodiments, we describe model compression for GANs via methods, systems, and techniques that extend traditional knowledge distillation. Semantic relationship knowledge is transferred from a teacher model to a selected student model, which then generates images with better detail and texture after explicitly learning pixel-to-pixel relationships using knowledge distillation, significantly reducing model size and computational requirements. Experiments conducted on five different datasets and three different architectures (e.g., embodiments) quantitatively and qualitatively demonstrate that the proposed methods, systems, and techniques help bring previously incompetent student networks up to the level of their teacher, capable of generating images at significantly higher quality levels. Furthermore, applying methods, systems, and techniques based on knowledge distillation relieves the burden of controlling the pruning rate and carefully structuring the pruned model, which also provides more degrees of freedom in model selection.

[0101] A practical implementation may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing a function, and other ways of combining the features described herein. Several embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided or steps may be eliminated from the described processes, and other components may be added to or removed from the described systems. Accordingly, other aspects are within the scope of the claims.

[0102] Throughout the description and claims of this specification, the words "comprise" and "contain" and variations thereof mean "including but not limited to" and are not intended to exclude other elements, integers, or steps. Throughout this specification, the singular encompasses the plural unless the context requires otherwise. In particular, where the indefinite article is used, it should be understood that the specification contemplates the plural as well as the singular, unless the context requires otherwise.

[0103] It should be understood that any feature, integer, characteristic, or group described in connection with a particular aspect, embodiment, or embodiment of the invention is applicable to any other aspect, embodiment, or embodiment, except where incompatible therewith. All features disclosed herein (including any accompanying claims, abstract, and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any of the foregoing examples or embodiments. The invention extends to any novel one or any novel combination of features disclosed herein (including any accompanying claims, abstract, and drawings) or any novel one or any novel combination of steps of any method or process disclosed. References 1. Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: International Conference on Learning Representations (2019) 2. Chen, G., Choi, W., Yu, X., Han, T., Chandraker, M.: Learning efficient object detection models with knowledge distillation. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 742-751. Curran Associates, Inc. (2017), URL: papers.nips.cc / paper / 6676-learning-efficient-object-detection-models-with-knowledge-distillation.pdf 3. Chen, Y., Yang, T., Zhang, X., Meng, G., Pan, C., Sun, J.: Detnas: Backbone search for object detection (2019) 4. Chen, Y., Wang, N., Zhang, Z.: Darkrank: Accelerating deep metric learning via cross sample similarities transfer. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018) 5. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp.8789-8797 (2018) 6. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213-3223 (2016) 7. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255. Ieee (2009) 8. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672-2680 (2014) 9. Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural network with pruning, trained quantization and human coding. In: Bengio, Y., Le-Cun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (2016), URL: arxiv.org / abs / 1510.00149 10. He, Y., Lin, J., Liu, Z., Wang, H., Li, L.J., Han, S.: Amc: Automl for model compression and acceleration on mobile devices. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 784-800 (2018) 11. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems. pp. 6626-6637 (2017) 12. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015) 13. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1314-1324 (2019) 14. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) 15. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125-1134 (2017) 16. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097-1105 (2012) 17. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4681-4690 (2017) 18. Lin, T., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D.J., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. Lecture Notes in Computer Science, vol. 8693, pp. 740-755. Springer (2014). URL: doi.org / 10.1007 / 978-3-319-10602-1_48 19. Liu, C., Chen, L.C., Schroff, F., Adam, H., Hua, W., Yuille, A.L., Fei-Fei, L.: Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019) 20. Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., Wang, J.: Structured knowledge distillation for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2604-2613 (2019) 21. Luo, J.H., Wu, J., Lin, W.: Thinet: A filter level pruning method for deep neural network compression. In: Proceedings of the IEEE international conference on computer vision. pp. 5058-5066 (2017) 22. Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3967-3976 (2019) 23. Peng, B., Jin, X., Liu, J., Li, D., Wu, Y., Liu, Y., Zhou, S., Zhang, Z.: Correlation congruence for knowledge distillation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5007-5016 (2019) 24. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018) 25. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510-4520 (2018) 26. Shaham, T.R., Dekel, T., Michaeli, T.: Singan: Learning a generative model from a single natural image. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 4570-4580 (2019) 27. Shu, H., Wang, Y., Jia, X., Han, K., Chen, H., Xu, C., Tian, Q., Xu, C.: Co-evolutionary compression for unpaired image translation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3235-3244 (2019) 28. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015) 29. Tung, F., Mori, G.: Similarity-preserving knowledge distillation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1365-1374 (2019) 30. Zagoruyko, S., Komodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016) 31. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223-2232 (2017) 32. Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. arXiv:1412.6980v9 (2014) <Other> <Means> The computing device of technical idea 1 is for transforming an image from a first domain space to a second domain space, and the storage unit includes a generative adversarial network (GAN)-based student generator (GAN) configured to transform the image from the first domain space to the second domain space. S ) and store its student generator (G S ) is a GAN-based supervised generator (G T ) and configured to transform the image from the first domain space to the second domain space, and the generator G S is the generator G T a processing unit for receiving the image and for generating the image in a generator G S , retrieving the transformed image, and providing the transformed image for display. The computing device of the technical concept 2 is the computing device according to the technical concept 1, further comprising the generator G S The configuration of the generator G T The semantic relation knowledge determined with respect to S to the generator G T The semantic relation knowledge is conditioned using distillation loss. The computing device of the technical idea 3 is the computing device according to the technical idea 1 or 2, further comprising the generator G S and the generator G T is defined according to a common network architecture. The computing device of the technical concept 4 is the computing device according to the technical concept 3, further comprising the generator G S is the generator G T , where the width refers to the number of filters in each layer and the depth refers to the number of residual blocks in the common network architecture. A computing device of technical idea 5 is a computing device according to any one of technical ideas 1 to 4, wherein the processing unit is configured to provide the transformed image to an augmented reality interface to simulate an effect applied to the image. The computing device of technical idea 6 is a computing device according to technical idea 5, wherein the effect is any of a makeup effect, a hair effect, a nail effect, and an age simulation effect applied to the image including applicable parts of the subject. The computing device of Technical Idea 7 is the computing device according to any one of Technical Ideas 1 to 6, which includes a mobile device. The computing device of technical idea 8 is the computing device according to any one of technical ideas 1 to 7, wherein the semantic relation knowledge is generated by the generator G T from the intermediate layer of the generator G S The packet is forwarded to the intermediate layer. The computing device of the technical idea 9 is the computing device of the technical idea 8, further comprising: a. a generator G in each intermediate layer during conditioning S and generator G T In each of the semantic relations, the generator G S and the generator G T b. The knowledge is transferred to the generator G by using a semantic relation knowledge distillation loss on the similarity matrix. S is configured. The computing device of technical idea 10 is the computing device of technical idea 9, further comprising the generator G S The configuration of the generator G S and the generator G T In addition to the traditional distillation loss on the final generated image, the semantic relation knowledge distillation loss is added. The computing device of technical idea 11 is the computing device of technical idea 9 or 10, wherein each semantic relation activation matrix is ​​calculated as a cross product of the respective feature encodings, and the semantic relation knowledge distillation loss is calculated by the generator G S and the generator G T The semantic relation activation matrices of the respective sets are compared. The method of technical idea 12 includes a generative adversarial network (GAN)-based student generator (GAN) configured to transform an image from a first domain space to a second domain space. S ) and its student generator (G S ) is a GAN-based supervised generator (G T ) and configured to transform the image from the first domain space to the second domain space, and the generator G S is the generator G T maintains the semantic relation knowledge determined for the generator G S a generator G for use by a computing device to transform an image S to provide. The method of Technical Idea 13 is the method of Technical Idea 12, further comprising the steps of: S The configuration of the generator G T The semantic relation knowledge determined with respect to S to the generator G T The semantic relation knowledge is conditioned using distillation loss. The method of Technical Idea 14 is the method of Technical Idea 12 or 13, further comprising the steps of: S and the generator G T is defined according to a common network architecture. The method of Technical Idea 15 is the method of Technical Idea 14, further comprising the steps of: S is the generator G T , where the width refers to the number of filters in each layer and the depth refers to the number of residual blocks in the common network architecture. A method of technical idea 16 is a method according to any one of technical ideas 12 to 15, wherein the computing device is configured to provide the transformed image to an augmented reality interface to simulate an effect applied to the image. The method of Technical Idea 17 is the method of Technical Idea 16, wherein the effect comprises any of a makeup effect, a hair effect, a nail effect, and an age simulation effect applied to the image including applicable parts of the subject. The method of technical idea 18 is the method of any one of technical ideas 12 to 17, wherein the computing device includes a mobile device. The method of technical idea 19 is the method of any one of technical ideas 12 to 18, wherein the semantic relation knowledge is generated by the generator G T from the intermediate layer of the generator G S The packet is forwarded to the intermediate layer. The method of technical idea 20 is the method of technical idea 19 which is dependent on technical idea 13, further comprising: a. S and generator G T In each of the semantic relations, the generator G S and the generator G T b. The knowledge is transferred to the generator G by using a semantic relation knowledge distillation loss on the similarity matrix. S is configured. The method of Technical Idea 21 is the method of Technical Idea 20, further comprising: S The configuration of the generator G S and the generator G T In addition to the traditional distillation loss on the final generated image, the semantic relation knowledge distillation loss is added. The method of technical idea 22 is the method of technical idea 20 or 21, wherein each semantic relation activation matrix is ​​calculated as a cross product of each feature encoding, and the semantic relation knowledge distillation loss is calculated by the generator G S and the generator G T The semantic relation activation matrices of the respective sets are compared. The method of Technical Idea 23 is the method of any one of Technical Ideas 12 to 22, further comprising the steps of: T is pre-trained to transform images from the first domain space to the second domain space prior to the conditioning. The method of technical idea 24 is the method of any one of technical ideas 1 to 23, wherein the generator G S is conditioned with a second student generator, and the second student generator is conditioned according to the method of using a second teacher generator to transform an image from the second domain space to the first domain space. The computing device of technical idea 25 includes a facial effect unit including processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the applied effect source image on an e-commerce interface, the facial effect unit including a generative adversarial network (GAN)-based adversarial generator (GAN) for transforming the image from a first domain space to a second domain space to which the at least one facial effect is applied. S ) and the generator G S is a GAN-based supervised generator (G T ), and configured to transform the image from the first domain space to the second domain space. The computing device of Technical Idea 26 is the computing device of Technical Idea 25, further comprising the generator GS is the generator G T Maintain the semantic relation knowledge determined for the The computing device of technical idea 27 is the computing device according to technical idea 25 or 26, wherein the user experience unit includes a processing circuit for determining at least one product from a data store and generating one or more virtual instances of recommendations on an e-commerce interface for purchasing the product, wherein the at least one product is associated with a respective facial effect, and the facial effect unit applies the respective facial effect to provide an experiential virtual try-on experience. The computing device of technical idea 28 is a computing device according to any one of technical ideas 25 to 27, including the computing device according to any one of technical ideas 1 to 11. A computer program product of technical idea 29 includes a non-transitory storage device storing instructions that, when executed by a client computing device, configure the client computing device, wherein the facial effect unit includes processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the applied effect source image on an e-commerce interface, the facial effect unit including a generative adversarial network (GAN)-based student generator (GAN) for transforming the image from a first domain space to a second domain space to which the at least one facial effect is applied. S ) and the generator G S is a GAN-based supervised generator (G T ), and configured to transform the image from the first domain space to the second domain space. The computer program product of the technical idea 30 is the computer program product of the technical idea 29, wherein the generator G S is the generator G T Maintain the semantic relation knowledge determined for the The computer program product of technical idea 31 is the computer program product according to technical idea 29 or 30, wherein the instructions cause the client computing device to include a user experience unit that includes a processing circuit for determining at least one product from a data store and generating one or more virtual instances of recommendations on an e-commerce interface for purchasing the product, wherein the at least one product is associated with a respective facial effect, and the facial effect unit applies the respective facial effect to provide an experiential virtual try-on experience. The computer program product of technical idea 32 is a computer program product according to any one of technical ideas 29 to 31, wherein the instructions cause the client computing device to function as a computing device according to any one of technical ideas 1 to 11. The computer program product of technical idea 33 is the computer program product according to any one of technical ideas 29 to 32, wherein the computer program product is a component of an electronic commerce system for communicating with a user device.

Claims

1. A computing device for transforming an image from a first domain space to a second domain space, comprising: a storage unit and a processing unit; The storage unit includes a generative adversarial network (GAN)-based student generator (GAN) configured to transform an image from a first domain space to a second domain space. Ｓ ) and store the student generator (G Ｓ ) is a GAN-based supervised generator (G Ｔ ) and configured to transform the image from the first domain space to the second domain space; Ｓ is the teacher generator G Ｔ a configuration for maintaining semantic relation knowledge determined for the a. The student generator G in each hidden layer during conditioning Ｓ and the teacher generator G Ｔ In each of the above, the semantic relations are Ｓ and the teacher generator G Ｔ It is represented by a similarity matrix defined by computing pairwise activation similarities for each pixel of each feature encoding performed by b. The semantic relation knowledge is transferred to the student generator G via semantic relation knowledge distillation loss on the similarity matrix. Ｓ is configured, The processing unit receives the image and transmits the image to the student generator G Ｓ a computing device configured to provide the transformed image to a display device, obtain a transformed image, and provide the transformed image for display.

2. The student generator G Ｓ and the teacher generator G Ｔ 10. The computing device of claim 1, wherein: is defined according to a common network architecture.

3. The student generator G Ｓ is the teacher generator G Ｔ 3. The computing device of claim 2, wherein the common network architecture is scaled downward in terms of width, depth, or both width and depth relative to the common network architecture, where the width refers to the number of filters in each layer and the depth refers to the number of residual blocks in the common network architecture.

4. 10. The computing device of claim 1, wherein the processing unit is configured to provide the transformed image to an augmented reality interface to simulate an effect applied to the image.

5. 5. The computing device of claim 4, wherein the effects comprise any of makeup effects, hair effects, nail effects, and age simulation effects applied to the image including applicable portions of a subject.

6. The student generator G Ｓ The configuration of the student generator G Ｓ and the teacher generator G Ｔ The computing device of claim 1, wherein the semantic relation knowledge distillation loss is adjusted by adding the semantic relation knowledge distillation loss in addition to the traditional distillation loss on the final generated image.

7. Each semantic relation activation matrix is ​​calculated as the cross product of each of the feature encodings, and the semantic relation knowledge distillation loss is calculated by the student generator G Ｓ and the teacher generator G Ｔ 2. The computing device of claim 1, wherein the semantic relation activation matrix for each of the following is compared:

8. The semantic relation activation matrix is ​​represented by the cross product of the feature encoding F, Feature coding F （ｉ） Let be the output matrix of the i-th image example in the last layer of the encoder Ê. [Equation 1] and H' and W' denote the feature coding height and width, and C ｔ / C ｘ indicates the number of channels, a batch size of 1 is used, and the semantic relation activation matrix A∈R （（Ｈ＾’・Ｗ＾’）×（Ｈ＾’・Ｗ＾’）） is calculated as the cross product of F, and the row-wise L2 normalization is [Equation 2] 8. The computing device of claim 7, wherein the method is performed by:

9. A computing device including a facial effects unit, comprising: The facial effects unit includes processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the applied effect source image on an e-commerce interface, and the facial effects unit includes a generative adversarial network (GAN)-based student generator (GAN) for transforming an image from a first domain space to a second domain space to which the at least one facial effect is applied. Ｓ ) and the student generator G Ｓ is a GAN-based supervised generator (G Ｔ ) and configured to transform the image from the first domain space to the second domain space; The student generator G Ｓ is the teacher generator G Ｔ maintain the semantic relation knowledge determined for a. The student generator G in each hidden layer during conditioning Ｓ and the teacher generator G Ｔ In each of the above, the semantic relations are Ｓ and the teacher generator G Ｔ It is represented by a similarity matrix defined by computing pairwise activation similarities for each pixel of each feature encoding performed by b. The semantic relation knowledge is transferred to the student generator G via semantic relation knowledge distillation loss on the similarity matrix. Ｓ 1. A computing device comprising:

10. The computing device includes a user experience unit, the user experience unit includes processing circuitry for determining at least one product from a data store and generating one or more virtual instances of a recommendation on an e-commerce interface for purchasing the product; 10. The computing device of claim 9, wherein at least one of the products is associated with a respective one of the facial effects, and the facial effects unit applies the respective one of the facial effects to provide an experiential virtual try-on experience.

11. Each semantic relation activation matrix is ​​calculated as the cross product of each of the feature encodings, and the semantic relation knowledge distillation loss is calculated by the student generator G Ｓ and the teacher generator G Ｔ 10. The computing device of claim 9, wherein the semantic relation activation matrix for each of the following is compared:

12. The semantic relation activation matrix is ​​represented by the cross product of the feature encoding F, Feature coding F （ｉ） Let be the output matrix of the i-th image example in the last layer of the encoder Ê. [Equation 3] and H' and W' denote the feature coding height and width, and C ｔ / C ｘ indicates the number of channels, a batch size of 1 is used, and the semantic relation activation matrix A∈R （（Ｈ＾’・Ｗ＾’）×（Ｈ＾’・Ｗ＾’）） is calculated as the cross product of F, and the row-wise L2 normalization is [Equation 4] 12. The computing device of claim 11, wherein the computing device is configured to:

13. a non-transitory storage medium having stored thereon instructions that configure a computing device, the instructions, when executed by the computing device, to cause the computing device to function as including a facial effects unit; The facial effects unit includes processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the applied effect source image on an e-commerce interface, and the facial effects unit includes a generative adversarial network (GAN)-based student generator (GAN) for transforming an image from a first domain space to a second domain space to which the at least one facial effect is applied. Ｓ ) and the student generator G Ｓ is a GAN-based supervised generator (G Ｔ ) and configured to transform the image from the first domain space to the second domain space; The student generator G Ｓ is the teacher generator G Ｔ maintains semantic relation knowledge determined for a. The student generator G in each hidden layer during conditioning Ｓ and the teacher generator G Ｔ In each of the above, the semantic relations are Ｓ and the teacher generator G Ｔ It is represented by a similarity matrix defined by computing pairwise activation similarities for each pixel of each feature encoding performed by b. The semantic relation knowledge is transferred to the student generator G via semantic relation knowledge distillation loss on the similarity matrix. Ｓ A storage medium comprising:

14. The instructions, when executed by the computing device, cause the computing device to function as including a user experience unit; the user experience unit includes processing circuitry for determining at least one product from a data store and generating one or more virtual instances of a recommendation on an e-commerce interface for purchasing the product; 14. The storage medium of claim 13, wherein at least one of the products is associated with a respective one of the facial effects, and the facial effects unit is configured to apply the respective facial effects to function to provide an experiential virtual try-on experience.

15. 15. The storage medium of claim 14, wherein the computing device comprises a client computing device of a consumer user, and the instructions, when executed, configure the client computing device to communicate with an electronic commerce system to determine at least one of the products.

16. The student generator G Ｓ The configuration of the teacher generator G Ｔ The semantic relation knowledge determined with respect to Ｓ To transfer to the teacher generator G Ｔ 16. The storage medium of claim 15, wherein the storage medium is conditioned using semantic relation knowledge distillation loss for

17. 14. The storage medium of claim 13, wherein at least one of the facial effects is a makeup effect or an age simulation effect applied to a subject from the source image.

18. Each semantic relation activation matrix is ​​calculated as the cross product of each of the feature encodings, and the semantic relation knowledge distillation loss is calculated by the student generator G Ｓ and the teacher generator G Ｔ 14. The storage medium of claim 13, wherein the semantic relation activation matrices of each of the following are compared:

19. The semantic relation activation matrix is ​​represented by the cross product of the feature encoding F, Feature coding F （ｉ） Let be the output matrix of the i-th image example in the last layer of the encoder Ê. [Equation 5] and H' and W' denote the feature coding height and width, and C ｔ / C ｘ indicates the number of channels, a batch size of 1 is used, and the semantic relation activation matrix A∈R （（Ｈ＾’・Ｗ＾’）×（Ｈ＾’・Ｗ＾’）） is calculated as the cross product of F, and the row-wise L2 normalization is [Equation 6] 20. The storage medium according to claim 18, wherein the storage medium is

Images