Knowledge distillation for semantic relation preservation in image-to-image translation

Abstract

GAN-based generators are useful for performing image-to-image translations. GAN models have large storage sizes and resource requirements, making them too large to be directly deployed on mobile devices. The system and method define a student GAN model with a student generator scaled down from the teacher GAN model (and the generator) using knowledge distillation. A semantic relation knowledge distillation loss is used to transfer semantic knowledge from the teacher's intermediate layer to the student's intermediate layer. The student generator, thus defined, is stored and executed by mobile devices such as smartphones and laptops to provide augmented reality experiences. Effects are simulated on images, including makeup, hair, nails, and age simulations.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and domestic interest to U.S. Provisional Application No. 63 / 045,291, filed June 29, 2020, entitled "Knowledge Extraction for Semantic Relation Preservation in Image-to-Image Translation," and to French Patent Application No. FR2009478, filed September 18, 2020, entitled "Knowledge Extraction for Semantic Relation Preservation in Image-to-Image Translation," the entire contents of which are incorporated herein by reference, where permitted. Technical Field

[0003] This application relates to image processing using machine learning and the creation of augmented reality through image processing. Background Technology

[0004] Generative Adversarial Networks (GANs) have shown great potential in simulating high-dimensional distributions of image data, particularly in image-to-image translation tasks. However, due to the complexity of these tasks, state-of-the-art models typically contain a large number of parameters, resulting in large model sizes and long inference times.

[0005] Real-world consumer (user) applications combining image-to-image translation tasks are desirable and popular. For example, real-time virtual makeup applications provide users with a highly realistic, interactive experience in augmented reality that retains identity and detail. Ideally, users want to preview cosmetics (such as lipstick or eyeshadow) without actually trying them on. Furthermore, virtual try-ons allow users to freely adjust relevant product attributes (such as color and shine) according to their personal preferences.

[0006] Therefore, there is a need to provide GAN-based models for typical user devices (such as smartphones, tablets, etc.) to meet user needs and enhance user experience. Summary of the Invention

[0007] In this implementation, a novel approach combines knowledge extraction with the extraction of a semantic relation preservation matrix. This matrix, derived from the teacher's feature encoding, helps the student model learn better semantic relations. In contrast to existing compression methods designed for classification tasks, this method adapts well to image-to-image translation tasks on GANs through guided stable training.

[0008] In this implementation, the GAN-based generator performs image-to-image transformations. GAN models have large storage sizes and resource requirements, making them too large to be directly deployed on mobile devices. The system and method define a student GAN model with a student generator (scaled down from the teacher GAN model (and the generator) using knowledge distillation) by conditioning. In this implementation, a semantic relation knowledge distillation loss is used to transfer semantic knowledge from the teacher's intermediate layer (e.g., the last layer of the encoder component of the teacher generator) to the student's intermediate layer (e.g., the last layer of the encoder component of the student generator). In this implementation, the student generator, thus defined, is stored and executed by a mobile device such as a smartphone or laptop to provide an augmented reality experience. In this implementation, the effects simulated on the images include makeup, hair, nails, age simulation effects, etc. Attached Figure Description

[0009] Figure 1 This is an illustration of the method taught herein, based on an implementation method compared to known methods.

[0010] Figure 2 This is a diagram illustrating an overview of the training pipeline based on an example.

[0011] Figure 3 It is a graphical representation of the semantic similarity matrix among the 5 samples.

[0012] Figure 4 The outputs of the teacher model, student model, previous model, and current model are compared based on the selected input examples used in this disclosure. Image array for the task.

[0013] Figure 5 The outputs of the teacher model, student model, previous model, and current model are compared based on the selected input examples used in this disclosure. Image array for the task.

[0014] Figure 6 The outputs of the teacher model, student model, previous model, and current model are compared based on the selected input examples used in this disclosure. Image row for the task (showing common error examples).

[0015] Figure 7 This is based on an example input selected according to the use of this disclosure to compare the outputs of the teacher model, student model, previous model, and current model. An image array for the task. This array consists of pairs of corresponding images and their magnified portions showing details.

[0016] Figure 8 This is an ablation study comparing the outputs of a baseline model and the current model using image arrays from four different datasets, based on one or more implementations.

[0017] Figure 9 and 10 It is an array of images from various ablation studies based on experiments with a specific network architecture.

[0018] Figure 11 It is a block diagram of a computer system according to an implementation method.

[0019] Figure 12 This is a flowchart illustrating the embedded operations according to this document.

[0020] This concept is best described through certain embodiments thereof, which are described herein with reference to the accompanying drawings, wherein the same reference numerals denote the same features throughout. It should be understood that, when used herein, the term "inventive" is intended to imply the inventive concept of the embodiments described below, and not merely the embodiments themselves. It should be further understood that the overall inventive concept is not limited to the illustrative embodiments described below, and the following description should be read in this manner. Detailed Implementation

[0021] 1. Introduction

[0022] In short, Generative Adversarial Networks (GANs)[8] are useful in a variety of image tasks, including image-to-image transformation, but there are challenges in using GANs on target devices such as mobile devices (e.g., on consumer user devices such as smartphones or tablets).

[0023] First, GAN models are typically too large to be directly deployed on mobile devices with limited storage and computing power. Second, applying existing compression methods designed for regularized discriminative convolutional neural networks to GANs does not work well because GANs differ significantly from those models in structure and training scheme. Third, in terms of the fidelity of the generated images, the compressed model must maintain a similar performance level to the original model, while being smaller and faster to run on the target device.

[0024] Known knowledge extraction techniques transfer knowledge from the teacher model to the student model, where the student model is typically chosen as a narrower or shallower version of the teacher model. Applying traditional (“ordinary”) knowledge extraction to GANs presents challenges. Traditional knowledge extraction extracts knowledge from the teacher model's output, such as probability distributions used for classification. In the case of GANs, the output as generated images does not provide information to guide the training of the student model. Instead, we propose transferring the teacher's intermediate representations to the student model at the end of the encoder component. Since the teacher and student models are trained separately, their intermediate representations are mostly in different latent spaces (i.e., they have different numbers of channels). Therefore, the challenge lies in properly matching these two latent spaces to improve traditional knowledge extraction on GANs. However, methods like projecting vectors from one space to another are often difficult. Therefore, we need to find another way to compare the vectors in the teacher's latent space and the student's latent space.

[0025] More specifically, GANs have shown great potential in modeling the high-dimensional distribution of image data for a variety of visual tasks. Many of these tasks (such as style transfer [31, 15] and super-resolution

[17] ) are considered image-to-image translation tasks, in which we train a model to map images from one domain to another. (It should be understood that meaningful features in an image define “attributes”, such as hair color, age, gender, facial expression, etc., and “attribute values” are specific examples of attribute values ​​(e.g., blonde / black / brown / red hair, 20 / 50, male / female, happy / sad / angry, etc.). A “domain” is then a set of images that share the same attribute values. The group has been successful in researching solutions for generating high-fidelity images [1, 26] and in handling unpaired data

[31] . The success of these works has also led to a growing trend of developing mobile applications based on generative models. However, little work has been done to make these models efficient on mobile devices. As a result, state-of-the-art GAN models are often bulky and slow on resource-constrained edge devices. For example, the CycleGAN

[31] model takes 2.69 seconds to run on an Intel device.) (Supreme Power) A single CPU core processes an image with a resolution of 256x256, and the model is 44MB in size.

[0026] With the advancements in convolutional neural networks (CNNs), numerous works have been proposed for model compression [9, 25, 14, 10, 13] to improve model efficiency in a variety of computer vision tasks, including classification, object detection, and semantic segmentation. In 2016, Han et al. [9] proposed a three-stage pipeline that first prunes the model by cutting less important connections, then quantizes the weights and applies Huffman coding. They successfully reduced the efficiency of AlexNet

[16] and VGG-16

[28] on the image network dataset [7] by 35 to 49 times. This approach has a complex training pipeline that requires a significant amount of manual intervention at each stage. In [25, 14], efforts were made to improve model efficiency by redesigning convolutional layers into separable convolutional layers. Redesigning network architectures typically requires domain experts to explore a large design space and conduct extensive experiments.

[0027] Subsequent work, such as [10, 13], has leveraged techniques from neural architecture search and reinforcement learning to effectively reduce the amount of manual work by performing pruning and network design based on predictions from trained agents. Following successful results in compressing networks for classification tasks, research [3, 24, 19] has further extended the aforementioned techniques to object detection and semantic segmentation.

[0028] However, the above solutions do not fit well into GANs, as GANs typically require a large amount of training process and manual design work. Training generative adversarial networks is often made more difficult and unstable due to the design of alternating training strategies for discriminators and generators. Therefore, we explored methods that can both improve model efficiency and provide guidance during training. Hinton et al.

[12] reintroduced the concept of knowledge extraction, transforming dark knowledge in the overall teacher model into individual student models, demonstrating the potential of knowledge extraction in model compression. In this case, the student can be guided at training time by utilizing the ambiguous and intermediate information (such as probability distributions) from the teacher network. Given the intuition of this concept, knowledge extraction naturally aligns with our goal of compressing GAN generators by guiding the training process.

[0029] In this implementation, the techniques and methods apply knowledge extraction to image-to-image conversion tasks and employ novel approaches to extract information on semantic relationships from teacher to student. In this implementation, we assume that, given a feature tensor, feature pixels of the same semantic class can have similar activation patterns, while feature pixels of different semantic classes can be dissimilar. Therefore, referencing... Figure 1In the visualization 100, in the implementation, in the top row 102, semantic relation loss preservation 102D is used to transfer semantic relations 102A (e.g., similarity, dissimilarity) learned in the teacher model 102B to the student model 102C. In a high-dimensional space, feature encodings of pixels with the same semantic class can be located closer together. The bottom row 104 shows how conventional knowledge extraction works on image-to-image (e.g., 104A to 104B) transformation tasks, where conventional knowledge extraction is based on image loss preservation 104C.

[0030] For example, in a horse-to-zebra task, the horse's feature tensor might be located closer to but farther from other background pixels (such as the sky and grass in a high-dimensional space). A well-trained teacher model is able to capture these correlations between different semantic pixels better at both the dataset and image levels. We will also present evidence supporting this intuition.

[0031] In the implementation, a novel approach to knowledge extraction, including the extraction of semantic relations, is applied to the compressed GAN generator for image-to-image translation tasks. Pixel pairwise similarity of the student model is trained in a teacher-supervised setting. Furthermore, experiments on five different image-to-image translation benchmark datasets demonstrate the potential of the proposed method. Our results, both qualitative and quantitative, clearly show that our method trains student models to be on par with, and sometimes even better than, the original teacher models.

[0032] 2. Related work

[0033] 2.1 GANs for Image-to-Image Transformation

[0034] With the success of GANs in high-dimensional data modeling, image-to-image translation tasks are now dominated by GANs due to their advantages in generating high-fidelity and scalable images across different data domains. In

[15] , the authors proposed a model known as Pix2Pix, applying conditional GANs to paired image-to-image translation tasks, such as from sketches / semantic labels to photographs. Subsequent work CycleGAN

[31] (which handles unpaired image-to-image translation tasks between two domains) proposed constructing two generators that transfer images in both directions and enforced an additional cycle consistency loss during training. StarGAN [5] further extends the capabilities of CycleGAN to multi-domain translation by adding domain-specific attribute vectors to the input while training the generators.

[0035] 2.2 Knowledge Extraction for Semantic Relation Preservation

[0036] For a long time, people have been working to transfer knowledge from teacher models to student models. Hinton et al.

[12] reinvented the concept of knowledge distillation, in which a single student model learns knowledge from a collection of individually trained models. The information contained in the teacher’s soft logic provides more specific knowledge and helps guide the training of the student model compared to a one-time output. In addition to classification tasks, this idea has been widely applied to many computer vision tasks, such as object detection and semantic segmentation [2, 20].

[0037] Recently, it has been observed that learning class relationships has significantly improved model performance across a variety of problems. Numerous works [4, 23, 22, 29] have demonstrated progress in applying similarity and relationship learning in knowledge extraction contexts. In

[22] and

[23] , they both demonstrated that correlations between instances can be transferred and learned well in student models through geometric similarity learning of multiple instances. In

[29] , they empirically demonstrated that similar activation patterns appear on images of the same category (e.g., dogs). Based on this observation, they proposed guiding students using a similarity matrix of image instances (computed as the outer product of the teacher's feature encodings over certain layers). However, in image-to-image translation tasks, image relationships do not provide comprehensive information because they are often images from the same category (e.g., horses, zebras). Could similar correlation patterns exist between semantic pixels? In this work, we explore the idea of ​​preserving pixel semantic relationships in student models by transferring such knowledge from the teacher.

[0038] 2.3 GAN-based model compression

[0039] Image-to-image conversion tasks using generative models are fundamentally different from classification tasks using discriminative models. Traditional model compression methods are designed for classification tasks and do not fit well into GANs. Another work

[27] focuses on compressing GAN models through a co-evolutionary strategy of two generators in a recurrent GAN

[31] , resulting in an effective method for eliminating redundant convolutional filters.

[0040] However, external efforts are needed to maintain the quality of the generated images by controlling the model compression ratio and other hyperparameters.

[0041] In this work, we aim to reduce the amount of work required for hyperparameter tuning and achieve better image quality, while achieving efficient compression by transferring semantic knowledge from a trained teacher model.

[0042] 3. Methods

[0043] In this implementation, GAN efficiency is improved by leveraging knowledge extraction in the compressed generator. As discussed in the introduction, training GANs is challenging. In addition to the standard knowledge extraction loss, we split each generator into an encoder and a decoder, and formulate a semantic preservation loss based on the feature encoding produced by the encoder. Figure 2 A schematic overview of a distillation strategy 200 that preserves semantic relations is shown.

[0044] Figure 2 It is simplified because it shows a comparison with a pre-trained teacher model 204 (e.g., its generator G). t 202 includes its encoder 204A and Generator 204B) to train a single student model (e.g., its generator G) s 202 includes its encoder 202A and generator 202B) Related knowledge extraction strategies 200 are used to perform image transformations from a first domain space to a second domain space (e.g., zebra 206 to horse 208). In the CycleGAN framework, for example, a second student model and a corresponding pre-trained teacher model are also trained simultaneously (although not shown) to transform images from the second domain space to the first domain space (e.g., from horse to zebra). Also not shown are known features (losses) related to this image-to-image training (e.g., using CycleGAN or other architectures). For simplicity, only features related to knowledge extraction are illustrated. For example, in the Pix2Pix framework, with Figure 2 The corresponding generators 202B and 204B are applicable to individual model pairs (e.g., teacher and student models). Student feature encoding is described below. exist Figure 2 The Chinese character is represented as 210, while the teacher's characteristic code is... It is represented as 212.

[0045] It should be understood that student model 202 and teacher model 204 have the same general network architecture, where student 202 is downscaled relative to teacher 204. The teacher and student models are defined in terms of building blocks and connections based on a common (i.e., identical) network architecture. A "building block" refers to a processing unit used in the network, such as a downsampling block, residual block, or upsampling block. A "connection" refers to the way each layer / building block is connected / routed. The student model is a scaled-down version of the teacher model in terms of width, depth, or both width and depth. Width refers to the number of filters in each layer. Depth refers to the number of residual blocks in the network. For example, a teacher model represented as Resnet9, ngf64 means it has a residual depth of 9 and a filter width of 64. Therefore, an example student model could be represented as Resnet6, ngf16, where both width and depth are scaled. Another example is Resnet9, ngf32, where the width is preserved but the depth is scaled. Formally, considering that the width and depth numbers are usually constrained to integers (e.g., positive integers) and can have other constraints, scaling factors can be introduced to characterize the two models S and T, and can have other constraints: S(w,d)=T(α*w,β*d), where α and β are applicable scaling factors.

[0046] In the intermediate layer, we represent semantic relations by computing pairwise activation similarity on feature-encoded pixels and pass knowledge through a refinement loss on the similarity matrix. This refinement loss can be added to the final generated image in addition to the traditional refinement loss (line 214). The semantic relation activation matrix is ​​computed as the outer product of the feature encodings. The refinement loss is used to compare the teacher's activation matrix with the student's matrix. In this section, we will discuss the details of how we apply general knowledge refinement and semantic preservation refinement on GANs.

[0047] 3.1 General Knowledge Extraction Based on GAN

[0048] In traditional knowledge extraction (e.g., used to differentiate models rather than generate them), the task is formulated as:

[0049]

[0050] Where y i Indicates input x i The basic facts, f θ (x i ) and f t (x i) represent the student model output and the teacher model output, respectively. n is the number of inputs and α is a hyperparameter used to balance the teacher's output and the basic facts. Equation (1) encourages the network to minimize two terms: 1) the loss between the basic facts and the student output, and 2) the loss between the teacher's output and the student's output. The second part of the objective function aims to help students learn ambiguous knowledge for different tasks. For example, in a classification task, soft logic with temperature control is matched between the student and the teacher to encourage the student to imitate the teacher.

[0051] In generative adversarial training, one example of applying knowledge extraction is to use the teacher's generated image G. t (x) and the student's generated image G s Introduce another minimax game between (x):

[0052]

[0053] in

[0054]

[0055] The subscripts t and s indicate the composition of teachers and students. It is a discriminator between student output and real images, and at the same time Distinguish between student output and teacher output, and x and y are real images from each class.

[0056] Previous work [31, 15] has shown the benefits of combining GAN objectives with other conventional losses, such as L1. Therefore, we apply general knowledge refinement by computing the conventional reconstruction loss by comparing the outputs of the teacher and the student. For example, in CycleGAN

[31] , the original loss is weighted between two GAN losses and a cycle consistency loss. We only add the refinement loss to the cycle consistency loss (which is the L1 norm loss). Our general knowledge refinement setting has the following objectives:

[0057]

[0058] Among them G s and F s These represent generators for transferring style classes X to Y and Y to X, respectively. Therefore, X t and Y t The image is a reconstructed image generated by the teacher. The notation is adapted from

[31] . We also apply a similar setup in Pix2Pix

[15] training.

[0059] 3.2 Semantic Preservation Loss

[0060] We consider the generator G to consist of two parts: an encoder that encodes the input image. and a generator that decodes and produces the output image. We noticed y i It is the i-th input x i The output image, where

[0061] Semantic relation activation matrix. Figure 2 The semantic relation activation matrix determination block 215 is shown, which includes the operations described herein. Tung and Mori

[29] demonstrated interestingly different activation patterns between image instances of different categories and image instances of the same category. However, in image-to-image translation tasks, instance relevance contains less information because they often come from the same class (e.g., horse, orange). Our hypothesis is that similarity and dissimilarity may also exist in the feature encoding of different semantic pixels, which is more informative in image-to-image translation tasks. A refinement loss can be introduced to penalize the difference in encoding similarity between teacher and student. We use feature encoding similar to [30, 29] The activation matrix is ​​represented by the outer product of (e.g., 2^12 and 2^14). Here, we encode the features. Defined as encoder The output matrix of the i-th image example in the last layer:

[0062]

[0063]

[0064] Where H′ and W′ represent the feature encoding height and width, respectively, while C t / C x This refers to the number of channels. We use a batch size of 1. Then, we compute the semantic relation activation matrix. As The outer product is then processed by row-by-row L2 normalization.

[0065]

[0066] exist Figure 2 middle, Represented as 216, and It is represented as 218.

[0067] We are Figure 3 The paper presents some evidence to support our intuition about semantic relation activation matrices. Figure 3This is an array of 300 graphical representations used to enhance the interpretability of semantic similarity mapping. Pixels are grouped and aligned according to their semantic categories. Brighter colors (lighter shades in grayscale) represent higher relevance. The teacher model shows the similarity of semantic pixels within the same semantic class (diagonal block matrix) and the dissimilarity between different semantic classes (o-diagonal block matrix). This aligns with our hypothesis that the teacher model reveals clearer semantic relationships than the student model. About Figure 3 We sampled five horse and zebra images from the COCO dataset

[18] , which provided ground truth segmentation masks, and generated all the corresponding teacher and student activation matrices via Equation 7. We grouped pixel values ​​by the same semantic class to clearly reveal different activation patterns. The clear block patterns in the teacher model indicate that pixels of the same semantic class are more similar than pixels of different classes. On the other hand, this pattern is less obvious in the unrefined student model. This empirical finding strongly supports our hypothesis that there exist certain relational patterns that can be explicitly transferred from the teacher network to the student network. Secondly, the activation matrix... With features The number of channels is independent, which avoids introducing handcrafted feature loss to match different feature spaces. and

[0068] We extract our semantic preservation loss ( Figure 2 Line 220) is defined as the L1 loss between two activation matrices:

[0069]

[0070] In our initial experiments, we also tried L2 loss to improve the matching of the two matrices, but no significant improvement was observed. Our overall goal is...

[0071]

[0072] Where A and B represent the generators in each direction, and γ1, γ2, and α are hyperparameters.

[0073] 4. Experiment

[0074] 4.1 Different Image-to-Image Transformation Datasets

[0075] Setup. To illustrate the effectiveness of our method on GAN compression, we qualitatively and quantitatively evaluated it on five benchmark image-to-image translation datasets, including... and urban landscape

[0076] To ensure a fair comparison, we followed the CycleGAN implementation and setup from the official PyTorch implementation. (The official CycleGAN PyTorch implementation is available at: github.com / junyanz / pytorch-CycleGAN-and-pix2pix). Specifically, the teacher generator sequentially stacks a 7x7 stride-1 convolutional layer, two 3x3 stride-2 convolutional layers, six or nine remaining blocks, two 3x3 stride-2 transposed convolutional layers, and a final 7x7 stride-1 convolutional layer. The student generator has the same architecture as the teacher generator, but with each layer being 2 or 4 times narrower, depending on the training dataset.

[0077] Since the teacher and student generators share the same architecture in both downsampling and upsampling, we specify the generator architecture using the number of remaining blocks and filters in the first convolutional layer. This convention defines the depth and width of the model. Specifically, we use ResNet9, NGF64 and ResNet9, NGF16 (scale factor of 4) as our... The dataset contains primary teacher-student model pairs for all datasets other than the one containing ResNet9 and ngf32 (scale factor of 2) for the student model. It should be understood that the "9" in ResNet9 refers to the remaining block count (6 or 9) as described above, and the parameter ngfNN refers to the setting of the number of filters NN in the first layer of the generator. It is understood that a balance exists between scaling and performance. Experiments can be conducted to scale down from the teacher model, monitor one or more student performance measures (e.g., compare between students and teachers), and select a scale factor that produces the desired student performance measure.

[0078] Since the urban landscape dataset is essentially a paired dataset of urban landscape photographs and their corresponding semantic segmentation labels, we also conducted experiments in the Pix2Pix setting. The Pix2Pix framework is similar to the CycleGAN framework in terms of setting up knowledge extraction. In our Pix2Pix experiments, both the teacher and student generators have a UNet structure

[15] . The UNet structure has only one degree of freedom for narrowing the model (between teacher and student), namely the width. See Supplementary Notes.

[0079] In the CycleGAN and Pix2Pix implementations in this paper, the discriminator network follows the PatchGAN discriminator

[15] structure. For all datasets (each implementation), the model is trained and evaluated on images with a resolution of 256x256.

[0080] It should be understood that using existing teacher models, such as publicly or privately available models, is not necessary but advantageous, as these are typically pre-trained for the desired task.

[0081] Quantitative evaluation indicators. We... and The Frachet Initial Distance (FID) was used on the dataset

[11] . FID calculates the Wasserstein-2 distance between feature maps extracted from generated and real images by the initial network. As a distance metric, a lower score is preferred for higher correlation between synthetic and real images. In urban landscapes On the dataset [6], we use FCN scoring in accordance with the evaluation method

[15] used by Isola et al. This method uses a pre-trained semantic classifier FCN-8s network to score synthetic photos using standard segmentation evaluation metrics from urban landscape benchmarks, including average pixel precision, average class precision, and average class intersection (IoU).

[0082] Quantitative Comparison. In Table 1, we list the experiments conducted on four unpaired datasets trained using CycleGAN. We compare our results with two previous works [27, 21] on pruning and different design settings. As a reference for compression ratio, we present a table in Table 2 showing the computational model size, number of parameters, memory usage, and number of FLOPs.

[0083] h→z z→h s→w w→s a→o o→a t→l l→t teacher 84.01 136.85 76.99 74.39 132.37 130.72 76.68 77.60 student 94.95 141.64 76：47 74.90 132.99 137.10 93.98 89.37 ThiNet

[21] 189.28 184.88 81.06 80.17 - - - - Co-evolution

[27] 96.15 157.90 79.16 78.58 - - - - Regular KD 106.10 144.52 80.10 79.33 127.21 135.82 82.04 87.29 Mid-KD 97.20 143.43 77.75 74：67 126.90 133.16 86.82 92.99 +SP 90.65 143.03 78.75 76.21 125.90 132.83 81.53 86.52 +2 direction SP 86：31 140：15 76.59 75.69 121：17 132：83 81：17 80：75

[0084] Table 1

[0085] In Table 1, the FID values ​​for the reference / baseline are shown on the top row, while the changes for the immediate method are shown on the bottom row. We conducted experiments on datasets of horse to zebra (h→z, z→h), summer to winter (s→w, w→s), apple to orange (a→o, o→a), and tiger to leopard (t→l, l→t). Lower is better. Both Co-evolution

[27] and ThiNet

[27] apply pruning, but Co-evolution is specifically designed for compressing CycleGAN, and ThiNet is a pruning method adapted from a classification task. To make a fair comparison with Co-evolution and ThiNet, the models compared above have similar model sizes and computational requirements (see Table 2).

[0086]

[0087] Table 2

[0088] Table 2 shows the computational and storage results of the main experimental models, where T represents the teacher and S1 and S2 are the student references, respectively. Compared to co-evolution and ThiNet, our model achieves superior performance across all tasks with a smaller / similar model size and computational cost. S1 was selected for the first dataset, and S2 was selected for the remaining datasets. The selection was based on the gap between the baseline performance of teachers and students.

[0089] We explore the variations of our method on CycleGAN through the following experiments: 1) We introduce an intermediate refinement loss on the fake images generated by the first generator in the loop, calculating the L1 norm difference between the teacher-generated and student-generated images. We note that this is an intermediate KD. 2) We conduct experiments with semantic relation preservation loss in both parts of the loop. "Semantic Preservation (SP)" indicates that we apply the semantic refinement loss only on the first generator in the loop (i.e., γ2 = 0 in Equation (9)). "2-Directional SP" indicates that we apply the semantic refinement loss to both generators in the loop. "+" indicates that it is added in addition to the ordinary KD.

[0090] Although all the models compared are in While achieving similar performance on the dataset, our method outperforms other methods on the remaining datasets. Adding our proposed refinement loss to both generators significantly improves the performance of general knowledge refinement, potentially outperforming the original teacher model on certain tasks. We will further demonstrate visual evidence in the later discussion. However, on tasks from summer to winter… In our previous work, we did not observe any improvement in performance, which we suspect is because the baseline student model and teacher model are numerically almost indistinguishable. There is limited room and knowledge for improvement. Furthermore, we ran experiments on an urban landscape dataset and the FCN scores are shown in Table 3. Interestingly, we noted a significant increase in FCN scores when applying the proposed method, but only similar or slightly better image quality was observed compared to the original model (see Supplement). We believe our proposed semantic preservation loss responds strongly to this semantic segmentation dataset by making pixels easier to identify semantically.

[0091] Average pixels (Acc). Average class (Acc). Average IoU teacher 0.592 0.179 0.138 student 0.584 0.182 0.129 ThiNet

[21] 0.218 0.089 0.054 Co-evolution

[27] 0.542 0：212 0.131 Our 0：704 0.205 0：154

[0092] Table 3

[0093] Qualitative results.

[0094] In this section, we will present visual observations of images generated from our model and the reference model. To compare our results with

[27] , we also use our model in... Figure 4An image was generated on the selected input image shown. Therefore, Figure 4 This is an image array 400 used to compare the outputs of a teacher model, a previous model according to

[27] , and a current model according to this disclosure. The top row displays the input horse image and the zebra image generated using the corresponding model. The bottom row displays the input zebra image and the horse image generated using the corresponding model. Obviously, using the current model ( Figure 4 The rightmost column generates images that contain more realistic horses / zebras and minimizes artifacts in the background.

[0095] In the preceding discussion, we mentioned the potential of the student model to outperform the teacher model by incorporating our proposed semantic preservation loss, as evidenced in Table 1. The additional guidance signal from the teacher's pairwise semantic preservation activations not only encourages the student to learn more intra-pixel relationships within a specific image but also to learn semantic understanding across the entire training population. Furthermore, this approach accelerates the discriminator's learning, enabling it to capture more detail at an earlier stage. In some cases, combining these two effects allows the student model to even outperform the teacher model.

[0096] Figure 5 It is an image array 500 with contrast output. Figure 5 In the middle, we showed Two important examples of the task are provided, where our proposed method achieves very good results. Using our method, sharp and realistic textures are generated, even outperforming those of the instructor.

[0097] Figure 6 It is a 600-image array with contrast output. We are... Figure 6 An interesting example shown comes from a common failure case of CycleGAN, where it has multiple objects to transfer. However, our method achieves significant improvement in transforming more objects in the generated image—more zebras are modified into horses. Another interesting finding of our proposed method is that we observe significantly better detail and texture preservation across different tasks.

[0098] Figure 7 It is an array of 700 images, including the magnified portion for contrast output. Figure 7 In this study, the zebra image generated from the input horse image retains more detailed and realistic eyes. We also provide data from ablation studies. Figure 8 Examples from an array of 800 images across different datasets. Examples from multiple datasets compare the results of the baseline model and variations of our approach.

[0099] 4.2 Different Architectures

[0100] We also demonstrate that our method can be extended to other types of network architectures. In the horse-to-zebra task, we replaced the ResNet9 generator with a ResNet6 generator for both the teacher and student models. The FID evaluation is shown in Table 4, which lists... The FID values ​​of the ResNet6 generator on the dataset. Our method still has the largest improvement over other methods, but not as significant as the ResNet9 case. We speculate that removing 3 remaining blocks from the encoder impairs the teacher model's ability to extract more semantic relations, which limits the amount of knowledge to be conveyed during training. We also conducted additional experiments on paired data using UNet

[15] . This was done on an urban landscape dataset mapping from semantic labels to urban landscape photos. The UNet generator produces an encoder-decoder structure that encodes the input through 8 x 4x4 stride-2 convolutional layers and decodes it through 8 x 4x4 stride-2 transposed convolutional layers. Skip connections are used in the regular UNet manner.

[0101] teacher student Mid-KD Regular KD +SP +2 direction SP h→z 88.27 109.93 107.04 105.49 108.71 105.51 z→h 143.08 144.01 142.63 146.26 141.50 141.90

[0102] Table 4

[0103] Average pixels (Acc). Average class (Acc). Average IoU teacher 0.757 0.235 0.191 student 0.710 0.219 0.169 Regular KD 0.742 0.224 0.182 +SP Level 2 0.743 0:230 0.183 +SP Level 3 0:770 0.229 0.183

[0104] Table 5

[0105] For the ResNet generator, the instructor extracts the encoded output from the bottleneck layer with a spatial resolution of 64x64 to guide student training. Since the UNet encoder downsamples the input to a 1x1 spatial resolution at the bottleneck layer, the required spatial semantic information is lacking at this layer. Therefore, we choose to extract semantic relation activation matrices at layers 2 and 3, which have spatial dimensions of 64x64 and 32x32, respectively. Table 5 shows the FCN scoring results, listing the FCN scores of different models trained on the urban landscape dataset using Pix2Pix. The feature encoding extracted from layer 2 of UNet256 has a spatial resolution of 64x64, and the feature encoding extracted from layer 3 has a spatial resolution of 32x32.

[0106] In the implementation, the highest average pixel accuracy was found at layer 3, but the results for average class IoU were similar at both layers. Detailed model compression ratios and visual results can be found in the supplementary notes.

[0107] 5. Supplementary Explanation

[0108] 5.1 Knowledge Extraction from the Pix2Pix Experiment: Objective Function

[0109] The CycleGAN framework, which extracts general knowledge, consists of two generators and a given cycle consistency loss task. The Pix2Pix framework performs transformations only in one direction and trains the transformations in a supervised manner using paired data. Similar to how general knowledge extraction is applied to classification tasks, the objective function in the Pix2Pix framework has the following form:

[0110]

[0111] in The L1 norm loss is the difference between the basic fact labels and the generated image, where λ is... The balance coefficient, and α is a hyperparameter that weights the real labels and teacher labels.

[0112] Semantic knowledge extraction. Based on the general knowledge extraction objective, the semantic knowledge extraction loss is directly added to the above objective function:

[0113]

[0114] 5.2 Model size and calculation results of the Pix2Pix experiment

[0115] The teacher and student models used in Pix2Pix experiments with computational and stored statistics are shown in Table 6. In the implementation, the selection is based on the gap between the baseline performance of teachers and students.

[0116]

[0117] Table 6

[0118] 5.3 Qualitative Results of Urban Landscape

[0119] On the urban landscape dataset, we conducted paired and unpaired image transformation experiments using Pix2Pix and CycleGAN training methods, respectively. Figure 9 and Figure 10 Synthetic cityscape images transformed from their semantic masks are shown, along with instance segmentation masks generated by FCN-8s. Figure 9 This is an image array of 900 from ablation studies, showing urban landscape images with FCN-8s segmentation masks generated by training CycleGAN on an urban landscape dataset. Compared to the student-generated images, the images generated by our method (last column) have significantly fewer artifacts. While the teacher-generated image is more realistic, we observe that our model preserves the semantic class of pixels relative to the input mask. For example, in the top right corner, the teacher-generated image only includes buildings within the specified tree region. Figure 10This is an image array of 1000 from ablation studies, showing urban landscape images with FCN-8s segmented masks generated by training on the Pix2Pix dataset on the urban landscape dataset. Among all generated masks, our model (last column) shows the most distinctive segmentation mask, with clear boundaries for each semantic class. For example, on the right side of the segmented mask, we observe significant improvements in the boundaries of green belts and sidewalks.

[0120] 5.4 Experimental Details

[0121] All models were trained on 256x256 input images with a batch size of 1 and optimized using Adam

[32] (an adaptive learning rate optimization algorithm for training deep neural networks). Other settings for GAN training were the same as for CycleGAN and Pix2Pix.

[0122] Download using the script provided by CycleGAN and Dataset. Sample images with segmentation masks used to plot the semantic similarity matrix. Downloaded from COCO

[18] . Obtained from ImageNet[7] using the keywords tiger and leopard. Dataset. The Cityscapes dataset can be downloaded from the official website (URL:Cityscapes-dataset.com).

[0123] The FID score implementation is adapted from the official PyTorch port version (URL: github.com / mseitzer / pytorch-fid). The FCN score calculation is provided in the official Pix2Pix Torch implementation (URL: github.com / phillipi / Pix2Pix).

[0124] In general knowledge extraction training, we set all experiments with λ = 10 and α = 0.05. γ (γ1 = γ2) in The value is set to 0.9. The value is set to 0.5. The value is set to 0.8. The value is set to 0.2 in the medium and 0.2 in the urban landscape. In the paired transformation experiment, γ is set to 1 and λ is set to 100.

[0125] 6. Application

[0126] In various implementations, the GAN-based model has a generator for image-to-image translation to provide any of the following image processing effects: style transfer, object transformation, seasonal transformation, photo enhancement, or other image processing effects. For example, makeup, hair, nails, or other effects are applied to the source image, transforming it from a first domain space (e.g., in the absence of the desired effect) to a second domain space with the desired effect. For example, through appropriate training, the image-to-image translation can simulate other effects. In some implementations, such effects are applied to (human) skin and / or hair. In one example, an age simulation effect is applied, such as simulating an age effect on an image of a person's face.

[0127] In one implementation, a model having a generator for image-to-image conversion is incorporated into a computer-implemented method (e.g., an application) or computing device or system to provide virtual reality, augmented reality, and / or modified reality experiences. In another implementation, a user can use a smartphone or tablet equipped with a camera to take a selfie (or video), and the generator applies the desired effect in real time, such as playback or other presentation via the smartphone or tablet.

[0128] Models with generators compressed according to the teachings of this paper can be loaded and executed on commonly available consumer smartphones or tablets (e.g., the target device). Experiments were conducted using the device hardware specifications: an Intel processor configured with only 1 core and 1 thread. (Supreme Power) E5-2686 v4 @ 2.30GHz. As mentioned above, the teacher model was defined with ResNet9 and NGF64 as baselines. The new student model was defined with ResNet9 and NGF16. The teacher model size is 44MB, and the student model size is 2.8MB. The inference time for the teacher is 2.69 seconds, and for the student it is 0.43 seconds. More comparison results are shown in Table 2.

[0129] Examples of use of GANS are shown and described in the applicant’s U.S. Patent Application No. 16 / 683,398, filed November 14, 2019, entitled “System and Method for Augmenting Reality by Transforming Images Using Conditional Cyclic Consistent Generative Adversarial Networks (ccGANs),” which is incorporated herein by reference.

[0130] In implementations, the disclosed techniques and methods include developer-related methods and systems for defining (e.g., by tuning) a student model having a generator for image-to-image transformation (maintaining pixel semantic knowledge determined by a teacher model configured for the same task). The student model is a compressed form of the teacher model, reducing the parameters, space footprint, and inference time execution associated with the teacher model. User-related methods and systems are also shown, for example, using the student model (e.g., the generator) at runtime to process images for image-to-image transformation.

[0131] In the implementation, in addition to the aspects of the computing devices of the developer (e.g., used during training) and the target (used during inference), those skilled in the art will understand that aspects of the computer program product are disclosed, wherein instructions are stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, optical disc, etc.) to configure the computing device to perform any of the methods disclosed herein. Aspects related to e-commerce systems are also shown and described. In the implementation, the user's computing device is configured as a client computing device associated with the e-commerce system; for example, the e-commerce system stores a computer program for such a client computing device. Therefore, the e-commerce system has a computer program product as its component, which stores instructions that, when executed by the client computing device (e.g., the processing unit of the client computing device), configure such a client computing device. These and other aspects will be readily apparent.

[0132] Figure 11 This is a block diagram of computer system 1100. In one embodiment, computer system 1100 includes multiple computing devices, including servers, developer computers (PCs, laptops, etc.), and mobile devices such as smartphones and tablets. A network model training environment 1102 is shown, which includes hardware and software to define and configure (e.g., by adjusting) a student generator 1106 (generator G). s A GAN-based student model 1104. Student model 1104 (and student generator 1106) is generated via a teacher generator 1110 (generator G). T The GAN-based teacher model 1108 uses knowledge extraction techniques for conditioning. The teacher model (and student model) are configured for image processing tasks, such as image-to-image transformations.

[0133] In the implementation, the semantic knowledge developed for teacher model 1108 is transferred to student model 1104 by adjusting the semantic knowledge retention loss (a form of refinement loss added to the traditional refinement loss between teacher and student outputs (images)). The semantic knowledge retention loss will be used in teacher model 1108 (e.g., generator G) to transfer the semantic knowledge developed for teacher model 1108 to student model 1104. TThe semantic knowledge developed in the intermediate layer is transferred to the intermediate layer. The teacher model and student model are defined according to the common network architecture. For example, the student model is scaled down relative to the teacher model to reduce size and / or processing resource requirements, thereby enabling the generator to be used on a target device such as system 1100, mobile devices 1110, 1112 (such as 1110).

[0134] In this implementation, the network model training environment 1102 employs a teacher model pre-trained for the image task. For example, the teacher model 1108 is pre-trained using an image dataset 1114 stored at a data server 1116. In this implementation, the teacher model 1108 is a model developed "internally." In this implementation, the teacher model 1108 is publicly available, for example, via an open-source license. The dataset can be similarly developed and available. Depending on the type of image task and the network architecture (e.g., supervised), training is supervised, and the dataset is annotated according to this training. In other cases, training is unsupervised, and the data is also defined accordingly.

[0135] In one implementation, student generator 1106 generates or incorporates augmented reality (AR) application 1120. Although not shown, in this implementation, the application is developed using an application developer's computing device for a specific target device with specific hardware and software, particularly an operating system configuration. In another implementation, AR application 1120 is a native application configured to execute in a specific local environment, such as a local environment defined for a specific operating system (and / or hardware). In yet another implementation, AR application 1120 takes the form of a browser-based application, for example, configured to execute in the browser environment of the target device.

[0136] In this implementation, AR application 1120 is distributed (e.g., downloaded) by user devices such as mobile devices 1110 and 1112. Local applications are typically distributed via application distribution server 1122 (e.g., a "store" operated by a third-party service), although this is not mandatory.

[0137] In this implementation, AR application 1120 is configured to provide an augmented reality experience to a user (e.g., via an interface). For example, effects are applied to images through processing by student generator 1106. The mobile device includes a camera (not shown) to capture images (e.g., still or video images, whether selfies or not). The effect is applied in real-time to the image, such as a video image (and displayed on the mobile device's display device), to simulate the effect on the user when the video is captured. As the camera's position changes, the effect is applied in response to the captured video image to simulate augmented reality.

[0138] In an implementation, the computing device is coupled for communication via one or more networks (e.g., 1122), including wireless networks or other means, public networks or other means, etc.

[0139] As an example, but not limited to, e-commerce system 1124 is web-based and provides a browser-based AR application 1120A as a component of the e-commerce services provided by e-commerce system 1124. E-commerce system 1124 includes configured computing devices and data storage devices 1126 (e.g., databases or other configurations). Data storage device 1126 stores data about products, services, and related information (e.g., technologies used to apply the products). Data storage device 1126 or other data storage devices (not shown) store recommendation rules or other forms of product and / or service recommendations to help users choose from available products and services. Products and services are presented through a user experience interface displayed on the user's (mobile) computing device. It should be understood that e-commerce system 1124 is simplified.

[0140] In this implementation, the browser-based AR application 1120A (or AR application 1120) provides an augmented reality customer experience, such as simulating products, technologies, or services offered or facilitated by the e-commerce system 1124. It should be understood that in this implementation, the AR application 1120 is also configured to provide e-commerce services, for example, via a connection to the e-commerce service 1124.

[0141] As an example, but not limited to, products include cosmetic (e.g., cosmetic products), anti-aging or rejuvenation products, and services include cosmetic, anti-aging or rejuvenation services. Services include treatments or other procedures. Products or services involve a part of the human body, such as the face, hair, or nails. In an embodiment, a computing device thus configured (e.g., mobile device 1112) therefore provides a facial effects unit 1112A, which includes processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the applied effect source image (e.g., displayed) on an e-commerce interface of the computing device facilitated by an e-commerce system. In an embodiment, the facial effects unit 112A utilizes a student generator (G) based on a generative adversarial network (GAN) as described herein. S The computing device provides a user experience unit 112B, including processing circuitry, to determine at least one product or service from a data storage device 1126 and generate one or more virtual instances of recommended products or services for purchase on an e-commerce interface. In one embodiment, at least one product is associated with a corresponding facial effect, and the facial effect unit applies the corresponding facial effect to provide a virtual trial experience.

[0142] In one implementation, the user experience unit 112B is configured to present a graphical user interface (e.g., browser-based or otherwise) to work with the computing device 1112 and the e-commerce system 1124. In another implementation, the e-commerce system 1124 is thus configured to provide AR applications for execution by client computing devices (e.g., 1112), such as mobile devices, and is cooperatively configured to provide e-commerce services to client computing devices to facilitate recommendations of AR-simulated (products / services) and promote purchases via the client computing devices (e.g., 1112).

[0143] Therefore, any computing device, but especially a mobile device, provides a means for transforming images from a first domain space to a second domain space. The computing device includes storage for a student generator (G) based on a generative adversarial network (GAN). S (A storage unit configured to transform an image from a first domain space to a second domain space), generator G S In contrast to a GAN-based teacher generator (G...), which is also configured to transform images from a first domain space to a second domain space... T (Shrinkage). In the implementation, generator G s Includes maintenance for generator G T The configuration of defined semantic relationship knowledge (e.g., a regulated configuration). In an implementation, the computing device includes a processing unit configured (e.g., via AR application 1120) to: receive an image; provide the image to a generator G. s To obtain the converted image; and to provide the converted image for rendering.

[0144] Use with generator G T The loss from extracting relevant semantic relation knowledge is used to adjust the generator G. S Configuration, to be used for generator G T Determined semantic relation knowledge is transferred to the generator G. S The generator G is defined according to the public network architecture. S and generator G T Relative to generator G T Generator G S Scale down in width, depth, or both, where width represents the number of filters in each layer and depth represents the number of remaining blocks in the common network architecture.

[0145] In one implementation, the processing unit is configured to provide a transformed image to the augmented reality interface to simulate effects applied to the image. In another implementation, the effects include any of the following: makeup effects, hair effects, nail effects, and age simulation effects applied to the image. For example, the image may include applicable portions of an object such as a device user (e.g., face, hair, nails, or body parts).

[0146] In the implementation, semantic relation knowledge is derived from the generator G. T The intermediate layer is passed to the generator G S The middle layer.

[0147] In the implementation, generator G S The configuration was adjusted so that:

[0148] a. During adjustment, the generator G at the corresponding intermediate layer S and generator G T In each of the matrices, semantic relations are determined by a similarity matrix (computed by the generator G). S and generator G T The representation is defined by the pairwise activation similarity on the corresponding feature-encoded pixels; and

[0149] b. Knowledge is transferred through semantic relation knowledge extraction loss on the similarity matrix.

[0150] In the implementation, through generator G S and generator G T The generator G is adjusted by adding a semantic relation knowledge extraction loss, in addition to the traditional extraction loss, to the final generated image. S Configuration. The corresponding semantic relation activation matrix is ​​calculated as the outer product of the corresponding feature encodings, and the semantic relation knowledge extraction loss is compared with the generator G. S and generator G T The corresponding semantic relation activation matrix. In the implementation, a computing device such as mobile device 1110 is configured to execute the computing device aspect of the method described herein. Other aspects will be apparent, such as computer program product aspects.

[0151] The network model training environment 1102 provides computing devices configured to execute methods (e.g., methods configured by adjusting a GAN-based student generator). It should be understood that the implementation of the computing devices in the network model training environment 1102, as well as any related implementations such as the student generator or model, are appropriately adapted to the training method. Figure 12 A flowchart of operation 1200 according to the embeddings herein is shown. In an implementation, the operation provides methods such as those for training (e.g., by tuning). In step 1202, the student generator (G) based on a generative adversarial network (GAN) is tuned. S The generator G operates on the configuration to transform the image from the first domain space to the second domain space. S Compared to a GAN-based teacher generator (G...) configured to transform images from a first domain space to a second domain space... T) Scale down, where knowledge distillation is used to adjust the generator G s To maintain as generator G T Determined semantic relation knowledge. In step 1204, the operation provides the generator G. S This is for use by computing devices to convert images. The relevant computing devices and computer program products will be obvious, as will other aspects.

[0152] 7. Conclusion

[0153] In various implementations, model compression of GANs via methods, systems, and techniques extended from traditional knowledge extraction is described. Semantic relation knowledge is transferred from the teacher model to a selected student model, which generates images with better detail and texture after explicitly learning the relationships between pixels, while knowledge extraction is used to significantly reduce model size and computational requirements. Experiments conducted on five different datasets and three different architectures (e.g., implementations) quantitatively and qualitatively demonstrate that the proposed methods, systems, and techniques help bring previously inadequate student networks to the level of their teacher, with the ability to generate images at a significantly higher quality level. Furthermore, applying knowledge extraction-based methods, systems, and techniques alleviates the burden of controlling pruning rates and carefully constructing pruned models, and also provides greater freedom in model selection.

[0154] A practical implementation may include any or all of the features described herein. These and other aspects, features, and various combinations may be represented as methods, apparatus, systems, devices, programming products, and other combinations of the features described herein for performing functions. Many embodiments have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the methods and techniques described herein. Furthermore, additional steps may be provided from the described process, or other steps may be eliminated from the described process, and other components may be added to or removed from the described system. Therefore, other embodiments are within the scope of the following claims.

[0155] Throughout the description and claims of this specification, the words “comprising” and “including” and variations thereof mean “including, but not limited to”, and they are not intended to exclude other components, integers, or steps. In this specification, the singular includes the plural unless the context requires otherwise. In particular, where the indefinite article is used, the specification should be understood to consider both the plural and singular unless the context requires otherwise.

[0156] Features, integers, characteristics, or groups described in connection with a particular aspect, embodiment, or example of the invention should be understood to be applicable to any other aspect, embodiment, or example, unless incompatible therewith. All features disclosed herein (including any appended claims, abstract, and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except that at least some of these features and / or steps are mutually exclusive combinations. The invention is not limited to the details of any of the foregoing examples or embodiments. The invention extends to any novel one or any novel combination of features disclosed in this specification (including any appended claims, abstract, and drawings), or to any novel one or any novel combination of steps of any disclosed method or process.

[0157] References

[0158] 1.Brock,A.,Donahue,J.,Simonyan,K.:Large scale GAN training for high fidelity natural image synthesis.In:International Conference on LearningRepresentations(2019)

[0159] 2.Chen, G., Choi, W., Yu, 30, pp.742-751. Curran Associates, Inc. (2017), URL: papers.nips.cc / paper / 6676-learning-efficient-object-detection-models-with-knowledge-dist illation.pdf

[0160] 3.Chen,Y.,Yang,T.,Zhang,X.,Meng,G.,Pan,C.,Sun,J.:Detnas:Backbonesearch for object detection(2019)

[0161] 4.Chen,Y.,Wang,N.,Zhang,Z.:Darkrank:Accelerating deep metric learningvia cross sample similarities transfer.In:Thirty-Second AAAI Conference onArtificial Intelligence(2018)

[0162] 5.Choi,Y.,Choi,M.,Kim,M.,Ha,J.W.,Kim,S.,Choo,J.:Stargan:Unifiedgenerative adversarial networks for multi-domain image-to-imagetranslation.In:Proceedings of the IEEE conference on computer vision andpattern recognition.pp.8789-8797(2018)

[0163] 6.Cordts,M.,Omran,M.,Ramos,S.,Rehfeld,T.,Enzweiler,M.,Benenson,R.,Franke,U.,Roth,S.,Schiele,B.:The cityscapes dataset for semantic urban sceneunderstanding.In:Proceedings of the IEEE conference on computer vision andpattern recognition.pp.3213-3223(2016)

[0164] 7.Deng,J.,Dong,W.,Socher,R.,Li,L.J.,Li,K.,Fei-Fei,L.:Imagenet:Alarge-scale hierarchical image database.In:2009IEEE conference on computervision and pattern recognition.pp.248-255.Ieee(2009)

[0165] 8.Goodfellow,I.,Pouget-Abadie,J.,Mirza,M.,Xu,B.,Warde-Farley,D.,Ozair,S.,Courville,A.,Bengio,Y.:Generative adversarial nets.In:Advances inneural information processing systems.pp.2672-2680(2014)

[0166] 9.Han,S.,Mao,H.,Dally,W.J.:Deep compression:Compressing deep neuralnetwork with pruning,trained quantization and human coding.In:Bengio,Y.,Le-Cun,Y.(eds.)4th International Conference on Learning Representations,ICLR2016,San Juan,Puerto Rico,May 2-4,2016,Conference Track Proceedings(2016),URL:arxiv.org / abs / 1510.00149

[0167] 10.He,Y.,Lin,J.,Liu,Z.,Wang,H.,Li,L.J.,Han,S.:Amc:Automl for modelcompression and acceleration on mobile devices.In:Proceedings of the EuropeanConference on Computer Vision(ECCV).pp.784-800(2018)

[0168] 11.Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nashequilibrium.In:Advances in neural information processing systems.pp.6626-6637(2017)

[0169] 12.Hinton,G.,Vinyals,O.,Dean,J.:Distilling the knowledge in a neuralnetwork.arXiv preprint arXiv:1503.02531(2015)

[0170] 13.Howard,A.,Sandler,M.,Chu,G.,Chen,L.C.,Chen,B.,Tan,M.,Wang,W.,Zhu,Y.,Pang,R.,Vasudevan,V.,et al.:Searching for mobilenetv3.In:Proceedings ofthe IEEE International Conference on Computer Vision.pp.1314-1324(2019)

[0171] 14.Howard,A.G.,Zhu,M.,Chen,B.,Kalenichenko,D.,Wang,W.,Weyand,T.,Andreetto,M.,Adam,H.:Mobilenets:Efficient convolutional neural networks formobile vision applications.arXiv preprint arXiv:1704.04861(2017)

[0172] 15.Isola,P.,Zhu,J.Y.,Zhou,T.,Efros,A.A.:Image-to-image translationwith conditional adversarial networks.In:Proceedings of the IEEE conferenceon computer vision and pattern recognition.pp.1125-1134(2017)

[0173] 16.Krizhevsky,A.,Sutskever,I.,Hinton,G.E.:Imagenet classificationwith deep convolutional neural networks.In:Advances in neural informationprocessing systems.pp.1097-1105(2012)

[0174] 17.Ledig,C.,Theis,L.,Huszar,F.,Caballero,J.,Cunningham,A.,Acosta,A.,Aitken,A.,Tejani,A.,Totz,J.,Wang,Z.,et al.:Photo-realistic single imagesuper-resolution using a generative adversarial network.In:Proceedings of theIEEE conference on computer vision and pattern recognition.pp.4681-4690(2017)

[0175] 18.Lin,T.,Maire,M.,Belongie,S.J.,Hays,J.,Perona,P.,Ramanan,D.,Dollar,P.,Zitnick,C.L.:Microsoft COCO:common objects in context.In:Fleet,D.J.,Pajdla,T.,Schiele,B.,Tuytelaars,T.(eds.)Computer Vision-ECCV 2014-13thEuropean Conference,Zurich,Switzerland,September 6-12,2014,Proceedings,PartV.Lecture Notes in Computer Science,vol.8693,pp.740-755.Springer(2014).URL:doi.org / 10.1007 / 978-3-319-10602-1_48

[0176] 19.Liu,C.,Chen,L.C.,Schroff,F.,Adam,H.,Hua,W.,Yuille,A.L.,Fei-Fei,L.:Auto-deeplab:Hierarchical neural architecture search for semantic imagesegmentation.In:The IEEE Conference on Computer Vision and PatternRecognition(CVPR)(June 2019)

[0177] 20.Liu,Y.,Chen,K.,Liu,C.,Qin,Z.,Luo,Z.,Wang,J.:Structured knowledgedistillation for semantic segmentation.In:Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition.pp.2604-2613(2019)

[0178] 21.Luo,J.H.,Wu,J.,Lin,W.:Thinet:A filter level pruning method fordeep neural network compression.In:Proceedings of the IEEE internationalconference on computer vision.pp.5058-5066(2017)

[0179] 22.Park,W.,Kim,D.,Lu,Y.,Cho,M.:Relational knowledge distillation.In:Proceedings of the IEEE Conference on Computer Vision and PatternRecognition.pp.3967-3976(2019)

[0180] 23.Peng,B.,Jin,X.,Liu,J.,Li,D.,Wu,Y.,Liu,Y.,Zhou,S.,Zhang,Z.:Correlation congruence for knowledge distillation.In:Proceedings of the IEEEInternational Conference on Computer Vision.pp.5007-5016(2019)

[0181] 24.Redmon,J.,Farhadi,A.:Yolov3:An incremental improvement.arXivpreprint arXiv:1804.02767(2018)

[0182] 25.Sandler,M.,Howard,A.,Zhu,M.,Zhmoginov,A.,Chen,L.C.:Mobilenetv2:Inverted residuals and linear bottlenecks.In:Proceedings of the IEEEconference on computer vision and pattern recognition.pp.4510-4520(2018)

[0183] 26.Shaham,T.R.,Dekel,T.,Michaeli,T.:Singan:Learning a generativemodel from a single natural image.In:Proceedings of the IEEE InternationalConference on Computer Vision.pp.4570-4580(2019)

[0184] 27.Shu,H.,Wang,Y.,Jia,X.,Han,K.,Chen,H.,Xu,C.,Tian,Q.,Xu,C.:Co-evolutionary compression for unpaired image translation.In:Proceedings of theIEEE International Conference on Computer Vision.pp.3235-3244(2019)

[0185] 28.Simonyan,K.,Zisserman,A.:Very deep convolutional networks forlarge-scale image recognition.In:International Conference on LearningRepresentations(2015)

[0186] 29.Tung,F.,Mori,G.:Similarity-preserving knowledge distillation.In:Proceedings of the IEEE International Conference on Computer Vision.pp.1365-1374(2019)

[0187] 30.Zagoruyko,S.,Komodakis,N.:Paying more attention to attention:Improving the performance of convolutional neural networks via attentiontransfer.arXiv preprint arXiv:1612.03928(2016)

[0188] 31.Zhu,J.Y.,Park,T.,Isola,P.,Efros,A.A.:Unpaired image-to-imagetranslation using cycle-consistent adversarial networks.In:Proceedings of theIEEE international conference on computer vision.pp.2223-2232(2017)

[0189] 32.Diederik P.Kingma and Jimmy Lei Ba.Adam:A method for stochasticoptimization.arXiv:1412.6980v9(2014).

Claims

1. A computing device for converting an image from a first domain space to a second domain space, the computing device comprising: Storage unit, storing the student generator G based on Generative Adversarial Network (GAN) S It is configured to transform an image from the first domain space to the second domain space, the student generator G S In contrast to a GAN-based teacher generator G, which is also configured to transform images from the first domain space to the second domain space. T Scale down, where the student generator G s Including maintenance for the teacher generator G T The configuration of defined semantic relation knowledge, wherein the student generator G S The configuration was adjusted so that: a. During the adjustment period, the student generator G at the corresponding intermediate layer S and the teacher generator G T In each of the sequences, semantic relations are represented by a similarity matrix, which is calculated by the student generator G. S and the teacher generator G T Defined by the pairwise activation similarity on the corresponding feature-encoded pixels; and b. The knowledge is transferred via a semantic relation knowledge extraction loss on the similarity matrix; and The processing unit is configured to: receive the image; and provide the image to the student generator G. s To obtain a converted image; and to provide the converted image for presentation.

2. The computing device according to claim 1, wherein, Using the teacher generator G T The relevant semantic relation knowledge extraction loss is used to adjust the student generator G. S Configured to provide the teacher generator G T The determined semantic relation knowledge is passed to the student generator G. S .

3. The computing device according to claim 1 or 2, wherein, The student generator G S and the teacher generator G T Defined according to the public network architecture.

4. The computing device according to claim 3, wherein, The student generator G S Relative to the teacher generator G T It is scaled down in terms of width, depth, or both, where width represents the number of filters in each layer and depth represents the number of remaining blocks in the common network architecture.

5. The computing device according to claim 1 or 2, wherein, The processing unit is configured to provide the transformed image in an augmented reality interface to simulate the effect applied to the image.

6. The computing device according to claim 5, wherein, The effect is any one of the makeup effect, hair effect, nail effect, and age simulation effect applied to the image, including the applicable portion of the object.

7. The computing device according to claim 1 or 2, including a mobile device.

8. The computing device according to claim 1 or 2, wherein, The semantic relation knowledge comes from the teacher generator G. T The intermediate layer passes the data to the student generator G. S The middle layer.

9. The computing device according to claim 1, wherein, Through the student generator G S and the teacher generator G T In addition to the traditional extraction loss, the semantic relation knowledge extraction loss is added to the final generated image to adjust the student generator G. S Configuration.

10. The computing device according to claim 1, wherein, The corresponding semantic relation activation matrix is ​​calculated as the outer product of the corresponding feature encoding, and the semantic relation knowledge extraction loss is compared with the student generator G. S and the teacher generator G T The corresponding semantic relation activation matrix.

11. The computing device according to claim 10, wherein, Through the feature encoding The activation matrix is ​​represented by the outer product of the two methods, which encodes the features. Defined as encoder The output matrix of the i-th image example in the last layer: Among them, H ′ and W ′ These represent the feature encoding height and width, respectively, while C t / C s This is the number of channels, and it uses a batch size of 1 to calculate the semantic relation activation matrix. As The outer product is then performed row-by-row L2 normalization:

12. A method for transforming an image from a first domain space to a second domain space, comprising: By adjusting the student generator G based on Generative Adversarial Network (GAN) S The student generator G is configured to transform the image from a first domain space to a second domain space. S Compared to a GAN-based teacher generator G configured to transform images from the first domain space to the second domain space T Scale down, where knowledge distillation is used to adjust the student generator G. s To maintain the teacher generator G T Definite semantic relation knowledge; and Provide the student generator G S For use by computing devices to convert images, The method adjusts the student generator G. S Make: a. During the adjustment period, the student generator G at the corresponding intermediate layer S and the teacher generator G T In each of the sequences, semantic relations are represented by a similarity matrix, which is calculated by the student generator G. S and the teacher generator G T Defined by the pairwise activation similarity on the corresponding feature-encoded pixels; and b. The knowledge is transferred via the semantic relation knowledge extraction loss on the similarity matrix.

13. The method according to claim 12, wherein, Using the teacher generator G T The relevant semantic relation knowledge extraction loss is used to adjust the student generator G. S To provide the teacher generator G T The determined semantic relation knowledge is transferred to the student generator G. S .

14. The method of claim 12 or 13, comprising defining the student generator G according to a public network architecture. S and the teacher generator G T .

15. The method according to claim 14, wherein, The student generator G S Relative to the teacher generator G T It is scaled down in terms of width, depth, or both, where width represents the number of filters in each layer and depth represents the number of remaining blocks in the common network architecture.

16. The method according to claim 12 or 13, wherein, The computing device is configured to provide the transformed image in an augmented reality interface to simulate the effect applied to the image.

17. The method according to claim 16, wherein, The effect is any one of the makeup effect, hair effect, nail effect, and age simulation effect applied to the image, including the applicable portion of the object.

18. The method according to claim 12 or 13, wherein, The computing device includes a mobile device.

19. The method according to claim 12 or 13, wherein, The method extracts the semantic relation knowledge from the teacher generator G. T The intermediate layer passes the data to the student generator G. S The middle layer.

20. The method according to claim 12, wherein, The adjustment is in the student generator G S and the teacher generator G T In addition to the traditional extraction loss, the semantic relationship knowledge extraction loss is also added to the final generated image.

21. The method according to claim 12, wherein, The corresponding semantic relation activation matrix is ​​calculated as the outer product of the corresponding feature encoding, and the semantic relation knowledge extraction loss is compared with the student generator G. S and the teacher generator G T The corresponding semantic relation activation matrix.

22. The method according to claim 21, wherein, Through the feature encoding The activation matrix is ​​represented by the outer product of the two methods, which encodes the features. Defined as encoder The output matrix of the i-th image example in the last layer: Among them, H ′ and W ′ These represent the feature encoding height and width, respectively, while C t / C s This is the number of channels, and it uses a batch size of 1 to calculate the semantic relation activation matrix. As The outer product is then performed row-by-row L2 normalization:

23. The method according to claim 12 or 13, wherein, The teacher generator G is pre-trained. T The image is converted from the first domain space to the second domain space before the adjustment.

24. The method according to claim 12 or 13, wherein, The student generator G S Together with a second student generator, the second student generator is adjusted using a second teacher generator according to the method to transform the image from the second domain space to the first domain space.

25. A computing device for converting an image from a first domain space to a second domain space, comprising: A facial effects unit includes processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the source image with the applied effect on an e-commerce interface. The facial effects unit utilizes a student generator G based on a Generative Adversarial Network (GAN). S The student generator G transforms the image from a first domain space to a second domain space in which at least one facial effect is applied. S In contrast to a GAN-based teacher generator G, which is also configured to transform images from the first domain space to the second domain space. T Scale down, and Among them, the student generator G s Including maintenance for the teacher generator G T The configuration of defined semantic relation knowledge, wherein the student generator G S The configuration was adjusted so that: a. During the adjustment period, the student generator G at the corresponding intermediate layer S and the teacher generator G T In each of the sequences, semantic relations are represented by a similarity matrix, which is calculated by the student generator G. S and the teacher generator G T Defined by the pairwise activation similarity on the corresponding feature-encoded pixels; and b. The knowledge is transferred via the semantic relation knowledge extraction loss on the similarity matrix.

26. The computing device of claim 25, further comprising: The user experience unit includes processing circuitry configured to identify at least one product from a data storage device storing the product and generate product recommendations on an e-commerce interface to purchase the product; In this embodiment, at least one product is associated with a corresponding facial effect, and the facial effect unit applies the corresponding facial effect to provide a virtual trial experience.

27. The computing device according to claim 25 or 26, wherein, The computing device includes the computing device according to any one of claims 1 to 11.

28. The computing device according to claim 25, wherein, The corresponding semantic relation activation matrix is ​​calculated as the outer product of the corresponding feature encoding, and the semantic relation knowledge extraction loss is compared with the student generator G. S and the teacher generator G T The corresponding semantic relation activation matrix.

29. The computing device according to claim 28, wherein, Through the feature encoding The activation matrix is ​​represented by the outer product of the two methods, which encodes the features. Defined as encoder The output matrix of the i-th image example in the last layer: Among them, H ′ and W ′ These represent the feature encoding height and width, respectively, while C t / C s This is the number of channels, and it uses a batch size of 1 to calculate the semantic relation activation matrix. As The outer product is then performed row-by-row L2 normalization:

30. A computer program product for converting an image from a first domain space to a second domain space, comprising a non-transitory storage device for storing instructions, wherein, when executed by a computing device, the computing device is configured to include: A facial effects unit includes processing circuitry configured to apply at least one facial effect to a source image and generate one or more virtual instances of the source image with the applied effect on an e-commerce interface. The facial effects unit utilizes a student generator G based on a Generative Adversarial Network (GAN). S The student generator G transforms the image from a first domain space to a second domain space in which at least one facial effect is applied. S In contrast to a GAN-based teacher generator G, which is also configured to transform images from the first domain space to the second domain space. T Scale down, and Among them, the student generator G s Including maintenance for the teacher generator G T The configuration of defined semantic relation knowledge, wherein the student generator G S The configuration was adjusted so that: a. During the adjustment period, the student generator G at the corresponding intermediate layer S and the teacher generator G T In each of the sequences, semantic relations are represented by a similarity matrix, which is calculated by the student generator G. S and the teacher generator G T Defined by the pairwise activation similarity on the corresponding feature-encoded pixels; and b. The knowledge is transferred via the semantic relation knowledge extraction loss on the similarity matrix.

31. The computer program product according to claim 30, wherein, When executed, the instruction further configures the client computing device to include: The user experience unit includes processing circuitry configured to identify at least one product from a data storage device storing the product and generate product recommendations on an e-commerce interface to purchase the product; In this embodiment, at least one product is associated with a corresponding facial effect, and the facial effect unit applies the corresponding facial effect to provide a virtual trial experience.

32. The computer program product according to claim 30 or 31, wherein, The instructions further configure the client computing device as a computing device including any one of claims 1 to 11.

33. The computer program product according to claim 30 or 31, wherein, The computer program product is a component of an e-commerce system used to communicate with user equipment.

34. The computer program product according to claim 30, wherein, Using the teacher generator G T The relevant semantic relation knowledge extraction loss is used to adjust the student generator G. S Configured to provide the teacher generator G T The determined semantic relation knowledge is passed to the student generator G. S .

35. The computer program product according to claim 30, wherein, The at least one facial effect is a makeup effect or age simulation effect applied to an object in the source image.

36. The computer program product according to claim 30, wherein, The corresponding semantic relation activation matrix is ​​calculated as the outer product of the corresponding feature encoding, and the semantic relation knowledge extraction loss is compared with the student generator G. S and the teacher generator G T The corresponding semantic relation activation matrix.

37. The computer program product according to claim 36, wherein, Through the feature encoding The activation matrix is ​​represented by the outer product of the two methods, which encodes the features. Defined as encoder The output matrix of the i-th image example in the last layer: Among them, H ′ and W ′ These represent the feature encoding height and width, respectively, while C t / C s This is the number of channels, and it uses a batch size of 1 to calculate the semantic relation activation matrix. As The outer product is then performed row-by-row L2 normalization:

Images