Method and system of data-free adversarial knowledge distillation

Abstract

A method and system for dataless adversarial knowledge distillation. The method includes: receiving random input at a generator; generating a synthesized output of the received random input at the generator; receiving the synthesized output at a teacher network; receiving the synthesized output at a student network; minimizing the maximum distance between the output of the teacher network and the output of the student network; and a constraint generator.

Description

Technical Field

[0001] This disclosure is generally related to neural networks using knowledge distillation. [Priority]

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 62 / 993,258, filed on Mar. 23, 2020, the entire content of which is incorporated herein by reference.

Background Art

[0003] Deep learning currently leads to many performance breakthroughs in various computer vision tasks. The modern performance of deep learning is accompanied by over-parameterized deep neural networks, which enables the automatic extraction of useful representations (features) of data for a target task when training on extremely large datasets. The optimization framework of deep neural networks using stochastic gradient descent has recently become extremely fast and efficient by using backpropagation techniques with hardware units dedicated to matrix / tensor operations, such as graphical processing units (GPUs). Experience shows that the benefits of over-parameterization are key factors in the great success of deep learning, but once a well-trained high-accuracy model is discovered, its deployment on various inference platforms faces different requirements and challenges. Specifically, for deploying a pre-trained model on resource-constrained platforms such as mobile devices or edge devices, computational cost and memory requirements are key factors to carefully consider for efficient inference. Therefore, model compression (also known as network compression) is an important procedure for developing efficient inference models.

Summary of the Invention

[0004] According to one embodiment, a method for data-free adversarial knowledge distillation includes: receiving a random input at a generator; generating a synthetic output of the received random input at the generator; receiving the synthetic output at a teacher network; receiving the synthetic output at a student network; minimizing a maximum value of a distance between an output of the teacher network and an output of the student network; and constraining the generator.

[0005] According to one embodiment, a system for data-free adversarial knowledge distillation includes: a memory; and a processor configured to receive a random input at a generator; generate a synthetic output of the received random input at the generator; receive the synthetic output at a teacher network; receive the synthetic output at a student network; minimize a maximum value of a distance between an output of the teacher network and an output of the student network; and constrain the generator.

Embodiments

[0006] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that although the same elements are illustrated in different drawings, the same elements will be denoted by the same reference numerals. In the following description, only specific details such as detailed configurations and components are provided to assist in the overall understanding of the embodiments of the present disclosure. Therefore, it will be apparent to those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. In addition, descriptions of well-known functions and configurations are omitted for clarity and conciseness. The terms described below are defined in consideration of the functions in the present disclosure and may vary according to the user, the user's intention, or habit. Therefore, the definitions of the terms should be determined based on the content throughout this specification.

[0007] The present disclosure can have various modifications and various embodiments, and embodiments among the various embodiments will be described in detail below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the embodiments, but the present disclosure includes all modifications, equivalents, and alternatives within the scope of the present disclosure.

[0008] Although terms including ordinal numbers such as first and second may be used to describe various elements, the structural elements are not limited by such terms. The terms are only used to distinguish one element from another. For example, without departing from the scope of the present disclosure, the first structural element may be referred to as the second structural element. Similarly, the second structural element may also be referred to as the first structural element. As used herein, the term "and / or" includes any and all combinations of any one of the one or more related items.

[0009] The terms used herein are only used to describe the various embodiments of the present disclosure and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In the present disclosure, it should be understood that the terms "comprising" or "having" indicate the presence of features, numbers, steps, operations, structural elements, parts, or combinations thereof, and do not exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, structural elements, parts, or combinations thereof.

[0010] Unless otherwise defined differently, all terms used herein have the same meaning as understood by those of ordinary skill in the art to which the present disclosure pertains. Terms such as those defined in common dictionaries should be interpreted as having the same meaning as the context in the relevant technical field, and should not be interpreted as having an ideal or overly formalized meaning unless clearly defined in the present disclosure.

[0011] An electronic device according to an embodiment may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to the electronic devices described above.

[0012] The terms used in the present disclosure are not intended to limit the present disclosure, but are intended to include various changes, equivalents, or alternatives of the corresponding embodiments. Regarding the description of the accompanying drawings, like reference numerals may be used to refer to like elements or related elements. Unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to an item may include one or more of the things. As used herein, each of such phrases as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include all possible combinations of the items enumerated together in the corresponding phrase. As used herein, terms such as "first", "second", "primary", and "secondary" may be used to distinguish a corresponding component from another component, but are not intended to limit the component in other aspects (e.g., importance or order). It is intended that, with or without the terms "operatively" or "communicatively", if one element (e.g., a first element) is referred to as "coupled with", "coupled to", "connected with", or "connected to" another element (e.g., a second element), it indicates that the element may be directly (e.g., wired), wirelessly, or via a third element coupled with the other element.

[0013] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic", "logic block", "portion", and "circuitry". A module may be a single integrated component or its smallest unit or portion for performing one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0014] Model compression includes various methods such as weight pruning, network quantization, and distillation to a network with a more efficient architecture. Weight pruning and network quantization reduce the computational cost and storage / memory size without changing the network architecture. Weight pruning compresses the model by completely removing the redundant weights from the weight pruning by setting the redundant weights to zero, so that the operations and memory for the pruned weights can be skipped. Network quantization reduces the memory footprint for weights and activations by quantization and is usually followed by lossless source coding for compression. In addition, convolutional and fully connected layers can be implemented using low-precision fixed-point operations (such as 8-bit fixed-point operations) to reduce latency and improve power efficiency. On the other hand, the network architecture can be modified to be simpler and more easily implemented on the target platform. For example, the number of layers and / or channels in each layer can be reduced. Conventional spatial domain convolutions can be replaced with more efficient depthwise separable convolutions as in MobileNet.

[0015] Knowledge distillation (KD) is a knowledge transfer framework for training a smaller "student" network under the guidance of a larger pre-trained "teacher" model. One problem with existing model compression methods (including KD) is that existing model compression methods are developed under the strong assumption that the original training data is accessible during the compression process. As datasets become larger, the distribution of the datasets also becomes more expensive and difficult. In addition, data privacy and security have become one of the major issues in deep learning, and the resulting regulations and compliance requirements regarding security and privacy complicate both the data sharing by the original model trainer and the data collection by the model compressor, such as in the case of medical and biometric data. Therefore, there is a strong need to compress pre-trained models without accessing the original or even alternative datasets.

[0016] The system and method of the present invention include an adversarial knowledge distillation framework that minimizes the worst-case possible loss (maximum loss) via adversarial learning when the loss of the original training data is not accessible. Taking into account any metadata, the system and method utilize the metadata to provide an auxiliary loss for the generator to minimize the adversarial learning framework. To avoid the additional effort of creating new metadata for sharing, the statistics stored in the batch normalization layer are used to constrain the generator to generate synthetic samples that mimic the original training data. In addition, the system and method use various synthetic samples by using multiple generators. The system and method can also perform adversarial KD on multiple students simultaneously.

[0017] The system and method of adversarial knowledge distillation without data can minimize the maximum value of the Kullback-Leibler (KL) divergence between the teacher output and the student output. In the maximization step for training a generator to generate adversarial images, the system and method constrain the generator to match the statistics from the batch normalization layer of the teacher; promote a smaller entropy of the output class distribution (softmax output) for each sample and a larger entropy of the output class distribution (softmax output) averaged over each batch to generate synthetic images similar to the original data. The system and method match the mean and variance of the batch normalization layer inputs by using the KL divergence of two Gaussian distributions. In the minimization step for KD, in addition to minimizing the KL divergence between the teacher softmax output and the student softmax output, the intermediate layer outputs can also be matched as appropriate.

[0018] The system and method of the present invention use multiple generators and multiple students to generate various synthetic images in adversarial knowledge distillation, perform data-free network quantization using adversarial knowledge distillation by making the student a quantized teacher, and perform data-free network compression using adversarial knowledge distillation by making the student a smaller-sized network.

[0019] In the case of considering any metadata, the system and method utilize the metadata to provide additional constraints for the generator to conform to the adversarial learning framework. To avoid the additional effort of creating new metadata for sharing, the system and method use the statistics stored in the batch normalization layer to train the generator to generate synthetic samples that mimic the original training data. By adding an auxiliary loss for the generator, the system and method constrain the generator such that the generator generates synthetic images that generate teacher-side statistics similar to the original data, which helps to minimize the minimax optimization from avoiding any adversarial samples that are very different from the original data and generate better distillation performance (reduce the loss caused by fitting the model to "bad" examples that are not close to the original dataset).

[0020] The system and method utilize batch normalization statistics to constrain the generator. In addition, to match the mean and variance, the system and method use the KL divergence of two Gaussian distributions, and the KL divergence is a scaled and normalized distance measure (i.e., standard deviation).

[0021] Robust optimization is an optimization subfield that addresses data uncertainty in optimization problems. In this framework, it is assumed that the objective and constraint functions belong to certain sets called "uncertainty sets". The goal is to make decisions that remain feasible regardless of how the constraints change and are optimal for the worst-case objective function. In the absence of data, the no-data KD problem is formulated as a robust optimization problem, and the uncertainty set can be constrained based on the statistics at the batch normalization layer of a pre-trained teacher.

[0022] Generating synthetic data to deceive a pre-trained model is closely related to the adversarial attack problem. In adversarial attacks, there are also two methods. First, directly generate adversarial images in the image domain; and second, use a generator to generate adversarial images.

[0023] A generator network containing a series of convolutional layers can be used as a good regularization term that can be applied to image generation as before. Thus, instead of adding any prior regularization, the generator is used, and this regularization is used to obtain synthetic images in the absence of a generator.

[0024] Adversarial learning is used in Generative Adversarial Networks (GANs). Mode collapse is one of the well-known problems in GANs. A simple but effective way to overcome mode collapse is to introduce multiple generators and / or multiple discriminators. Using multiple generators and / or multiple students (students can act as discriminators) helps to generate various samples and avoid overfitting in our no-data KD network.

[0025] FIG. 1 shows a schematic diagram of a no-data adversarial KD network according to an embodiment. The network 100 includes a generator 102, a student 104, and a teacher 106. The teacher 106 includes a convolutional layer 110, a batch normalization layer 112, a rectified linear unit (ReLU) layer 114, and a softmax layer 116. The network minimizes the maximum value of the KL divergence between the output of the teacher 106 and the output of the student 104. In the maximization step for training the generator 102 to generate adversarial images, the generator 102 is constrained to generate synthetic images similar to the original data by matching the statistics from the batch normalization layer 112 of the teacher 106.

[0026] FIG. 2 shows a flowchart 200 of a method for data-free knowledge distillation according to an embodiment. At step 202, the system receives a random input at a generator. At step 204, the system uses the generator to generate a synthetic output of the received random input. The synthetic output may be similar to the original data used to train the teacher. At step 206, the system receives the synthetic output at a teacher network and a student network. At step 208, the system minimizes the maximum value of the distance between the output of the teacher network and the output of the student network. The distance may be the KL divergence. At step 210, the system constrains the generator. The generator may be constrained by matching the mean and variance of at least one batch normalization layer.

[0027] KD data-free model compression is described below. For a general non-linear neural network for classification, the general non-linear neural network is designed to generate a categorical probability distribution of the labels of the input over a label set (i.e., ). is a one-hot encoded ground truth label over a set of inputs. The network is pre-trained using a labeled dataset of probability distributions (referred to as the training dataset), as in Equation (1): (1)

[0028] Where is actually the empirical expectation value with respect to the training dataset, and represents the Kullback-Leibler (KL) divergence. Minimization of the KL divergence is equivalent to minimization of the cross-entropy, given the distribution.

[0029] Another neural network, called the "student", may be smaller and less complex compared to the pre-trained network called the "teacher". The student also generates its estimate of the categorical probability distribution of the input, such that. KD proposes to optimize the student as in Equation (2): (2)

[0030] Where. For simplicity, the temperature parameter is omitted, and the temperature parameter may be applied before the softmax of and in the second KL divergence term of Equation (2).

[0031] As shown in Equation (2), under the assumption of a training dataset given the expectation value with respect to, the original KD is formed. However, sharing a large dataset is expensive and sometimes even impossible due to privacy and security issues. Therefore, there is interest in designing a KD method in the case where the training dataset is not accessible, but only the pre-trained teacher is given.

[0032] Robust optimization recommends minimizing the possible loss in the worst-case scenario (maximum loss) using adversarial learning under data uncertainty, where the worst-case scenario is similar to the situation encountered when no training dataset for optimization is provided. To adopt robust minimax optimization (known as adversarial learning) in KD, a generator network is introduced, which is used to generate synthetic adversarial data for the input to KD. Then, using the minimax method, the data-free adversarial KD is given by Equation (3): (3)

[0033] For, where is the additional or auxiliary loss that can be provided by the pre-trained teacher for the generator based on the generator output. In Equation (3), corresponds to Steps 202 and 204, corresponds to Step 206, the minimax operation corresponds to Step 208, and corresponds to Step 210. Comparing Equation (2) with Equation (1), the first KL divergence term related to the ground truth label is omitted, as in Equation (4). (4)

[0034] If there exists a generator that is optimized to accurately mimic the training data such that, then Equation (4) simplifies to Equation (5). (5)

[0035] However, there is no access to the original training data, and the optimal generator cannot be found. Instead, the upper bound of is minimized by solving the minimax problem in Equation (3), thereby using the auxiliary loss of the generator to provide some constraints to the generator to generate data similar to the original training data.

[0036] In the maximization step of Equation (3), the following three auxiliary loss terms are considered for the generator to make the generator generate as many "good" adversarial samples similar to the original data as possible based on the teacher. The first term contains batch normalization statistics. The batch normalization layer contains the mean and variance of the layer input, which can be used as a proxy to confirm that the generator output is similar to the original training data. The KL divergence of two Gaussian distributions can be used to match the mean and variance stored in the bath normalization layer (the mean and variance are obtained from the original data) with the empirical statistics obtained using the generator output.

[0037] Another auxiliary loss term includes the instance class entropy. If the teacher is trained well enough for accurate classification, then the generator output is of interest only when the teacher's class distribution output (i.e., the softmax output) has a small entropy (the probability of one class should be high; if one class has a probability of 1, then minimizing the entropy results in zero). That is, a small entropy is desired for each sampled z.

[0038] The third auxiliary loss term includes the batch class entropy. Assuming that each class appears in the dataset with a similar probability, the class probability distribution averaged over any batch should tend towards a uniform distribution, where maximizing the entropy results in. That is, a high entropy is desired.

[0039] And represent the mean and variance stored in the batch normalization layer l of channel c, where the batch normalization layer l of channel c is learned based on the original training data. And are the corresponding mean and variance calculated for the synthetic samples from the generator. The auxiliary loss of the generator is given by equation (6): (6)

[0040] Where H represents entropy, and is the KL divergence of two Gaussian distributions, which can be expressed as equation (7). (7)

[0041] If in equation (3), then the scheme reduces adversarial belief matching. Adding the auxiliary loss constrains the generator so that it generates synthetic images that produce statistics similar to the original data in the teaching, which helps the minimax optimization avoid any adversarial samples that are very different from the original data and results in better distillation performance. The system and method utilize batch normalization statistics to constrain the generator. To match the mean and variance, the KL divergence of two Gaussian distributions is used, which is a scaled and normalized distance measure.

[0042] The system and method train multiple generators in the data-free KD framework to increase the diversity of the generated samples. Using multiple discriminators can reduce the mode collapse problem in GANs. The minimax optimization is maximized to increase the average KL divergence between the teacher and the students. Obtaining the mean not only reduces the noise in the minimax optimization using stochastic gradient descent but also guides the generator to produce better adversarial samples that do not match each student well on average. The ultimate goal of using multiple generators and multiple students is given by equation (8): (8)

[0043] where is the i-th student and is the j-th generator, where and.

[0044] During the warm-up phase, the generator is specifically trained to minimize the auxiliary loss, so that its output matches the batch normalization statistics and entropy constraints when fed to the teacher. This pre-training procedure reduces the generation of unreliable samples in the early steps of data-free KD. In addition, updating the student more frequently than the generator reduces the probability of falling into any local maximum in the minimax optimization. In the minimization step, we can additionally match the intermediate layer outputs. Implement data-free network quantization by making the student a quantized version of the teacher.

[0045] The above algorithm is shown in Table 1. [Table 1] Algorithm 1 Data-Free Adversarial Knowledge Distillation Generator update interval: m ≥ 1 Warm-up training for the generator (optional): For n: 1 to N warm-up , execute For j: 1 to G, execute End of loop End of loop Adversarial knowledge distillation: For n: 1 to N, execute Maximize: If n = 0 mod m, then For j: 1 to G, execute For i: 1 to S, execute End of loop End of loop End if the following occurs: Minimize For j: 1 to G, execute ← Concatenate End of loop For i: 1 to S, execute End of loop End of loop [

[0046] ]FIG. 3 shows a diagram of a data-free adversarial KD network according to an embodiment. The network 300 includes a generator 302, a student backbone network 304, a first teacher backbone network 306, and a second teacher backbone network 308. The network 300 also includes a discriminator 310, a first fully connected layer 312 with a first softmax layer 314, and a second fully connected layer 316 with a second softmax layer 318. As an alternative embodiment, in addition to or instead of the statistics from the batch normalization layer, the system and method may use any metadata provided by the original model trainer or data owner. For example, the mean and covariance of the feature maps at some intermediate layers can be collected by the original model owner and provided as metadata for model compression and / or quantization. [

[0047] ]Providing metadata can generate a GAN that matches the statistics stored in the metadata and the statistics from the generator output. For example, we can model the final fully connected layer input feature map as a Gaussian mixture and store the Gaussian mixture parameters in the metadata. Then, the GAN is trained to generate the same distribution as the metadata at the teacher. [

[0048] ]In FIG. 3, represents a random input to the generator 302. The generator 302 uses the random input to generate a synthetic image. When the synthetic image is fed into the teacher 306 and the student 304, it generates and serves as the input feature maps to the last fully connected layer 312 and the last fully connected layer 316, respectively. The Gaussian mixture parameters stored in the metadata, where are the input feature maps obtained using the original training data. To generate synthetic data similar to the original data, the statistics of the synthetic data are matched with respect to the teacher 308 by minimizing equation (9). (9) [

[0049] ]This can be performed by the GAN using the minimax optimization in equation (10). (10) [

[0050] ]In this framework, a discriminator 310 is introduced, which evaluates whether the teacher feature maps from the generator output have statistics similar to the statistics stored in the metadata. After training the GAN, the generator 302 is used to generate synthetic data and use the synthetic data for KD. When metadata is provided, the auxiliary loss term can include the distance to the metadata, such as the Jensen-Shannon (JS) divergence.

[0051] FIG. 4 shows a block diagram of an electronic device 401 in a network environment 400 according to an embodiment. Referring to FIG. 4, the electronic device 401 in the network environment 400 may communicate with an electronic device 402 via a first network 498 (e.g., a short-range wireless communication network) or communicate with an electronic device 404 or a server 408 via a second network 499 (e.g., a long-range wireless communication network). The electronic device 401 may communicate with the electronic device 404 via the server 408. The electronic device 401 may include a processor 420, a memory 430, an input device 450, a sound output device 455, a display device 460, an audio module 470, a sensor module 476, an interface 477, a haptic module 479, a camera module 480, a power management module 488, a battery 489, a communication module 490, a subscriber identification module (SIM) 496, or an antenna module 497. In an embodiment, at least one of the components (e.g., the display device 460 or the camera module 480) may be omitted from the electronic device 401, or one or more other components may be added to the electronic device 401. In an embodiment, some of the components may be implemented as a single integrated circuit (IC). For example, the sensor module 476 (e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) may be embedded in the display device 460 (e.g., a display).

[0052] The processor 420 can execute software (such as program 440) to control at least one other component (such as a hardware component or a software component) of the electronic device 401 coupled to the processor 420, and can perform various data processing or operations. As at least a part of the data processing or operations, the processor 420 can load commands or data received from another component (such as the sensor module 476 or the communication module 490) in the volatile memory 432, process the commands or data stored in the volatile memory 432, and store the resulting data in the non-volatile memory 434. The processor 420 can include a main processor 421 (such as a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 423 (such as a graphics processing unit (GPU), an image signal processor (ISP), a sensor aggregator processor, or a communication processor (CP)). The auxiliary processor 423 can operate independently of the main processor 421 or operate in combination with the main processor 421. Additionally or alternatively, the auxiliary processor 423 can be used to consume less power than the main processor 421, or perform specific functions. The auxiliary processor 423 can be implemented separately from the main processor 421, or implemented as a part of the main processor 421.

[0053] The auxiliary processor 423 can replace the main processor 421 when the main processor 421 is in a non-active (such as a sleep) state, or control at least some of the functions or states related to at least one of the components (such as the display device 460, the sensor module 476, or the communication module 490) of the electronic device 401 together with the main processor 421 when the main processor 421 is in an active state (such as executing an application). According to one embodiment, the auxiliary processor 423 (such as an image signal processor or a communication processor) can be implemented as a functionally related part of another component (such as the camera module 480 or the communication module 490).

[0054] The memory 430 can store various data used by at least one component (such as the processor 420 or the sensor module 476) of the electronic device 401. The various data can include, for example, software (such as program 440) and input data or output data for commands related thereto. The memory 430 can include a volatile memory 432 or a non-volatile memory 434.

[0055] The program 440 can be stored as software in the memory 430 and can include, for example, an operating system (OS) 442, middleware 444, or an application program 446.

[0056] The input device 450 can receive commands or data to be used by other components (such as the processor 420) of the electronic device 401 from outside the electronic device 401 (such as a user). The input device 450 can include, for example, a microphone, a mouse, or a keyboard.

[0057] The sound output device 455 can output a sound signal to the outside of the electronic device 401. The sound output device 455 can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or recording, and the receiver can be used for receiving incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0058] The display device 460 can visually provide information to the outside of the electronic device 401 (such as a user). The display device 460 can include, for example, a display, a holographic device, or a projector, and control circuitry for controlling the corresponding one of the display, the holographic device, and the projector. According to one embodiment, the display device 460 can include touch circuitry for detecting a touch or sensor circuitry (such as a pressure sensor) for measuring the intensity of a force caused by the touch.

[0059] The audio module 470 can convert sound into an electrical signal and vice versa. According to one embodiment, the audio module 470 can obtain sound via the input device 450 or output sound via the sound output device 455 or headphones of an external electronic device 402 that is directly (such as wired) or wirelessly coupled to the electronic device 401.

[0060] The sensor module 476 can detect the operating state of the electronic device 401 (such as power or temperature) or the environmental state outside the electronic device 401 (such as the state of a user), and then generate an electrical signal or data value corresponding to the detected state. The sensor module 476 can include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0061] The interface 477 may support one or more specified protocols for an electronic device 401 that is to be used to be directly (e.g., wired) or wirelessly coupled to an external electronic device 402. According to one embodiment, the interface 477 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0062] The connection terminal 478 may include a connector via which the electronic device 401 may be physically connected to the external electronic device 402. According to one embodiment, the connection terminal 478 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0063] The tactile module 479 may convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus, which may be recognized by a user via touch or kinesthesia. According to one embodiment, the tactile module 479 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0064] The camera module 480 may capture a still image or a moving image. According to one embodiment, the camera module 480 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0065] The power management module 488 may manage the power supplied to the electronic device 401. The power management module 488 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0066] The battery 489 may supply power to at least one component of the electronic device 401. According to one embodiment, the battery 489 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0067] The communication module 490 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 401 and an external electronic device (such as the electronic device 402, the electronic device 404, or the server 408), and performing communication via the established communication channel. The communication module 490 can include one or more communication processors that can operate independently of the processor 420 (such as the AP), and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 490 can include a wireless communication module 492 (such as a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 494 (such as a local area network (LAN) communication module or a power line communication (PLC) module). The corresponding one of these communication modules can communicate with the external electronic device via the first network 498 (such as a short-range communication network, such as BluetoothTM, Wi-Fi Direct, or the Infrared Data Association (IrDA) standard) or the second network 499 (such as a long-range communication network, such as a cellular network, the Internet, or a computer network (such as a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (such as a single IC), or can be implemented as multiple components separated from each other (such as multiple ICs). The wireless communication module 492 can use the user information (such as the international mobile subscriber identity (IMSI)) stored in the user identification module 496 to identify and authenticate the electronic device 401 in a communication network (such as the first network 498 or the second network 499).

[0068] The antenna module 497 can transmit signals or power to the outside of the electronic device 401 (such as an external electronic device) or receive signals or power from the outside of the electronic device 401 (such as an external electronic device). According to one embodiment, the antenna module 497 can include one or more antennas, and can select at least one antenna suitable for the communication scheme used in the communication network, such as the first network 498 or the second network 499, from the one or more antennas, for example, by the communication module 490 (such as the wireless communication module 492). Then, signals or power can be transmitted or received between the communication module 490 and the external electronic device via the selected at least one antenna.

[0069] At least some of the above components may be coupled to each other and communicate signals (such as commands or data) therebetween via a peripheral communication scheme (such as a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0070] According to one embodiment, commands or data may be transmitted or received between the electronic device 401 and the external electronic device 404 via the server 408 coupled to the second network 499. Each of the electronic device 402 and the external electronic device 404 may be a device of the same type or a different type as the electronic device 401. All or some of the operations to be performed at the electronic device 401 may be performed at one or more of the external electronic device 402, the external electronic device 404, or the external electronic device 408. For example, if the electronic device 401 is to perform a function or service automatically or in response to a request from a user or another device, then instead of performing the function or service or in addition to performing the function or service, the electronic device 401 may request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices receiving the request may perform at least a part of the requested function or service or additional functions or additional services related to the request, and transmit the result of the performance to the electronic device 401. The electronic device 401 may provide the result as at least a part of a reply to the request with or without further processing the result. For this purpose, techniques such as cloud computing, distributed computing, or master-slave computing may be used.

[0071] One embodiment may be implemented as software (e.g., program 440) including one or more instructions stored in a storage medium (e.g., internal memory 436 or external memory 438) readable by a machine (e.g., electronic device 401). For example, a processor of electronic device 401 may invoke at least one of the one or more instructions stored in the storage medium and execute at least one of the one or more instructions under the control of the processor with or without using one or more other components. Thus, the machine can be operated to perform at least one function according to the at least one invoked instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between the case where data is stored semi-permanently in the storage medium and the case where data is stored temporarily in the storage medium.

[0072] According to one embodiment, the method of the present disclosure may be included in and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM), or directly distributed between two user devices (e.g., smart phones). If distributed online, at least a part of the computer program product may be generated temporarily or stored at least temporarily in a machine-readable storage medium (such as the memory of a manufacturer's server, an application store's server, or a relay server).

[0073] According to one embodiment, each of the above components (e.g., module or program) may include a single entity or multiple entities. One or more of the above components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as each of the multiple components performed one or more functions by the corresponding one of the multiple components before integration. Operations performed by a module, program, or another component may be sequential, parallel, repeated, or exploratory, or one or more of the operations may be performed in a different order or omitted, or one or more other operations may be added.

[0074] Although certain embodiments of the present disclosure have been described in the detailed description of the present disclosure, the present disclosure can be modified in various forms without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be determined solely based on the described embodiments, but should be determined based on the appended claims for patent and their equivalents.

Brief Description of the Drawings

[0076] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG. 1 shows a schematic diagram of a data-free adversarial KD network according to an embodiment. FIG. 2 shows a flowchart of a method for data-free adversarial knowledge distillation according to an embodiment. FIG. 3 shows a schematic diagram of a data-free adversarial KD network according to an embodiment. FIG. 4 shows a block diagram of an electronic device in a network environment according to an embodiment.

Claims

1. A data-free adversarial knowledge distillation method, comprising: Receive random input at the generator; The generator produces a synthesized output of the received random input; The synthesized output is received at the teacher's network; Receive the synthesized output at the student network; minimize the maximum distance between the output of the teacher network and the output of the student network; The generator is constrained by an auxiliary loss term that includes the distance to the metadata when providing the metadata, wherein the metadata is a Gaussian mixture parameter describing the input feature map of the fully connected layer of the teacher network obtained using the original training data in the teacher network.

2. The method as described in request 1, wherein the auxiliary loss term comprises matching statistics from at least one batch normalization layer of the teacher network.

3. The method as described in claim 2, wherein the constraint on the generator includes matching the mean and variance of the at least one batch normalization layer.

4. The method as described in claim 3, wherein matching the mean and variance of the at least one batch normalization layer is performed using the KL divergence of two Gaussian distributions.

5. The method as described in Request 1, wherein the auxiliary loss term includes individual category entropy.

6. The method as described in Request 1, wherein the auxiliary loss term includes batch category entropy.

7. The method as described in claim 1, wherein the maximum value of minimizing the distance between the output of the teacher network and the output of the student network includes matching intermediate layer outputs.

8. A data-free adversarial knowledge distillation system, comprising: Memory; And the processor, configured to receive random input at the generator; The generator produces a synthesized output of the received random input; Receive the synthesized output at the teacher network; receive the synthesized output at the student network; minimize the maximum value of the distance between the output of the teacher network and the output of the student network; The generator is constrained by an auxiliary loss term that includes the distance to the metadata when providing the metadata, wherein the metadata is a Gaussian mixture parameter describing the input feature map of the fully connected layer of the teacher network obtained using the original training data in the teacher network.

9. The system as described in claim 8, wherein the auxiliary loss term comprises matching statistics from at least one batch normalization layer of the teacher network.

10. The system of claim 9, wherein the processor is configured to constrain the generator by matching the mean and variance of the at least one batch normalization layer.

11. The system as claimed in claim 10, wherein matching the mean and variance of the at least one batch normalization layer is performed using KL divergence of two Gaussian distributions.

12. The system as described in claim 8, wherein the auxiliary loss term includes individual category entropy.

13. The system as described in claim 8, wherein the auxiliary loss term includes batch category entropy.

14. The system of claim 8, wherein the maximum value of the distance between the output of the teacher network and the output of the student network is minimized by matching intermediate layer outputs.

Images