Artificial intelligence-based generative model reduction device and method

Abstract

The present disclosure relates to an AI-based apparatus and method for lightweighting a generative model, including a memory for storing data for lightweighting a generative model based on AI, and a processor for performing operations related to the lightweighting of the generative model, wherein the processor assigns randomly initialized scores(s) to each weight for a dense network based on an edge pop-up algorithm, and after searching an arbitrary sub-network, sorts the scores assigned in each forward path, retains weights with the top k% scores already set, and updates the scores using back propagation.

Description

[Technical Field]

[0001] The present disclosure relates to an artificial intelligence-based generative model lightweighting apparatus and method. [Background technology]

[0002] State-of-the-art generative models tend to use very large and complex structures for better performance.

[0003] However, one drawback of large models is that the high computing costs for training limit their application to edge devices such as mobile environments.

[0004] So, naturally, there is a need to design new lightweight architectures or new compression methods for generative modeling.

[0005] Generally, to reduce the weight of AI-based generative models, a training-prune-retraining process is used, but this requires additional networks due to instability and incurs additional costs due to the complex training process.

[0006] Therefore, it is necessary to develop technology that can stably compress artificial intelligence-based generative models and make them lighter without reducing performance. Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide an AI-based generative model lightweighting device and method via a subnetwork that can stably compress a generative model using a subnetwork (strong lottery tickets, SLTs) algorithm for stably searching a network such as a learned generative model among subnetworks, thereby enabling lightweighting without degrading performance.

[0008] The problems to be solved by the present disclosure are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0009] To achieve the above-mentioned technical objectives, an AI-based generative model lightweighting device according to one aspect of the present disclosure includes a memory that stores data for lightweighting a generative model based on AI, and a processor that performs operations related to the lightweighting of the generative model, wherein the processor assigns a randomly initialized score(s) to each weight for a dense network based on an edge pop-up algorithm, and after searching an arbitrary subnetwork, sorts the scores assigned in each forward path, retains weights having the top k% scores already set, and updates the scores using back propagation.

[0010] The processor may also be characterized in that it leaves weights having the top k% scores already set and sets other weights to 0, calculates the loss of the sub-network in the reverse path, and utilizes the backpropagation.

[0011] Furthermore, when calculating the loss of the sub-network, the processor may be characterized by transmitting the image generated through the sub-network and the actual image into an embedding space and calculating a Maximum Mean Discrepancy (MMD) score.

[0012] The processor may also calculate the maximum average mismatch score by matching moments of all orders for the real samples and the fake samples as two sample sets.

[0013] Furthermore, the maximum average mismatch score can be calculated based on the following formula (1):

[0014]

number

[0015] The processor may also be characterized by using a pre-trained VGG network as a kernel for the moment matching.

[0016] Furthermore, the processor may be characterized by repeatedly updating the maximum average mismatch score to find Strong Lottery Tickets (SLTs).

[0017] The system may further include a communication unit electrically connected to the processor and communicating with an external device that provides data for reducing the weight of the generative model.

[0018] Meanwhile, an AI-based generative model lightweighting method according to one aspect of the present disclosure may include a step of assigning a randomly initialized score(s) to each weight for a dense network based on an edge pop-up algorithm, a step of searching for an arbitrary sub-network, a step of sorting the scores assigned in each forward path, and a step of retaining weights having the top k% scores already set and updating the scores using back propagation.

[0019] Also, the updating step may be characterized by leaving weights having the top k% scores already set and setting other weights to 0, and calculating the loss of the sub-network in the reverse path and utilizing the back propagation.

[0020] Furthermore, the updating step may be characterized in that, when calculating the loss of the sub-network, an image generated through the sub-network and an actual image are transmitted to an embedding space to calculate a Maximum Mean Discrepancy (MMD) score.

[0021] The updating step may also be characterized by calculating the maximum average mismatch score by matching moments of all orders for two sample sets, real samples and fake samples.

[0022] Furthermore, the maximum average mismatch score can be calculated based on the following formula (2):

number

[0023] The updating step may be characterized by using a VGG network pre-trained for the moment matching as a kernel.

[0024] Furthermore, the updating step may be characterized by repeatedly updating the maximum average mismatch score to find Strong Lottery Tickets (SLTs).

[0025] In addition, a computer program stored on a computer-readable recording medium that causes a computer to execute a method for realizing the present disclosure can also be provided.

[0026] In addition, a computer-readable recording medium can be provided that records a computer program that causes a computer to execute a method for realizing the present disclosure. [Effects of the Invention]

[0027] According to the solution to the above-mentioned problem of the present disclosure, a subnetwork (strong lottery tickets, SLTs) algorithm is used to stably search for networks such as learned generative models among subnetworks, making it possible to stably compress generative models and reduce their size without reducing performance.

[0028] The effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned above will be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0029] [Figure 1] FIG. 1 is a block diagram of an artificial intelligence neural network that explains the operation for reducing the weight of a general artificial intelligence-based generative model. [Figure 2] FIG. 10 is a diagram illustrating a series of operational procedures for reducing the weight of an artificial intelligence-based generative model via a sub-network according to one embodiment of the present disclosure. [Figure 3] FIG. 10 is a diagram illustrating the operation of searching STLs for lightweighting artificial intelligence-based generative models via sub-networks according to the present disclosure. [Figure 4] FIG. 10 illustrates a diagram illustrating the operation of assigning appropriate scores to weights according to the present disclosure. [Figure 5] 10A and 10B are diagrams illustrating the operation of modeling stable scores through moment matching according to the present disclosure. [Figure 6] 10A and 10B are diagrams illustrating the operation of modeling stable scores through moment matching according to the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating the structure of an artificial intelligence-based generative model reduction device via a sub-network according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram illustrating a method for reducing artificial intelligence-based generative models via a sub-network according to an embodiment of the present disclosure. [Figure 9] FIG. 10 is a diagram showing an example of the results of an experiment conducted to confirm whether SLTs exist in a generative model according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0030] The advantages and features of the present disclosure, as well as methods for achieving them, will become apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and can be realized in various different forms. However, the embodiments are provided to complete the disclosure and to allow those skilled in the art to fully understand the scope of the disclosure, and the disclosure is only defined by the scope of the claims.

[0031] The terms used in this specification are for the purpose of describing the embodiments and are not intended to limit the present disclosure. In this specification, the singular includes the plural unless otherwise specified. The terms "comprises" and / or "comprising" used in this specification do not exclude the presence or addition of one or more other elements other than the elements listed. The same reference numerals refer to the same elements throughout this specification, and "and / or" includes each and every combination of one or more of the listed elements. Even if "first," "second," etc. are used to describe various elements, it should be understood that these elements are not limited by these terms. These terms are used merely to distinguish one element from another. Therefore, it goes without saying that a first element referred to below may also be a second element within the technical spirit of this disclosure.

[0032] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the meaning commonly understood by a person of ordinary skill in the art to which this disclosure belongs. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless they are clearly and specifically defined.

[0033] The same reference numerals refer to the same components throughout this disclosure. This disclosure does not describe all elements of the embodiments, and overlapping content in the general technical field to which the disclosure pertains or in the embodiments will be omitted. The terms "module" or "module" used in this specification refer to software or hardware components such as FPGAs or ASICs, and a "module" or "module" performs a specific function. However, the term "module" or "module" is not limited to software or hardware. A "module" or "module" may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, by way of example, a "module" or "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and "units" or "modules" may be combined into fewer components and "units" or "modules" or may be further separated into additional components and "units" or "modules," etc.

[0034] Throughout this specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, including connection via a wireless communication network.

[0035] Furthermore, when a part is described as "comprising" a certain component, this does not mean that other components are excluded, unless otherwise specified, and means that other components may also be included.

[0036] Throughout this specification, when an element is said to be "on" another element, this includes not only when the element is in contact with the other element, but also when there is another element between the two elements.

[0037] The terms "first," "second," etc. are used to distinguish one component from another, and the components are not limited to the terms described above.

[0038] The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0039] In each step, the identifying numbers are used for convenience of explanation, and the identifying numbers do not describe the order of each step, and each step may be performed in a different order than specified unless the context clearly dictates a specific order.

[0040] The terms used in the following description are defined as follows.

[0041] In this specification, the AI-based pre-learning model is a deep learning-based prediction model, which can predict the probability of re-rupture at the surgical site of a patient before or during surgery and generate prediction information. The deep learning method is not limited, and at least one method can be applied depending on the situation (necessity). Examples of AI algorithms that can be applied include, but are not limited to, a recurrent neural network (RNN) or a transformer. Other AI algorithms can also be applied.

[0042] Although the present specification has been limited to the "weighting device 100," this is a device for weighting and providing an AI-based generative model via a server network, and may include various devices capable of performing computational processing. That is, the weighting device 100 may further include a server, a computer, a server, and / or a portable terminal, or may be in the form of any one of them, but is not limited thereto.

[0043] Here, the computer may include, for example, a notebook computer, a desktop computer, a laptop computer, a tablet PC, a slate PC, etc., equipped with a web browser.

[0044] The server is a server that communicates with external devices and processes information, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, a web server, and the like.

[0045] The portable terminal may be, for example, a wireless communication device that ensures portability and mobility, and may include any kind of handheld-based wireless communication device such as a Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT)-2000, Code Division Multiple Access (CDMA)-2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) terminal, or a smartphone, as well as wearable devices such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0046] Many companies are conducting research into not only pruning methods but also weight quantization methods as a lightweight method for serving generative models.

[0047] Quantization is a method for approximating and expressing weights in artificial intelligence using a smaller amount of information.

[0048] Generally, weights are expressed in binary float32 format, so one weight requires 32 bits of information.

[0049] In the quantization method, these weights are approximated using data that generally uses 8 bits to represent the weights, which has the advantage of efficient memory usage and significantly reducing the calculation speed and complexity.

[0050] However, since the weights are approximated in the quantization process, there are drawbacks in that the performance of the network is reduced or the user must implement an additional library due to the weight approximation method.

[0051] In the present invention, since a randomly initialized network is used to search for a lower-level network without learning, the final weight distribution of the network is determined without change depending on the method for initializing the network weights.

[0052] In this process, it is not necessary to initialize the weight distribution to float32 type, and the performance recorded as a result of the actual invention can be realized as binary weights initialized with Kaiming normal constant. In this case, only three weights including 0 need to be expressed, so all network weights can be expressed using 2 bits.

[0053] In addition, unlike existing methods that require post-processing during the quantization process, the method of the present invention does not require additional post-processing for weight quantization because it selects only lower-level networks without updating weights in networks represented by binary weights.

[0054] Therefore, the present invention not only effectively reduces weight through the lower-level network, but also reduces weight by 16 times compared to a general network and more than four times compared to a general weight quantization method, and effectively overcomes the problem of performance degradation that occurs during the quantization process.

[0055] The working principle and embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0056] FIG. 1 is a block diagram of an artificial intelligence neural network that explains the operation for reducing the weight of a general artificial intelligence-based generative model.

[0057] As shown in Figure 1, a random network, i.e., an untrained model, exhibits poor performance, so it is trained to obtain a high-performance network (trained model). Then, pruning is performed by proposing specific criteria based on the trained model and eliminating weights, but this changes the network structure, reducing its performance. To restore this, retraining is performed, but ultimately, a network with poor performance is generated.

[0058] In other words, the problem of performance degradation is unavoidable due to multiple learning processes during this process.

[0059] As mentioned above, existing pruning algorithms suffer from problems such as excessive weight training costs, poor performance, limited generalization potential, and complex training.

[0060] To solve this problem, the present disclosure aims to search for SLTs, which are sub-networks that achieve excellent generation performance from a generative model without weight updates, by searching for SLTs using moment matching scores. The performance of a pre-trained classifier is utilized to assign scores to randomly initialized weights, and a sparse mask is found to ensure that the sub-network performs similarly to or better than the trained dense network (dense network / dense generator).

[0061] These SLTs are subnetworks at initialization (i.e., without weight updates) that perform similarly to or better than their dense counterparts whose weights have been learned. Here, we use the edge-popup algorithm as the earliest method for exploring SLTs in discriminative models. The edge-popup algorithm selects subnetwork masks based on the idea that the importance of each weight can be scored. Once such scores are assigned, we simply maintain weights with high scores according to the rarity of the desired target.

[0062] Since the performance of this edge pop-up algorithm is highly dependent on the updated scores, which serve as pruning criteria, it is essential to use an appropriate score function for pruning generative models. One might think of adversarial loss, a commonly used criterion for training high-quality generators, but it is very unstable and makes it difficult to find an appropriate score.

[0063] Here, this disclosure leverages a statistical hypothesis testing technique known as Maximum Mean Discrepancy (MMD), which derives a simple instantaneous matching score using features extracted from a fixed pre-trained ConvNet.

[0064] That is, this disclosure combines the edge pop-up algorithm and moment matching scores to provide a stable algorithm for searching for subnetworks with excellent generative performance with very few operations. This does not require weight updates, thereby avoiding the difficult problem of balancing the training and pruning processes. Furthermore, the stable properties of moment matching scores allow SLTs to be searched without additional features.

[0065] Randomly initialized weights θ∈R d After Rm, the goal is to explore SLTs. Next, when exploring SLTs, the mask m ∈ {0, 1} is satisfied such that the pruned neural network G(z; θ m) (where "·" is the symbol for the operation with a dot in the center of a circle) succeeds in the generation task.

[0066] FIG. 2 is a diagram illustrating a series of operational procedures for reducing the weight of an AI-based generative model via a sub-network according to one embodiment of the present disclosure.

[0067] Referring to FIG. 2, when there is a dense network, the lightening device 100 according to an embodiment of the present disclosure assigns arbitrary scores to weights and searches for arbitrary server networks.

[0068] Next, the generated images and real images are sent to the embedding space to calculate the MMD score, and then the server network is continuously updated by updating the previously assigned score and updating the mask.

[0069] In conclusion, we can find SLTs from generative models without weight updates.

[0070] FIG. 3 is a diagram illustrating the operation of searching for STLs for lightweighting of artificial intelligence-based generative models via sub-networks according to the present disclosure.

[0071] As mentioned above, this disclosure uses the edge pop-up algorithm as the first method to search for SLTs from a randomized discriminant network. In this edge pop-up algorithm, when there are randomly initialized weights (ω), it is necessary to find which weights are important among them and assign a score s to indicate how important they are. Specifically, a random score s is assigned to each weight θi. i assign (θ = [θ1, , θ d ].) In this case, we assume that the weights are kept at k%. Then, in each forward pass, the score s i Align the layers and i If the absolute value of belongs to the top k%, then m i = 1, otherwise m i = 0. For the backward path, calculate the network loss and use backpropagation to obtain the score s i Here, we use a straight-through estimator to update s i m i Process the indicator function that maps to

[0072] FIG. 4 is a diagram illustrating the operation of assigning appropriate scores to weights according to the present disclosure.

[0073] On the other hand, to assign an appropriate score, we update the randomly initialized score s using back-propagation. We search the subnetworks of the network to find their output, and then update the overall score using back-propagation. In other words, by updating the score, we can expect to find an appropriate score.

[0074] 5 and 6 are diagrams specifically illustrating the operation of modeling a stable score through moment matching according to the present disclosure.

[0075] On the other hand, pruning a generative model requires an appropriate score update function instead of the cross-entropy loss used in discriminative models. For this purpose, we use the Maximum Mean Discrepancy (MMD), which is known to provide a stable optimization for training generative models.

[0076] Given the actual sample {r t} t=1 N and the fake sample {f t} t=1 M Given two sets of MMD Minimizing L can be interpreted as matching all moments of the model distribution to the empirical data distribution. MMD can be calculated based on the following formula 3.

[0077]

number

[0078] First, in equation (1) of Mathematical Formula 3, Φ denotes a function that matches higher-order moments. Ideally, Φ must be calculated with infinite order. To efficiently calculate MMD, we call this equation (1) using the kernel trick.

[0079] In addition, a pre-trained VGG network is used as a fixed kernel ψ to match the mean μ and covariance σ of the features of real and fake samples in the VGG embedding space. This is because the more powerful the kernel, the better the performance, and the trained network (pre-trained fixed feature extractor) can play the role of a good kernel, so this VGG network is used as the fixed kernel ψ.

[0080] This allows us to calculate the MMD loss through the embedded real and fake data via this VGG network and update the score.

[0081] Meanwhile, Iv, wuv, σ, and α are defined as the input of node v, the network parameters of node u and node v, the activation function, and the learning rate, respectively. The change in score at time step t can be expressed as the following equation 4.

[0082]

number

[0083] It is worth noting that such methods use MMD loss to explore less important nodes rather than learning weights.

[0084] FIG. 7 is a diagram illustrating the structure of an AI-based generative model reduction device via a sub-network according to an embodiment of the present disclosure.

[0085] Referring to FIG. 7, an artificial intelligence-based generative model lightening device (hereinafter referred to as the “lightening device”) 100 via a subnetwork according to one embodiment of the present disclosure may be configured to include a communication unit 110, a memory 130, and a processor 150.

[0086] The communication unit 110 transmits and receives at least one piece of information or data to and from at least one device / terminal. Here, the at least one device / terminal may be a device / terminal that receives a lightweight model that is a lightweight version of an AI-based generative model, or a device / terminal that provides various data / information required to lighten the generative model, and the type and form of the device / terminal are not limited.

[0087] The communication unit 110 can also communicate with other devices, transmitting and receiving wireless signals in a communication network using wireless internet technology.

[0088] Examples of wireless internet technologies include WLAN (Wireless LAN), Wi-Fi (registered trademark) (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (registered trademark) (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (registered trademark) (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), and LTE-A (Long Term Evolution-Advanced), and the lightweight device 100 will send and receive data using at least one of the wireless internet technologies, including internet technologies not listed above.

[0089] The short-range communication network may support short-range communication using at least one of Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee (registered trademark), NFC (Near Field Communication), Wi-Fi (Wireless Fidelity), Wi-Fi Direct, and Wireless Universal Serial Bus (Wireless USB) technologies. Such short-range wireless communication networks may support wireless communication between the weight reduction device 100 and at least one user terminal (not shown). In this case, the short-range wireless communication network may be a wireless personal area network.

[0090] The memory 130 may store data for at least one process (algorithm) for providing a lightweight AI-based generative model via a sub-network, or a program that reproduces the process. In addition, the memory 130 may further store processes for performing other operations, but is not limited thereto.

[0091] The memory 130 may store various information / data necessary for lightweighting and providing an AI-based generative model via a sub-network, as well as various other data supporting various functions of the lightening device 100. The memory 130 may store a number of application programs (or applications) run by the lightening device 100, and data and commands for the operation of the lightening device 100. At least some of these application programs may be downloaded from an external server via wireless communication. Meanwhile, the application programs may be stored in the memory 130, installed on the lightening device 100, and executed via the processor 150 to perform operations (or functions) based on the data stored in the memory 130.

[0092] Meanwhile, the memory 130 may include at least one type of storage medium selected from the group consisting of flash memory, hard disk, multimedia card micro, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The memory may store information temporarily, permanently, or semi-permanently, and may be provided as a built-in or removable type.

[0093] Furthermore, the memory 130 may construct a database that stores various information necessary to provide a lightweight artificial intelligence-based generative model via a sub-network, or may be connected to a separate external server (including a cloud server).

[0094] Meanwhile, in addition to operations related to application programs, the processor 150 controls all components within the lightweighting device 100 to process input or output signals, data, information, etc., or executes instructions, algorithms, and application programs stored in at least one memory to perform various processes, and can provide or process appropriate information or functions to lightweight and provide an artificial intelligence-based generative model via a subnetwork.

[0095] Specifically, processor 150 assigns a randomly initialized score(s) to each weight for the dense network based on an edge pop-up algorithm, and after searching any sub-network, sorts the scores assigned in each forward pass, retains the weights with the top k% scores already set, and updates the scores using backpropagation.

[0096] At this time, the processor 150 leaves the weights having the top k% scores already set and sets the other weights to 0, and calculates the loss of the sub-network in the backward path to utilize back propagation.

[0097] Meanwhile, when calculating the loss of a sub-network, the processor 150 transmits the image generated through the sub-network and the actual image into an embedding space and calculates a maximum mean discrepancy (MMD) score.

[0098] At this time, the processor 150 calculates the actual samples {r t} t=1 N and fake samples {f t} t=1 M By matching all orders of moments for , the maximum average discrepancy can be calculated.

[0099] Specifically, as described above, the maximum average mismatch can be calculated based on Equation 3.

[0100] FIG. 8 is a diagram illustrating a method for reducing artificial intelligence-based generative models via a sub-network according to an embodiment of the present disclosure.

[0101] Referring to FIG. 8, the lightening device 100 assigns a randomly initialized score(s) to each weight for the dense network based on an edge pop-up algorithm (S210), and searches (searches) any sub-network (S220).

[0102] Next, the lightweighting device 100 sorts the scores assigned in each forward path in the sub-network searched by step S220 (S230), retains the weights with the top k% scores already set, and updates the scores using backpropagation (S240).

[0103] FIG. 9 shows an example of the results of an experiment conducted to confirm whether SLTs exist in a generative model according to the present disclosure, comparing the FID scores of a sub-network and a trained dense network (GFMN:LSUN-Bedroom).

[0104] Referring to Figure 9, we consider pruning a randomly initialized neural network without weight updates, where we visualize the FID for various k, the percentage of remaining weights in the pruned sub-network.

[0105] At this time, the server network was calculated while changing the proportion of remaining weights, i.e., the proportion of remaining weights, and its performance was shown. The FID score is an index that indicates the performance of the generative model, and the lower the index, the better.

[0106] In Figure 9, the solid line shows the performance of the trained model, and the dotted line shows the change in performance of the subnetwork depending on the amount of weight (k) remaining after applying the algorithm to a random network. It can be seen that the performance improves as k decreases. Performance is poor at high k points, but this is a natural phenomenon as the higher k, the closer it is to a random dense network. In contrast, the performance of the generative model improves as k decreases, and it can be seen that when k reaches 10%, the performance of the subnetwork overlaps with that of the trained dense network.

[0107] As a result, the locations are considered SLTs.

[0108] On the other hand, since the present disclosure uses a model realized by an artificial neural network method to perform inference for a predetermined purpose, the following will examine the artificial neural network in detail.

[0109] A model in this specification may refer to a network function, an artificial neural network, and / or any form of computer program that operates based on a neural network. In this specification, the terms model, network function, and neural network can be used interchangeably. A neural network has one or more nodes connected to each other via one or more links, forming a relationship between an input node and an output node within the neural network. The characteristics of a neural network can be determined by the number of nodes and links within the neural network, the correlation between the nodes and links, and the weight value assigned to each link. A neural network can be composed of a set of one or more nodes. A subset of the nodes that make up a neural network can form a layer.

[0110] A deep neural network (DNN) is a neural network that includes multiple hidden layers in addition to an input layer and an output layer, and a deep neural network has one or more, preferably two or more, intermediate hidden layers.

[0111] Such deep neural networks can include convolutional neural networks (CNNs), vision transformers, recurrent neural networks (RNNs), long short-term memory (LSTM) networks, generative pre-trained transformers (GPTs), autoencoders, generative adversarial networks (GANs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, generative adversarial networks (GANs), transformers, and others.

[0112] Alternatively, depending on the embodiment, the deep neural network may be a model trained using a transfer learning method. Here, transfer learning refers to a learning method in which a large amount of unlabeled training data is pre-trained using a semi-supervised or self-supervised learning method to obtain a pre-trained model (or base part) having a first task using a predetermined method (MLM, NSP, etc.), and labeled training data is trained using a supervised learning method to fine-tune the pre-trained model to suit a second task, thereby realizing a target model. One example of a model trained using such a transfer learning method is BERT (Bidirectional Encoder Representations from Transformers), but is not limited to this.

[0113] The above description of the deep neural network is merely an example, and the present disclosure is not limited thereto. The convolutional neural network described above includes a feature extraction unit (feature learning) that extracts features from an image and a classification unit (classification) that performs classification using the extracted features. The feature extraction unit may include, but is not limited to, a convolutional layer that extracts features from an image using a kernel, a ReLU layer, which is an activation function, and a pooling layer that reduces the dimensionality of the data. The classification unit may include, but is not limited to, a flattening layer that aligns the features extracted from the feature extraction unit, a fully connected layer where classification is essentially performed, and a softmax function.

[0114] A neural network can be trained by at least one of supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. Training a neural network can be a process of applying knowledge to the neural network to make it perform a particular operation.

[0115] Neural networks can be trained to minimize output errors. Training a neural network involves repeatedly inputting training data into the neural network, calculating the error between the neural network's output and the target for the training data, backpropagating the neural network's error from the output layer to the input layer in a direction that reduces the error, and updating the weights of each node in the neural network. In supervised learning, labeled data with a correct answer can be used for each training data, while in unsupervised learning, unlabeled data can be used for each training data. The amount of change in the connection weights of each node that is updated can be determined by the learning rate. The neural network calculation for input data and backpropagation of the error constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. In addition, methods such as increasing the amount of training data, regularization, dropout (which deactivates some nodes), and batch normalization layers can be applied to prevent overfitting.

[0116] Meanwhile, the model disclosed in one embodiment may borrow at least a portion of a Transformer. The Transformer may be configured with an encoder that encodes embedded data and a decoder that decodes the encoded data. The Transformer may have a structure that receives a sequence of data, goes through encoding and decoding steps, and outputs a sequence of data of a different type. In one embodiment, the sequence of data may be processed into a form that can be operated by the Transformer. The process of processing the sequence of data into a form that can be operated by the Transformer may include an embedding process. Expressions such as data token, embedding vector, embedding token, etc. may refer to data embedded in a form that can be processed by the Transformer.

[0117] In order for a transformer to encode and decode a series of data, the encoder and decoder in the transformer can be processed using an attention algorithm. The attention algorithm may refer to an algorithm that calculates the similarity of one or more keys for a given query, reflects the obtained similarity in values ​​corresponding to each key, and then calculates an attention value by weighting the values ​​reflecting the similarity.

[0118] Attention algorithms can be classified into various types depending on how the query, key, and value are set. For example, if the query, key, and value are all set to the same value and attention is calculated, this may be referred to as a self-attention algorithm. If the dimension of the embedding vector is reduced to process a series of input data in parallel, and attention is calculated by calculating a separate attention head for each divided embedding vector, this may be referred to as a multi-head attention algorithm.

[0119] In one embodiment, the transformer can be composed of modules that perform multiple multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer can also include additional components other than attention algorithms, such as embedding, normalization, and softmax. Methods for configuring the transformer using attention algorithms can include those disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0120] A transformer can be applied to various data domains, such as embedded natural language, segmented image data, and audio waveforms, to convert a series of input data into a series of output data. To convert data having various data domains into a series of data that can be input to the transformer, the transformer can embed data. The transformer can process additional data representing the relative positional or phase relationship between the series of input data. Alternatively, vectors representing the relative positional or phase relationship between the input data can be further reflected and embedded in the series of input data. As an example, the relative positional relationship between the series of input data can include, but is not limited to, the order of words in a natural language sentence, the relative positions of each segmented image, the time order of segmented audio waveforms, etc. The process of adding information representing the relative positional or phase relationship between the series of input data can be referred to as positional encoding.

[0121] The program may include code written in a computer language such as C, C++, JAVA, or machine language that is read by the computer's processor (CPU) through a device interface of the computer to load the program into a computer and cause the computer to execute the method implemented by the program. This code may include functional code related to functions defining the functions necessary to execute the method, and may also include control code related to execution procedures necessary for the computer's processor to execute the function in a predetermined order. This code may also include memory reference-related code for determining at which location (address) in the computer's internal or external memory additional information or media necessary for the computer's processor to execute the function should be referenced. Furthermore, if communication with a remote computer or server is required to cause the computer's processor to execute the function, the code may also include communication-related code for determining how to communicate with the remote computer or server using the computer's communication module, and what information or media should be sent or received during communication.

[0122] The storage medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers to which the computer can be connected, or various recording media on the user's computer. Furthermore, the medium can be distributed among computer systems connected via a network, storing computer-readable code in a distributed manner.

[0123] The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in software modules executed by hardware, or in a combination thereof, which may reside in Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, hard disk, removable disk, CD-ROM, or any other form of computer-readable storage medium known in the art to which the present invention pertains.

[0124] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art will understand that the present disclosure may be embodied in other specific forms without changing the technical spirit or essential features thereof. Therefore, the above-described embodiments should be understood as illustrative in all respects and not restrictive.

Claims

1. a memory for storing data for lightweighting a generative model based on artificial intelligence; a processor that performs operations related to reducing the generative model; Including, The processor: An AI-based generative model lightweighting device, characterized by assigning a randomly initialized score (s) to each weight for a dense network based on an edge pop-up algorithm, searching for an arbitrary sub-network, sorting the scores assigned in each forward path, retaining weights having the top k% scores already set, and updating the scores using backpropagation.

2. The processor: The AI-based generative model lightweighting device of claim 1, wherein weights having the top k% scores already set are left, other weights are set to 0, and the loss of the sub-network is calculated in the backward path, and the backpropagation is utilized.

3. The processor:

2. The AI-based generative model lightweighting device of claim 1, wherein, when calculating the loss of the sub-network, an image generated through the sub-network and an actual image are transmitted to an embedding space to calculate a maximum mean discrepancy (MMD) score.

4. The processor:

4. The AI-based generative model reduction device of claim 3, wherein the maximum average discrepancy score is calculated by matching moments of all orders for two sample sets, real samples and fake samples.

5. The maximum average discrepancy score is:

5. The AI-based generative model lightening device according to claim 4, wherein the calculation is performed based on the following formula (1): [Equation 1] Here, r ｉ represents the actual sample, and f ｊ represents a fake sample.

6. The processor:

5. The AI-based generative model reduction device according to claim 4, wherein a pre-trained VGG network is used as a kernel for moment matching.

7. The processor: The AI-based generative model lightweighting device of claim 6, wherein the device finds Strong Lottery Tickets (SLTs) by repeatedly updating the maximum average mismatch score.

8. electrically connected to the processor; The AI-based generative model lightweighting device according to claim 1 , further comprising a communication unit for communicating with an external device that provides data for lightweighting the generative model.

9. 1. An AI-based generative model reduction method executed by a device, assigning a randomly initialized score(s) to each weight for the dense network based on an edge pop-up algorithm; searching for an arbitrary sub-network; aligning the scores assigned in each forward pass; Retaining the weights having the top k% scores already set and updating the scores using backpropagation; A method comprising:

10. The updating step includes: The method according to claim 9, characterized in that weights having the top k% scores already set are left, other weights are set to 0, and the loss of the sub-network is calculated in the backward path, and the backpropagation is utilized.

11. The updating step includes:

10. The method of claim 9, wherein when calculating the loss of the sub-network, the image generated through the sub-network and the actual image are transmitted to an embedding space to calculate a maximum mean discrepancy (MMD) score.

12. The updating step includes:

12. The method of claim 11, wherein the maximum average mismatch score is calculated by matching all orders of moments for two sample sets, real samples and fake samples.

13. The method of claim 12, wherein the maximum average mismatch score is calculated based on the following formula: [Equation 2] Here, r ｉ represents the actual sample, and f ｊ represents a fake sample.

14. The updating step includes: The method of claim 12, wherein the kernel is a VGG network pre-trained for moment matching.

15. The updating step includes:

15. The method of claim 14, wherein the operation of updating the maximum average mismatch score is repeated to find Strong Lottery Tickets (SLTs).

Images