Apparatus and method for lightweighting generative models for artificial intelligence infrastructure

Abstract

The present disclosure relates to an AI-based apparatus and method for lightweighting a generative model, including a memory for storing data for lightweighting a generative model based on AI, and a processor for performing operations related to the lightweighting of the generative model, wherein the processor assigns randomly initialized scores(s) to each weight for a dense network based on an edge pop-up algorithm, and after searching an arbitrary sub-network, sorts the scores assigned in each forward path, retains weights with the top k% scores already set, and updates the scores using back propagation.

Description

[Technical Field]

[0001] This disclosure relates to an apparatus and method for reducing the weight of generative models for artificial intelligence. [Background technology]

[0002] State-of-the-art generative models tend to employ very large and complex structures for better performance.

[0003] However, one drawback of large models is that the high computing costs for training limit their applicability to edge devices such as mobile environments.

[0004] Therefore, it becomes necessary to design new lightweight architectures or new methods for compressing generative modeling.

[0005] Generally, generative models for artificial intelligence foundations undergo a training-prune-retraining process to reduce their size. However, this process has problems: instability necessitates the use of additional networks, and the complex training process incurs additional costs.

[0006] Therefore, it is necessary to develop a technology that can stably compress generative models for artificial intelligence foundations, enabling them to be made lighter without degrading performance. [Overview of the project] [Problems that the invention aims to solve]

[0007] This disclosure has been made in view of the above circumstances, and its purpose is to provide a device and method for lightweighting generative models on an artificial intelligence platform via a subnetwork, which can stably compress generative models using a subnetwork (strong lottery tickets, SLTs) algorithm for stably searching for networks such as learned generative models among subnetworks, thereby reducing their weight without performance degradation.

[0008] The issues that this disclosure aims to address are not limited to those mentioned above, and other issues not mentioned can be clearly understood by an average engineer from the following description. [Means for solving the problem]

[0009] An artificial intelligence-based generative model lightweighting device relating to one aspect of the present disclosure for achieving the technical challenges described above includes a memory for storing data for lightweighting a generative model based on artificial intelligence, and a processor that performs operations related to lightweighting the generative model, wherein the processor assigns randomly initialized scores(s) to each weight of a dense network based on an edge pop-up algorithm, explores arbitrary subnetworks, sorts the scores assigned in each forward path, retains the weights having the top k% scores already set, and can update the scores by utilizing backpropagation.

[0010] Furthermore, the processor may be characterized by retaining the weights that have already set the top k% scores, setting the other weights to 0, calculating the loss of the subnetwork in the reverse path, and utilizing the backpropagation.

[0011] Furthermore, the processor may be characterized in that, when calculating the loss of the subnetwork, it transmits the image generated via the subnetwork and the actual image to the embedding space and calculates the Maximum Mean Discrepancy (MMD) score.

[0012] Furthermore, the processor may be characterized by calculating the maximum average mismatch score by matching all order moments for the actual sample and the false sample as two sample sets.

[0013] Furthermore, the maximum average discrepancy score can be calculated based on Equation (1) below.

[0014] <l [Equation] where r , , , ,

[0016] , , ,

[0018] , ,

[0017] , ,

[0020] , , ,

[0015] ,

[0019] , represents an actual sample, and f j represents a fake sample.

[0015] Also, the processor can be characterized by using a pre-trained VGG network for the moment matching as a kernel.

[0016] Furthermore, the processor can be characterized by repeatedly performing an operation of updating the maximum average discrepancy score to find SLTs (Strong Lottery Tickets).

[0017] Also, it can further include a communication unit that is electrically connected to the processor and communicates with an external device that provides data for lightweighting the generation model.

[0018] On the other hand, a method for lightweighting a generation model of an artificial intelligence infrastructure according to an aspect of the present disclosure can include: assigning randomly initialized scores (s) to respective weights of a dense network based on an edge pop-up algorithm; exploring an arbitrary sub-network; aligning the scores assigned in each forward path; leaving weights having a previously set top k% of scores, and updating the scores by utilizing backpropagation.

[0019] Also, the updating step can be characterized by leaving weights having a previously set top k% of scores, setting other weights to 0, calculating the loss of the sub-network in the reverse path, and utilizing the backpropagation.

[0020] Furthermore, the updating step may be characterized by the fact that, when calculating the loss of the subnetwork, the image generated via the subnetwork and the actual image are transmitted to the embedding space to calculate the Maximum Mean Discrepancy (MMD) score.

[0021] Furthermore, the updating step may be characterized by calculating the maximum average mismatch score by matching all order moments for the actual sample and the false sample as two sample sets.

[0022] Furthermore, the maximum average mismatch score can be characterized by being calculated based on the following formula 2.

number

[0023] Furthermore, the updating step may be characterized by using a VGG network pre-trained for moment matching as the kernel.

[0024] Furthermore, the updating step can be characterized by repeatedly performing the operation of updating the maximum average mismatch score in order to find SLTs (Strong Lottery Tickets).

[0025] In addition, we can provide a computer program stored on a computer-readable recording medium that causes a computer to execute a method for realizing this disclosure.

[0026] In addition, we can provide a computer-readable recording medium for recording a computer program that causes a computer to execute a method for realizing this disclosure. [Effects of the Invention]

[0027] According to the solution to the aforementioned problem described in this disclosure, generative models can be stably compressed using a subnetwork (strong lottery tickets, SLTs) algorithm for stably searching for networks such as learned generative models among subnetworks, thereby reducing their size without performance degradation.

[0028] The effects of this disclosure are not limited to those mentioned above, and any other effects not mentioned can be clearly understood by an ordinary engineer from the following description. [Brief explanation of the drawing]

[0029] [Figure 1] This is a block diagram of an artificial intelligence neural network illustrating how to optimize the generative model of a common AI foundation. [Figure 2] This figure shows a series of operational procedures for lightweighting a generative model of an artificial intelligence platform via a subnetwork, according to one embodiment of the present disclosure. [Figure 3] This diagram illustrates the operation of searching for STLs to optimize the generative model of the artificial intelligence infrastructure via the subnetwork related to this disclosure. [Figure 4] This diagram specifically illustrates the operation of assigning appropriate scores to weights as described in this disclosure. [Figure 5] This diagram specifically illustrates the operation of modeling a stable score through moment matching, as disclosed in this disclosure. [Figure 6] This diagram specifically illustrates the operation of modeling a stable score through moment matching, as disclosed in this disclosure. [Figure 7] This figure shows the structure of a device for lightweighting a generative model for an artificial intelligence platform via a subnetwork, according to one embodiment of the present disclosure. [Figure 8] This figure shows a method for reducing the size of a generative model for an artificial intelligence platform via a subnetwork, according to one embodiment of the present disclosure. [Figure 9] This figure shows an example of experimental results obtained to confirm whether or not SLTs exist in the generative model. [Modes for carrying out the invention]

[0030] The advantages and features of this disclosure, and the methods for achieving them, will become clear with reference to the embodiments described below in detail with the accompanying drawings. However, this disclosure is not limited to the embodiments disclosed below and can be implemented in a variety of different forms. These embodiments are provided to complete the disclosure and to enable a person of the ordinary skill in the art to fully understand the scope of this disclosure, and the disclosure is defined only by the claims.

[0031] The terms used herein are for illustrative purposes only and are not intended to limit the disclosure. In this specification, singular terms include plural terms unless otherwise specified. The terms “comprises” and / or “comprising” used in the specification do not exclude the existence or addition of one or more other components in addition to the components mentioned. Throughout the specification, the same reference numerals indicate the same component, and “and / or” includes each of the components mentioned and all combinations of one or more of them. Even if terms such as “first,” “second,” etc., are used to describe a variety of components, these components are not limited by these terms. These terms are used simply to distinguish one component from another. Accordingly, the first component mentioned below may also be the second component within the technical concept of this disclosure.

[0032] Unless otherwise defined, all terms used herein (including technical and scientific terms) are used in the sense that they would be commonly understood by an ordinary person skilled in the art to which this disclosure pertains. Furthermore, terms defined in commonly used dictionaries shall not be interpreted ideally or excessively unless explicitly defined otherwise.

[0033] Reference numerals identical throughout this disclosure indicate the same component. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure belongs or content that is redundant in the embodiments is omitted. As used in this specification, the terms “part” or “module” mean components of software, FPGA or ASIC, and “part” or “module” plays a specific role. However, “part” or “module” is not limited to software or hardware. A “part” or “module” may be configured to reside on an addressable storage medium and may be configured to regenerate one or more processors. Thus, as an example, a “part” or “module” includes components such as software components, object-oriented software components, class components and task components, and processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. The components and the functions provided within a “part” or “module” can be combined with fewer components and “parts” or “modules,” or further separated into additional components and “parts” or “modules,” etc.

[0034] When a part of the specification is described as being "connected" to another part, this includes not only direct connections but also indirect connections, and indirect connections include connections via wireless communication networks.

[0035] Furthermore, when a part is described as "containing" a certain component, unless otherwise specified, this does not mean that other components are excluded, but rather that other components may be included.

[0036] Throughout the specification, when a member is described as being "on top of" another member, this includes not only cases where the member is in contact with another member, but also cases where another member exists between the two members.

[0037] Terms such as "first," "second," etc., are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0038] Unless otherwise clearly stated in the context, singular expressions include plural forms.

[0039] In each stage, the identification codes are used for explanatory purposes only and do not indicate the order of the stages. The stages may be performed in a different order than specified unless the context explicitly states otherwise.

[0040] The terms used in the following explanation are defined as follows:

[0041] In this specification, the pre-trained artificial intelligence-based model is a deep learning-based predictive model that can predict the probability of re-rupture at the surgical site of a patient before or during surgery and generate predictive information. In this case, the deep learning method is not limited, and at least one method may be applied depending on the situation (necessity). In this case, examples of artificial intelligence algorithms that may be applied include RNN (Recurrent Neural Network) or transformer, but are not limited to these, and other artificial intelligence algorithms may also be applied.

[0042] Although this specification has described the "lightweighting device 100" in isolation, it is a device for providing a lightweight generative model for an artificial intelligence base via a server network, and can include all kinds of devices capable of performing computational processing. That is, the lightweighting device 100 may further include a server, a computer, a server and / or a portable terminal, or any one of these forms, and is not limited thereto.

[0043] Here, the computer may include, for example, a laptop computer, desktop computer, laptop computer, tablet PC, or slate PC equipped with a web browser.

[0044] The server in question is a server that communicates with external devices to process information, and may include application servers, computing servers, database servers, file servers, game servers, mail servers, proxy servers, and web servers.

[0045] The aforementioned portable terminal is, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, and smartphones, as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0046] Many companies are conducting extensive research not only on pruning methods but also on weight quantization methods as lightweighting techniques for providing generative models as services.

[0047] Quantization techniques are methods for approximating and representing the weights of artificial intelligence using less information.

[0048] Generally, weights are represented as the binary float32 type, meaning that each weight requires 32 bits of information.

[0049] Quantization techniques have the advantage of approximating these weights with data that typically uses 8 bits, thereby enabling efficient memory usage and significantly reducing computational speed and complexity.

[0050] However, the quantization process has the disadvantage of either reducing the network's performance or requiring the user to implement additional libraries, as weights are represented by approximations.

[0051] In this invention, since a randomly initialized network explores sub-networks without training, the final weight distribution of the network is determined without changing the method used to initialize the network's weights.

[0052] In this process, it is not always necessary to initialize the weight distribution to a float32 type. The performance recorded as a result of the actual invention can be realized as binary weights initialized with kaiming normal constants. In this case, only three weights, including 0, need to be represented, so all the network weights can be represented using 2 bits.

[0053] Furthermore, unlike existing methods that require post-processing during the quantization process, the present invention selects only the lower-level networks without updating the weights in a network represented by binary weights, thus eliminating the need for additional post-processing for weight quantization.

[0054] Therefore, the present invention not only enables effective weight reduction through the lower-level network, but also achieves weight reduction of 16 times compared to general networks and more than 4 times compared to general weight quantization methods, while also effectively overcoming the performance degradation problem that occurs during the quantization process.

[0055] The operating principle and embodiments of this disclosure will be described below with reference to the attached drawings.

[0056] Figure 1 is a block diagram of an artificial intelligence neural network illustrating the operation for lightweighting generative models in a common AI foundation.

[0057] As shown in Figure 1, a random network, i.e., an untrained model, exhibits poor performance. Therefore, it is trained to obtain a high-performance network (a trained model). Subsequently, pruning is performed by proposing specific criteria based on this trained model and removing weights, but in this case, the network structure changes, and the network's performance decreases. Retraining is performed to restore this performance, but in conclusion, a low-performance network is generated.

[0058] In other words, the problem of performance degradation inevitably occurs because learning is performed multiple times during that process.

[0059] As mentioned earlier, existing pruning algorithms suffer from problems such as excessive weight learning costs, poor performance, limited generalizability, and complex learning.

[0060] To address these issues, this disclosure aims to explore SLTs (Subnetworks with High Generative Performance) from a generative model without weight updates, by exploring SLTs through Moment Matching Scores. The performance of a pre-trained classifier is leveraged to assign scores to randomly initialized weights, and a sparse mask is found to ensure the subnetwork resembles, or performs better than, the trained dense network / dense generator.

[0061] These SLTs are subnetworks in their initial state (i.e., before weight updates) that perform similarly to, or better than, a dense counterpart with learned weights. Here, we use the edge-popup algorithm as the earliest method for exploring SLTs in a discriminative model. This algorithm selects subnetwork masks based on the idea that the importance of each weight can be scored. Once such scores are assigned, we only need to maintain weights with high scores according to the rarity of the desired target.

[0062] The performance of this edge-popup algorithm is heavily dependent on the updated score used as the pruning criteria; therefore, using an appropriate score function is essential for pruning the generative model. While adversarial loss, a commonly used criterion for training high-quality generators, is often considered, it is highly unstable and hinders the discovery of an appropriate score.

[0063] Herein, this disclosure utilizes a statistical hypothesis testing method known as Maximum Mean Discrepancy (MMD). This method derives a simple instantaneous matching score using features extracted from a fixed, pre-trained ConvNet.

[0064] In other words, this disclosure provides a stable algorithm that combines an edge pop-up algorithm and a moment matching score to search for subnetworks with excellent generative performance with very little operation. This avoids the difficult problem of balancing training and pruning processes because it does not require weight updates. Furthermore, the stable properties of the moment matching score allow for the search of SLTs without additional functionality.

[0065] Randomly initialized weights θ∈R d We generate a neural network (z;θ) having the following characteristics. After Rm, the goal is to search for SLTs. When the goal is to search for SLTs again, the mask m ∈ {0, 1} satisfies the requirement that the pruned neural network G(z;θ·m) (where "·" is an operation symbol with a dot in the center of a circle) can successfully perform the generation task.

[0066] Figure 2 shows a series of operational steps for lightweighting a generative model of an artificial intelligence platform via a subnetwork, according to one embodiment of the present disclosure.

[0067] Referring to Figure 2, the weight reduction device 100 according to one embodiment of the present disclosure, when a dense network is present, assigns arbitrary scores to the weights and searches for an arbitrary server network.

[0068] Next, the generated images and real images are transmitted to the embedding space to calculate the MMD score. The server network is then continuously updated by updating the previously assigned score and updating the mask.

[0069] In conclusion, SLTs can be found from the generative model without updating the weights.

[0070] FIG. 3 is a diagram for explaining an operation of searching for STLs for lightweighting a generation model of an artificial intelligence infrastructure via a subnetwork according to the present disclosure.

[0071] As described above, the present disclosure uses an edge pop-up algorithm as the most initial method of searching for SLTs from a randomized discrimination network. In this edge pop-up algorithm, when there are randomly initialized weights (ω), since it is necessary to find which weight is important among those weights, a score s for how important it is is assigned. Specifically, a random score s i is assigned to each weight θi (θ = [θ1, ···, θ d .). At this time, it is assumed that the weights are maintained at k%. Then, the scores s i in each layer in each forward path are aligned, and if the absolute value of s i belongs to the top k% within the corresponding layer, m i = 1 is assigned, and otherwise m i = 0 is assigned. For the reverse path, the loss of the network is calculated, and the score s i is updated using backpropagation. Here, an indicator function that maps s i to m i is processed using a straight-through estimator.

[0072] FIG. 4 is a diagram specifically explaining an operation of assigning an appropriate score to a weight according to the present disclosure.

[0073] On the other hand, in order to assign an appropriate score, the randomly initialized score s is updated using backpropagation. The subnetwork of the network is searched to obtain its output, and this is used to update the overall score by backpropagation. That is, by updating the score, it is expected to find an appropriate score.

[0074] Figures 5 and 6 illustrate in detail the operation of modeling a stable score through moment matching as described in this disclosure.

[0075] On the other hand, pruning a generative model requires an appropriate score update function instead of the cross-entropy loss used in discriminative models. For this purpose, we use Maximum Mean Discrepancy (MMD), which is known to provide a stable optimization for training generative models.

[0076] Given actual sample {r t} t=1 N and a fake sample {f t} t=1 M Given two sets, the MMD loss L MMD Minimizing this L can be interpreted as making the model distribution match the empirical data distribution at every moment. MMD This can be calculated based on equation 3 below.

[0077]

number

[0078] First, in equation (1) from Math 3, Φ is a function that matches the higher-order moments. Ideally, Φ must be calculated at an infinite order. To efficiently calculate MMD, we invoke equation (1) using a kernel trick.

[0079] Furthermore, a pre-trained VGG network is used as a fixed kernel ψ to match the mean μ and covariance σ of the actual and spurious sample features in the VGG embedding space. This is because the more powerful the kernel used, the better the performance, and since a pre-trained network (Pretrained fixed feature extractor) can play the role of a good kernel, this VGG network is used as the aforementioned fixed kernel ψ.

[0080] This allows the system to calculate MMD loss through embedded real and fake data via the VGG network and update the score.

[0081] On the other hand, Iv, wuv, σ, and α are defined as the input to node v, the network parameters of nodes u and v, the activation function, and the learning rate, respectively. The change in score at time step t can be expressed as shown in equation 4 below.

[0082]

number

[0083] This method is noteworthy because it uses MMD loss to explore less important nodes, rather than to learn weights.

[0084] Figure 7 shows the structure of a device for lightweighting generative models for an artificial intelligence platform via a subnetwork, according to one embodiment of the present disclosure.

[0085] Referring to Figure 7, the generation model lightweighting device (hereinafter referred to as the "lightweighting device") 100 for an artificial intelligence platform via a subnetwork according to one embodiment of the present disclosure can be configured to include a communication unit 110, a memory 130, and a processor 150.

[0086] The communication unit 110 transmits and receives at least one piece of information or data to and from at least one device / terminal. Here, the at least one device / terminal can be a device / terminal that receives a lightweight model of the artificial intelligence base generation model, or a device / terminal that provides various data / information necessary to lighten the generation model, and its type and form are not limited.

[0087] Furthermore, the communication unit 110 can also communicate with other devices and transmits and receives wireless signals in a communication network using wireless internet technology.

[0088] Wireless internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Registered Trademark) (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Registered Trademark) (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (Registered Trademark) (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), and LTE-A (Long Term Evolution-Advanced). The lightweight device 100 will transmit and receive data using at least one wireless internet technology, including internet technologies not listed above.

[0089] This device is for short-range communication and can support short-range communication using at least one of the following technologies: Bluetooth®, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee®, NFC (Near Field Communication), Wi-Fi® (Wireless-Fidelity), Wi-Fi Direct, or Wireless USB (Wireless Universal Serial Bus). Such a short-range wireless communication network (Wireless Area Network) can support wireless communication between the lightweight device 100 and at least one user terminal (not shown). In this case, the short-range wireless communication network may be a short-range wireless personal communication network (Wireless Personal Area Network).

[0090] Memory 130 can store data for at least one process (algorithm) for providing a lightweight generative model of an artificial intelligence base via a subnetwork, or for a program that replicates that process. In addition, memory 130 can store, without limitation, processes for performing other operations.

[0091] Memory 130 can store various information / data necessary for providing a lightweight generative model for the artificial intelligence infrastructure via a subnetwork, as well as other diverse data that supports the diverse functions of the lightweight device 100. Memory 130 can store numerous application programs (applications) driven by the lightweight device 100, data for the operation of the lightweight device 100, and instruction words. At least some of these application programs can be downloaded from an external server via wireless communication. On the other hand, the application programs are stored in memory 130, installed on the lightweight device 100, and executed via the processor 150 to perform operations (or functions) based on the data stored in memory 130.

[0092] On the other hand, memory 130 may include at least one type of storage medium from among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Furthermore, memory can store information temporarily, permanently, or semi-permanently and can be provided as an internal or removable type.

[0093] Furthermore, memory 130 can either build a database to store various information necessary for providing a lightweight generative model for the artificial intelligence platform via a subnetwork, or it can be linked with a separate external server (including a cloud server).

[0094] On the other hand, in addition to operations related to the application program, the processor 150 can control all the components within the lightweight device 100 to process input or output signals, data, information, etc., or execute various processes by running instructions, algorithms, and application programs stored in at least one memory, and provide or process appropriate information or functions for lightweighting and providing a generative model of the artificial intelligence base via the subnetwork.

[0095] Specifically, the processor 150 assigns a randomly initialized score(s) to each weight in the dense network based on an edge-popup algorithm, explores an arbitrary subnetwork, sorts the scores assigned in each forward path to retain the weights with the already set top k% scores, and updates the scores using backpropagation.

[0096] At this time, the processor 150 sets the other weights to 0, leaving the weights that have already been set to have the top k% score, and in the reverse path, it calculates the loss of the subnetwork and utilizes backpropagation.

[0097] On the other hand, when the processor 150 calculates the subnetwork loss, it transmits the image generated via the subnetwork and the actual image to the embedding space and calculates the Maximum Mean Discrepancy (MMD) score.

[0098] At this time, processor 150 uses two sample sets to represent the actual sample {r t} t=1 N and fake sample {f t} t=1 M By matching moments of all orders for this, the maximum mean mismatch can be calculated.

[0099] Specifically, as mentioned above, the maximum average mismatch can be calculated based on equation 3.

[0100] Figure 8 shows a method for reducing the size of a generative model for an artificial intelligence platform via a subnetwork, according to one embodiment of the present disclosure.

[0101] Referring to Figure 8, the lightweighting device 100 assigns randomly initialized scores (s) to each weight of the dense network based on the edge pop-up algorithm (S210), and searches for an arbitrary subnetwork (S220).

[0102] Next, the lightweight device 100 sorts the scores assigned to each forward path in the subnetwork explored in step S220 (S230), retains the weights that have already been set to the top k%, and updates the scores using backpropagation (S240).

[0103] Figure 9 shows an example of experimental results obtained to confirm whether or not SLTs exist in the generative model as described herein, and is a figure showing an example of comparing the FID scores of a subnetwork and a trained dense network (GFMN:LSUN-Bedroom).

[0104] Referring to Figure 9, when a randomly initialized neural network is pruned without weight updates, we visualize the FID for various values ​​of k, which is the percentage of remaining weights in the pruned subnetwork.

[0105] In this case, the server network was calculated while changing the proportion of weights remaining, i.e., the percentage of weights retained, and its performance was shown. The FID score is an indicator of the performance of the generative model, and a lower score indicates a better performance.

[0106] In Figure 9, the solid line shows the performance of the trained model, and the dotted line shows the change in subnetwork performance as the amount of weights (k) remaining after applying the algorithm to the random network increases. It can be seen that performance improves as k decreases. Performance is poor at high k values, but this is a natural phenomenon because the network is closer to a random dense network at higher k values. In contrast, the performance of the generative model improves as k decreases, and it can be confirmed that when k reaches 10%, the subnetwork performance overlaps with that of the trained dense network.

[0107] As a result, that location appears to be an SLT (Steel Landing Structure).

[0108] On the other hand, this disclosure uses a model implemented with an artificial neural network to perform predictions (inference) for a predetermined purpose; therefore, the artificial neural network will be described in detail below.

[0109] In this specification, "model" may mean a network function, an artificial neural network, and / or any form of computer program that operates on a neural network. In this specification, "model," "network function," and "neural network" can be used interchangeably. A neural network consists of one or more nodes connected to one or more links, forming relationships between input and output nodes within the neural network. The properties of a neural network can be determined by the number of nodes and links within the neural network, the relationships between nodes and links, and the weight values ​​assigned to each link. A neural network can consist of a set of one or more nodes. A subset of nodes that make up a neural network can constitute a layer.

[0110] A deep neural network (DNN) is a neural network that includes multiple hidden layers in addition to the input and output layers. In a deep neural network, the intermediate hidden layers consist of one or more, preferably two or more.

[0111] Such deep neural networks can include convolutional neural networks (CNNs), vision transformers, recurrent neural networks (RNNs), Long Short-Term Memory (LSTM) networks, Generative Pre-trained Transformers (GPTs), autoencoders, Generative Adversarial Networks (GANs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siam networks, Generative Adversarial Networks (GANs), and transformers.

[0112] Alternatively, depending on the embodiment, the deep neural network may be a model trained using transfer learning. Here, transfer learning refers to a learning method in which a large amount of unlabeled training data is pre-trained using semi-supervised or self-supervised learning to obtain a pre-trained model (or base part) having a first task through a pre-configured method (MLM and NSP), and then the pre-trained model is trained using labeled training data in a supervised learning manner to fine-tune it to suit a second task, thereby realizing the target model. One example of a model trained using such transfer learning is BERT (Bidirectional Encoder Representations from Transformers), but it is not limited to this.

[0113] The description of the deep neural network mentioned above is merely illustrative, and this disclosure is not limited thereto. In the case of the convolutional neural network mentioned above, it consists of a feature learning unit that extracts features from an image, and a classification unit that performs classification using the features thus extracted. The feature learning unit may, but is not limited to, a convolutional layer that extracts features from an image using a kernel, a ReLU layer which is one of the activation functions, and a pooling layer to reduce the dimensionality of the data. The classification unit may, but is not limited to, a flatten layer which arranges the features extracted from the feature learning unit in a line, a fully connected layer which essentially performs the classification, and a softmax function.

[0114] Neural networks can be trained using at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. Learning a neural network can be the process by which the neural network applies knowledge to perform specific actions.

[0115] Neural networks can be trained to minimize output errors. Neural network training involves iteratively inputting training data, calculating the network's output and target error for the training data, and backpropagating the network's errors from the output layer to the input layer to reduce errors, thereby updating the weights of each node in the neural network. Supervised learning uses labeled data where the correct answer is labeled for each training data point, while unsupervised learning uses unlabeled data where the correct answer is not labeled for each training data point. The amount of change in the linked weights of each node being updated can be determined by the learning rate. The neural network's calculations for the input data and the backpropagation of errors constitute a training cycle (epoch). The learning rate can be applied differently depending on the number of iterations in the neural network's training cycle. Furthermore, to prevent overfitting, methods such as increasing the training data, regularization, dropout (which deactivates some nodes), and batch normalization layers can be applied.

[0116] On the other hand, the model disclosed in one embodiment can borrow at least a part of a transformer. The transformer may consist of an encoder that encodes embedded data and a decoder that decodes the encoded data. The transformer may have a structure that receives a series of data, goes through the encoding and decoding stages, and outputs a series of data of different types. In one embodiment, the series of data can be processed into a form that the transformer can compute. The process of processing the series of data into a form that the transformer can compute may include an embedding process. Expressions such as data tokens, embedding vectors, and embedding tokens may mean data embedded in a form that the transformer can process.

[0117] To encode and decode a series of data, the transformer can utilize an attention algorithm for its encoder and decoder. An attention algorithm can be defined as an algorithm that calculates the similarity between a given query and one or more keys, reflects this similarity in the values ​​corresponding to each key, and then calculates an attention value by weighting these similarity-reflected values.

[0118] Attention algorithms can be classified into various types depending on how the query, key, and value are set. For example, if attention is sought by setting the query, key, and value to be all the same, this may be called a self-attention algorithm. If attention is sought by reducing the dimensionality of the embedding vectors and finding a separate attention head for each divided embedding vector in order to process a series of input data in parallel, this may be called a multi-head attention algorithm.

[0119] In one embodiment, the transformer may consist of modules that perform multiple multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer may also include additional components other than attention algorithms, such as embedding, normalization, and softmax. Methods for configuring the transformer using attention algorithms may include the method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0120] The transformer can be applied to diverse data domains, such as embedded natural language, segmented image data, and audio waveforms, to transform a series of input data into a series of output data. To transform data with diverse data domains into a series of data that can be input to the transformer, the transformer can embed data. The transformer can process additional data that represents the relative positional or phase relationships between a series of input data. Alternatively, vectors representing the relative positional or phase relationships between the input data can be further reflected in the series of input data and embedded within the series of input data. For example, the relative positional relationships between a series of input data may include, but are not limited to, word order in a natural language text, the relative positional relationships of individual segmented images, or the temporal order of segmented audio waveforms. The process of adding information that represents the relative positional or phase relationships between a series of input data can be called positional encoding.

[0121] The aforementioned program may include code encoded in a computer language such as C, C++, Java®, or machine language, which is read by the computer's processor (CPU) via the computer's device interface, in order to load the program into the computer and execute the method implemented by the program. Such code may include functional code related to functions that define the functions necessary to execute the method, and may include execution procedure-related control code necessary for the computer's processor to execute the functions in a predetermined order. Furthermore, such code may further include memory reference-related code indicating where (address) in the computer's internal or external memory should be referenced for additional information or media necessary for the computer's processor to execute the functions. In addition, if communication with any other computer or server located remotely is required for the computer's processor to execute the functions, the code may further include communication-related code indicating how to communicate with any other computer or server located remotely using the computer's communication module, and what information or media should be sent and received during communication.

[0122] The aforementioned storage medium refers not to a medium that stores data for a short time, such as a register, cache, or memory, but rather to a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In other words, the program can be stored on various recording media on various servers to which the computer can connect, or on various recording media on the user's computer. Furthermore, the medium can be distributed across computer systems connected via a network, and can store code that can be read by computers in a distributed manner.

[0123] The steps of the methods or algorithms described in relation to embodiments of the present invention can be implemented directly in hardware, in software modules executed by hardware, or in combination thereof. The software modules may always reside on RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, hard disk, removable disk, CD-ROM, or any form of computer-readable recording medium known in the art to which the present invention belongs.

[0124] Although embodiments of the present invention have been described above with reference to the attached drawings, a person ordinary in the art to which this disclosure belongs will understand that this disclosure can be implemented in other specific forms without altering its technical idea or essential features. Accordingly, the embodiments described above should be understood in all respects as illustrative and not restrictive.

Claims

1. Memory for storing data to lighten the generative model based on artificial intelligence, A processor that performs operations related to the optimization of the aforementioned generation model, Includes, The aforementioned processor, Based on the edge-popup algorithm, randomly initialized scores (s) are assigned to each weight in the dense network, an arbitrary subnetwork is explored, the scores assigned in each forward path are sorted, the weights with the already set top k% scores are kept, and the scores are updated using backpropagation. When calculating the loss of the subnetwork, the image generated via the subnetwork and the actual image are transmitted to the embedding space, and the Maximum Mean Discrepancy (MMD) score is calculated by matching all order moments for the actual sample and the false sample as two sample sets. A VGG network pre-trained for moment matching is used as the kernel. An artificial intelligence-based generative model lightweighting device characterized by repeatedly performing the operation of updating the aforementioned maximum average mismatch score to find SLTs (Strong Lottery Tickets).

2. The aforementioned processor, The generative model lightweighting device for an artificial intelligence base according to claim 1, characterized in that it retains weights with already set top k% scores, sets other weights to 0, calculates the loss of the subnetwork in the reverse path, and utilizes the backpropagation.

3. The aforementioned maximum average mismatch score is, The device for reducing the generation model of an artificial intelligence base according to claim 1, characterized in that it is calculated based on the following formula 1. [Math 1] Here, ri represents an actual sample, and fj represents a false sample.

4. The aforementioned processor is electrically connected, The artificial intelligence base generation model lightweighting device according to claim 1, further comprising a communication unit that communicates with an external device that provides data for lightweighting the generation model.

5. In a method for reducing the size of generative models for artificial intelligence infrastructure executed by a device, Based on the edge pop-up algorithm, the step involves assigning randomly initialized scores (s) to each weight in a dense network, The stage of exploring arbitrary subnetworks, The stage of aligning the scores assigned to each forward path, The process involves retaining weights with already established top k% scores and updating the scores using backpropagation. Includes, The aforementioned update stage is: When calculating the loss of the subnetwork, the image generated via the subnetwork and the actual image are transmitted to the embedding space, and the Maximum Mean Discrepancy (MMD) score is calculated by matching all order moments for the actual sample and the false sample as two sample sets. A VGG network pre-trained for moment matching is used as the kernel. A method characterized by including the search for SLTs (Strong Lottery Tickets) by repeatedly performing the operation of updating the aforementioned maximum average mismatch score.

6. The aforementioned update stage is: The method according to 5, characterized in that weights having already set the top k% score are retained, other weights are set to 0, the loss of the subnetwork is calculated in the reverse path, and the backpropagation is utilized.

7. The method according to claim 5, characterized in that the aforementioned maximum average mismatch score is calculated based on the following formula 2. [Math 2] Here, ri represents an actual sample, and fj represents a false sample.

Images