Apparatus and method for lightweighting generative model based on artificial intelligence

Abstract

The present disclosure relates to an apparatus and a method for lightweighting a generative model based on artificial intelligence, the apparatus comprising: a memory for storing data for lightweighting a generative model based on artificial intelligence; and a processor for performing an operation related to the lightweighting of the generative model, wherein the processor can: assign a randomly initialized score (S) to each weight for a dense network on the basis of an edge pop-up algorithm; after finding an arbitrary subnetwork, align the assigned scores in each forward path; and update the scores by using backpropagation while leaving a weight having a preset upper k% score.

Description

Device and method for lightweighting artificial intelligence-based generative models

[0001] The present disclosure relates to a lightweight device and method for generating a model based on artificial intelligence.

[0002] State-of-the-art generative models tend to use very large and complex structures for better performance.

[0003] However, one drawback of large models is that their high computational cost for training limits their application to edge devices such as mobile environments.

[0004] Therefore, there is a natural need to design new lightweight architectures or new compression methods using generative modeling.

[0005] In general, to make an artificial intelligence-based generative model lightweight, it goes through the process of training - pruning - retraining, but there are problems such as requiring an additional network due to instability or requiring additional costs due to a complex learning process.

[0006] Therefore, there is a need to develop a technology that can reliably compress AI-based generative models and reduce their weight without compromising performance.

[0007] The present disclosure provides a device and method for reducing the weight of an artificial intelligence-based generative model through a sub-network, which can stably compress a generative model and reduce its weight without deteriorating performance, by using a sub-network (strong lottery tickets, SLTs) algorithm to stably find a network such as a learned generative model among sub-networks.

[0008] The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] According to one aspect of the present disclosure for achieving the above-described technical task, an artificial intelligence-based generative model lightweighting device comprises: a memory for storing data for lightweighting a generative model based on artificial intelligence; and a processor for performing operations related to lightweighting the generative model, wherein the processor can assign a randomly initialized score (s) to each weight for a dense network based on an edge pop-up algorithm, find an arbitrary sub-network, sort the assigned scores in each forward path, and update the scores by utilizing backpropagation while leaving weights having preset top k% scores.

[0010] In addition, the processor may be characterized in that it leaves the weights having the preset top k% scores, makes other weights 0, calculates the loss of the sub-network in the reverse path, and utilizes the backpropagation.

[0011] In addition, the processor may be characterized in that, when calculating the loss of the sub-network, it transmits the image generated through the sub-network and the actual image to an embedding case to calculate a maximum mean discrepancy (MMD) score.

[0012] Additionally, the processor may be characterized in that it calculates the maximum average mismatch score by matching moments of all orders for real samples and fake samples as two sets of samples.

[0013] In addition, the maximum average mismatch score may be characterized as being calculated based on the following <Mathematical Formula 1>.

[0014] [Mathematical Formula 1]

[0015]

[0016] Here, r i represents a real sample, and fj represents a fake sample.

[0017] Additionally, the processor may be characterized by having a pre-trained VGG network as a kernel for the moment matching.

[0018] Additionally, the processor may be characterized in that it finds SLTs (Strong Lottery Tickets) by repeatedly performing an operation of updating the maximum average discrepancy score.

[0019] In addition, the device may further include a communication unit that is electrically connected to the processor and communicates with an external device that provides data for reducing the weight of the generation model.

[0020] Meanwhile, a method for lightweighting an artificial intelligence-based generative model according to one aspect of the present disclosure may include a step of assigning a randomly initialized score(s) to each weight for a dense network based on an edge pop-up algorithm; a step of finding a random sub-network; a step of sorting the assigned scores in each forward path; and a step of updating the scores using backpropagation while leaving weights with a preset top k% score.

[0021] In addition, the updating step may be characterized by leaving the weights having the preset top k% scores, making other weights 0, calculating the loss of the sub-network in the reverse path, and utilizing the backpropagation.

[0022] In addition, the updating step may be characterized by calculating a maximum mean discrepancy (MMD) score by transmitting an image generated through the sub-network and an actual image to an embedding case when calculating the loss of the sub-network.

[0023] Additionally, the updating step may be characterized by calculating the maximum average mismatch score by matching all orders of moments for real and fake samples as two sample sets.

[0024] In addition, the maximum average mismatch score may be characterized as being calculated based on the following <Mathematical Formula 1>.

[0025] [Mathematical Formula 1]

[0026]

[0027] Here, r i represents a real sample, and f j represents a fake sample.

[0028] Additionally, the updating step may be characterized by using a pre-trained VGG network as a kernel for the moment matching.

[0029] In addition, the updating step may be characterized by finding SLTs (Strong Lottery Tickets) by repeatedly performing an operation of updating the maximum average discrepancy score.

[0030] In addition, a computer program stored in a computer-readable recording medium for executing a method for implementing the present disclosure may be further provided.

[0031] In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

[0032] According to the aforementioned problem solving means of the present disclosure, a subnetwork (strong lottery tickets, SLTs) algorithm for stably finding a network such as a learned generative model among subnetworks can be used to stably compress the generative model and make it lightweight without deteriorating performance.

[0033] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0034] Figure 1 is a block diagram of an artificial intelligence neural network to explain the operation for lightweighting a general artificial intelligence-based generative model.

[0035] FIG. 2 is a diagram illustrating a series of operational procedures for lightweighting an artificial intelligence-based generative model through a sub-network according to one embodiment of the present disclosure.

[0036] FIG. 3 is a diagram for explaining the operation of finding STLs for lightweighting an artificial intelligence-based generative model through a sub-network according to the present disclosure.

[0037] FIG. 4 is a drawing specifically explaining an operation of assigning an appropriate score to a weight according to the present disclosure.

[0038] FIG. 5 and FIG. 6 are drawings specifically explaining an operation of modeling a stable score through moment matching according to the present disclosure.

[0039] FIG. 7 is a diagram showing the structure of an artificial intelligence-based lightweight generation model device through a sub-network according to one embodiment of the present disclosure.

[0040] FIG. 8 is a diagram illustrating a method for lightweighting an artificial intelligence-based generation model through a sub-network according to one embodiment of the present disclosure.

[0041] FIG. 9 is a diagram showing an example of the results of an experiment conducted to confirm whether SLTs exist in a generation model according to the present disclosure.

[0042] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure that the present disclosure is complete and to fully inform those skilled in the art of the scope of the present disclosure, and the present disclosure is defined solely by the scope of the claims.

[0043] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present disclosure. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the specification, and "and / or" includes each and any combination of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present disclosure.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which this disclosure pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0045] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general contents in the technical field to which this disclosure belongs or contents overlapping between embodiments are omitted. The term "part" or "module" as used in the specification means a hardware component such as a software or FPGA or ASIC, and the "part" or "module" performs certain roles. However, the "part" or "module" is not limited to software or hardware. The "part" or "module" may be configured to be in an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the "part" or "module" includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and “sub-components” or “modules” may be combined into a smaller number of components and “sub-components” or “modules” or further separated into additional components and “sub-components” or “modules”.

[0046] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0047] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0048] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0049] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0050] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0051] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0052] The terms used in the following explanation are defined as follows.

[0053] In this specification, the AI-based pre-training model is a deep learning-based predictive model, and each of these can be used to predict the probability of re-rupture of a patient's surgical site before or during surgery, thereby generating predictive information. The deep learning method is not limited, and at least one method may be applied depending on the situation (need). Examples of AI algorithms that can be applied include, but are not limited to, a Recurrent Neural Network (RNN) or a Transformer, and other AI algorithms may also be applied.

[0054] Although the description herein is limited to a 'lightweight device (100)', this is a device for providing an artificial intelligence-based generative model in a lightweight form through a server network, and may include various devices capable of performing computational processing. That is, the lightweight device (100) may further include a server, a computer, a server, and / or a portable terminal, or may be in the form of any one of them, but is not limited thereto.

[0055] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0056] The above server is a server that processes information by communicating with external devices, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0057] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0058] In order to service generative models in various companies, a lot of research is being conducted on not only pruning methods but also weight quantization methods as lightweight techniques.

[0059] Quantization is a technique for approximating and expressing the weights of artificial intelligence with less information.

[0060] Since the commonly used weights are expressed in binary float32 format, each weight requires 32 bits of information.

[0061] In quantization techniques, the weights are expressed by approximating them with data that generally uses 8 bits, which has the advantage of greatly reducing efficient memory usage, computational speed, and computational complexity.

[0062] However, since the quantization process approximates the weights, there is a disadvantage that the network's performance may decrease depending on the method of approximating the weights or that the user must implement an additional library.

[0063] In the present invention, since a randomly initialized network finds a subnetwork without learning, the final weight distribution of the network is determined without change depending on the method of initializing the weights of the network.

[0064] This process does not necessarily require initializing the weight distribution to a float32 format. The performance recorded as a result of the actual invention can be implemented as binary weights initialized with the network's weight distribution as a Kaiming normal constant. In this case, only three weights, including 0, are required to be expressed, so all network weights can be expressed using 2 bits.

[0065] In addition, unlike existing techniques that require post-processing during the quantization process, the method of the present invention does not require additional post-processing for weight quantization by selecting only sub-networks without updating weights in a network expressed as binary weights.

[0066] Therefore, the present invention can not only effectively reduce weight through a subnetwork, but also reduce weight by 16 times compared to a general network and by more than 4 times compared to a general weight quantization technique, and can effectively overcome the problem of performance degradation during the quantization process.

[0067] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0068] Figure 1 is a block diagram of an artificial intelligence neural network to explain the operation for lightweighting a general artificial intelligence-based generative model.

[0069] As illustrated in Figure 1, a random network, i.e., an untrained model, exhibits poor performance. Therefore, it is trained to obtain a high-performance network (a trained model). Subsequently, pruning is performed based on the trained model, using specific criteria to remove weights. However, this process alters the network structure, resulting in reduced performance. Retraining is then performed to restore the network, ultimately resulting in a low-performance network.

[0070] That is, as learning progresses multiple times in the process, the problem of performance deterioration is bound to arise.

[0071] As discussed above, existing pruning algorithms suffer from problems such as excessive weight training cost, poor performance, limited generalizability, and complex training.

[0072] To address these issues, the present disclosure seeks to identify subnetworks (SLTs) that achieve superior generation performance in generative models without weight updates. This is accomplished by using moment matching scores. The performance of pre-trained classifiers is leveraged to assign scores to randomly initialized weights, and a sparse mask is identified to ensure that the subnetwork performs similarly to or better than the trained dense network (dense generator).

[0073] These SLTs are subnetworks that, upon initialization (i.e., without weight updates), perform similarly or better than their dense counterparts with trained weights. Here, we use the edge-popup algorithm as the initial method for finding SLTs in a discriminative model. This edge-popup algorithm selects subnetwork masks based on the idea that the importance of each weight can be scored. Once these scores are assigned, the desired target sparsity is achieved by simply retaining weights with high scores.

[0074] Since the performance of this edge pop-up algorithm is heavily dependent on the updated score, which serves as the pruning criterion, using an appropriate score function is essential for pruning the generative model. While adversarial loss, a commonly used criterion for training high-quality generators, is easily considered, it is highly unstable and hinders the search for an appropriate score.

[0075] Here, the present disclosure utilizes a statistical hypothesis testing technique known as Maximum Mean Discrepancy (MMD). This technique derives a simple instantaneous matching score using features extracted from a fixed, pre-trained ConvNet.

[0076] In other words, the present disclosure provides a robust algorithm for finding subnetworks with superior generative performance in very sparse systems by combining the edge pop-up algorithm and the moment matching score. This algorithm avoids the challenging problem of balancing training and pruning processes, as it does not require weight updates. Furthermore, the robust nature of the moment matching score allows for the identification of SLTs without the need for additional features.

[0077] Randomly initialized weights θ ∈ R dGenerate a neural network (z; θ) with Rm. Next, we aim to find SLTs. When we aim to find SLTs, the mask m ∈ {0, 1} satisfies that the pruned neural network G(z; θ⊙m) performs well in the generation task.

[0078] FIG. 2 is a diagram illustrating a series of operational procedures for lightweighting an artificial intelligence-based generative model through a sub-network according to one embodiment of the present disclosure.

[0079] Referring to FIG. 2, a lightweight device (100) according to one embodiment of the present disclosure assigns an arbitrary score to a weight when there is a dense network and finds an arbitrary server network.

[0080] Next, the server network is continuously updated by sending the generated images and real images to the embedding space to calculate the MMD score, and then updating the mask by updating the previously assigned score.

[0081] In conclusion, we can find SLTs in generative models without weight updates.

[0082] FIG. 3 is a diagram for explaining the operation of finding STLs for lightweighting an artificial intelligence-based generative model through a sub-network according to the present disclosure.

[0083] As previously explained, the present disclosure utilizes the edge pop-up algorithm as the earliest method for finding SLTs in randomized discriminative networks. In this edge pop-up algorithm, when there are randomly initialized weights (ω), it is necessary to find which weight among the weights is important, and thus assigns a score s indicating how important it is. Specifically, for each weight θi, a random score s is assigned. i Assign (θ = [θ1, · · · , θd ]). At this time, it is assumed that the weight is kept at k%. Then, in each forward pass, the score si at each layer is sorted by |s within that layer. i |If m is in the top k% i = assign 1, otherwise m i = 0 is assigned. For the backward path, the loss of the network is calculated and the score s is calculated using backpropagation. i Update . Here, we use a straight-through estimator to find s i to m i Processes the indicator function that maps to .

[0084] FIG. 4 is a drawing specifically explaining an operation of assigning an appropriate score to a weight according to the present disclosure.

[0085] Meanwhile, to assign appropriate scores, backpropagation is used to update the randomly initialized score s. This involves finding a subnetwork within the network, obtaining its output, and then updating the overall score through backpropagation. In other words, by updating the score, we can expect to find an appropriate score.

[0086] FIG. 5 and FIG. 6 are drawings specifically explaining an operation of modeling a stable score through moment matching according to the present disclosure.

[0087] Meanwhile, pruning a generative model requires an appropriate score update function instead of the cross-entropy loss used in discriminative models. To achieve this, we utilize Maximum Mean Discrepancy (MMD), known to provide a stable optimization for training generative models.

[0088] given actual sample and fake samples Given two sets of MMD loss L MMD Minimizing L can be interpreted as matching every moment of the model distribution to the empirical data distribution. This L MMD can be calculated based on the following <Mathematical Formula 1>.

[0089] [Mathematical Formula 1]

[0090]

[0091] First, in equation (1) represents a function that matches higher-order moments. Ideally, must be calculated with infinite order. To efficiently calculate MMD, this equation (1) is called through the kernel trick.

[0092] Additionally, we use a pretrained VGG network with a fixed kernel ψ, matching the mean μ and covariance σ of the features of real and fake samples in the VGG embedding space. This is because utilizing a stronger kernel can lead to better performance, and the trained network (pretrained fixed feature extractor) can serve as a good kernel, so we use this VGG network with a fixed kernel ψ.

[0093] In this way, the MMD loss is calculated using the embedded real data and fake data through this VGG network and the score is updated.

[0094] Meanwhile, Iv, wuv, σ, and α are defined as the input of node v, the network parameters of nodes u and v, the activation function, and the learning rate, respectively. The change in the score at time step t can be expressed as in <Mathematical Formula 2> below.

[0095] [Equation 2]

[0096]

[0097] It is worth noting that this method uses the MMD loss to find low-importance nodes rather than learning weights.

[0098] FIG. 7 is a diagram showing the structure of an artificial intelligence-based lightweight generation model device through a sub-network according to one embodiment of the present disclosure.

[0099] Referring to FIG. 7, an artificial intelligence-based generation model lightweight device (hereinafter referred to as a “light device”) (100) through a sub-network according to one embodiment of the present disclosure may be configured to include a communication unit (110), a memory (130), and a processor (150).

[0100] The communication unit (110) transmits and receives at least one piece of information or data with at least one device / terminal. Here, the at least one device / terminal may be a device / terminal that receives a lightweight model that is a lightweight version of an artificial intelligence-based generation model, or a device / terminal that provides various data / information necessary to lightweight the generation model, and the type and form thereof are not limited.

[0101] In addition, the communication unit (110) can perform communication with other devices and transmit and receive wireless signals in a communication network according to wireless Internet technologies.

[0102] Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), and LTE-A (Long Term Evolution-Advanced), and the lightweight device (100) transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

[0103] As for short range communication, short range communication can be supported using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. Such short range wireless communication networks (Wireless Area Networks) can support wireless communication between a lightweight device (100) and at least one user terminal (not shown). In this case, the short range wireless communication network may be a short range wireless personal area network (Wireless Personal Area Networks).

[0104] The memory (130) can store data for at least one process (algorithm) for providing a lightweight artificial intelligence-based generative model through a sub-network, or a program that reproduces the process. In addition, the memory (130) can further store processes for performing other operations, but this is not limited thereto.

[0105] The memory (130) can store various information / data required to provide a lightweight artificial intelligence-based generation model through a sub-network, as well as other various data that support various functions of the lightweight device (100). The memory (130) can store a plurality of application programs (or applications) driven by the lightweight device (100), data for the operation of the lightweight device (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. Meanwhile, the application programs can be stored in the memory (130), installed on the lightweight device (100), and driven to perform operations (or functions) based on the data stored in the memory (130) through the processor (150).

[0106] Meanwhile, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory may store information temporarily, permanently, or semi-permanently, and may be provided as a built-in or removable type.

[0107] In addition, the memory (130) may be configured to build a database that stores various information necessary to provide a lightweight artificial intelligence-based generation model through a sub-network, or may be linked to a separate external server (including a cloud server).

[0108] Meanwhile, in addition to operations related to the application program, the processor (150) controls all components within the lightweight device (100) to process input or output signals, data, information, etc., or executes commands, algorithms, and application programs stored in at least one memory to perform various processes, and can provide or process appropriate information or functions for providing a lightweight artificial intelligence-based generation model through a sub-network.

[0109] Specifically, the processor (150) assigns a randomly initialized score (s) to each weight for the dense network based on the edge pop-up algorithm, finds a random sub-network, sorts the assigned scores in each forward path, leaves the weights with the preset top k% scores, and updates the scores using backpropagation.

[0110] At this time, the processor (150) leaves the weights with the preset top k% scores and makes the other weights 0, and in the reverse path, calculates the loss of the sub-network to utilize backpropagation.

[0111] Meanwhile, when calculating the loss of a sub-network, the processor (150) sends the image generated through the sub-network and the actual image to the embedding case to calculate the maximum mean discrepancy (MMD) score.

[0112] At this time, the processor (150) generates the actual sample as two sample sets. and fake samples By matching all orders of moments, the maximum average disparity can be calculated.

[0113] Specifically, as explained above, the maximum average discrepancy can be calculated based on the above [Mathematical Formula 1].

[0114] FIG. 8 is a diagram illustrating a method for lightweighting an artificial intelligence-based generation model through a sub-network according to one embodiment of the present disclosure.

[0115] Referring to FIG. 8, the lightweight device (100) assigns a randomly initialized score (s) to each weight for the dense network based on the edge pop-up algorithm (S210) and finds (searches) an arbitrary sub-network (S220).

[0116] Next, the lightweight device (100) sorts the scores assigned in each forward path in the sub-network found by step S220 (S230), and updates the scores using backpropagation while leaving the weights with the preset top k% scores (S240).

[0117] FIG. 9 is a diagram showing an example of the results of an experiment conducted to confirm whether SLTs exist in a generative model according to the present disclosure, and is a diagram showing an example of comparing the FID scores of a sub-network and a trained dense network (GFMN; LSUN-Bedroom).

[0118] Referring to Figure 9, when pruning a randomly initialized neural network without weight update, the FID is visualized for various k, which is the proportion (%) of the remaining weights of the pruned sub-network.

[0119] At this time, the server network was obtained by changing the ratio of remaining weights, that is, how much weight to leave, and its performance was shown. The FID score is an indicator of the performance of the generation model, and the lower the score, the better the indicator.

[0120] In Figure 9, the solid line represents the performance of the learned model, and the dotted line represents the change in performance of the subnetwork according to the amount of remaining weight (k) when the algorithm is applied to a random network. As can be seen, the performance improves as k decreases. At high k points, the performance deteriorates, which is natural because the higher k is, the closer it is to a random dense network. On the other hand, as k decreases, the performance of the generative model improves, and when k reaches 10%, the performance of the subnetwork overlaps with the learned dense network.

[0121] As a result, those points can be viewed as SLTs.

[0122] Meanwhile, the present disclosure performs prediction (inference) for a predetermined purpose using a model implemented in an artificial neural network manner. In the following, the artificial neural network will be examined.

[0123] A model in this specification may refer to any form of computer program that operates based on a network function, an artificial neural network, and / or a neural network. Throughout this specification, the terms model, neural network, network function, and neural network may be used interchangeably. A neural network is a network in which one or more nodes are interconnected through one or more links to form input node and output node relationships within the neural network. The characteristics of a neural network can be determined based on the number of nodes and links within the neural network, the correlation between the nodes and links, and the weight value assigned to each link. A neural network may be composed of a set of one or more nodes. A subset of the nodes constituting the neural network may constitute a layer.

[0124] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to the input layer and the output layer, and the hidden layer in the middle is composed of one or more, preferably two or more, layers in the deep neural network.

[0125] These deep neural networks may include convolutional neural networks (CNNs), vision transformers, recurrent neural networks (RNNs), long short-term memory (LSTM) networks, generative pre-trained transformers (GPTs), autoencoders, generative adversarial networks (GANs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, generative adversarial networks (GANs), transformers, etc.

[0126] Alternatively, depending on the embodiment, the deep neural network may be a model trained using transfer learning. Here, transfer learning refers to a learning method in which a large amount of unlabeled training data is pre-trained using a semi-supervised learning or self-learning method to obtain a pre-trained model (or base part) having a first task using techniques such as MLM and NSP as shown in FIGS. 11 and 12, and then the pre-trained model is trained using labeled training data using a supervised learning method to fine-tune it to be suitable for a second task, thereby implementing a target model. One of the models trained using this transfer learning method includes, but is not limited to, BERT (Bidirectional Encoder Representations from Transformers).

[0127] The description of the aforementioned deep neural network is merely an example, and the present disclosure is not limited thereto. In the case of the aforementioned convolutional neural network, it is composed of a feature extraction unit (feature learning) that extracts features from an image, and a classification unit (classification) that performs classification using the extracted features. The feature extraction unit may include, but is not limited to, a convolution layer that extracts features from an image using a kernel, a ReLU layer as one of the activation functions, and a pooling layer for reducing the dimensionality of the data. In addition, the classification unit may include, but is not limited to, a flatten layer that lines up the features extracted from the feature extraction unit, and a fully connected layer and a softmax function that actually perform classification.

[0128] Neural networks can learn through at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. Neural network learning can be the process of applying knowledge to the neural network to perform a specific action.

[0129] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, labeled data is used for each training data, while unsupervised learning uses unlabeled data. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate can vary depending on the number of iterations in the neural network's training cycle. Additionally, to prevent overfitting, methods such as increasing the learning data, regularization, dropout that disables some nodes, and batch normalization layers can be applied.

[0130] Meanwhile, the model disclosed in one embodiment may borrow at least a portion of a transformer. The transformer may be composed of an encoder that encodes embedded data and a decoder that decodes the encoded data. The transformer may have a structure that receives a series of data, performs encoding and decoding steps, and outputs a series of data of different types. In one embodiment, the series of data may be processed into a form operable by the transformer. The process of processing the series of data into a form operable by the transformer may include an embedding process. Expressions such as data tokens, embedding vectors, and embedding tokens may refer to data embedded in a form operable by the transformer.

[0131] To encode and decode a series of data, a transformer can utilize an attention algorithm to process the encoders and decoders within the transformer. An attention algorithm can refer to an algorithm that, for a given query, calculates the similarity for one or more keys, reflects this similarity in the values ​​corresponding to each key, and then weights and sums the values ​​to which the similarity is reflected to calculate an attention value.

[0132] Depending on how the query, key, and value are configured, various types of attention algorithms can be categorized. For example, if attention is obtained by setting the query, key, and value all to the same value, this could be a self-attention algorithm. If attention is obtained by reducing the dimensionality of the embedding vector to process a series of input data in parallel and then generating individual attention heads for each segmented embedding vector, this could be a multi-head attention algorithm.

[0133] In one embodiment, the transformer may be composed of modules that perform multiple multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer may also include additional components other than attention algorithms, such as embedding, normalization, and softmax. Methods for constructing a transformer using attention algorithms may include methods disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0134] A transformer can be applied to various data domains, such as embedded natural language, segmented image data, and audio waveforms, to transform a series of input data into a series of output data. To transform data with various data domains into a series of data that can be input to a transformer, the transformer can embed the data. The transformer can process additional data that expresses the relative positional relationship or phase relationship between the series of input data. Alternatively, vectors expressing the relative positional relationship or phase relationship between the input data can be additionally reflected in the series of input data to embed the series of input data. In one example, the relative positional relationship between the series of input data may include, but is not limited to, word order within a natural language sentence, the relative positional relationship between each segmented image, and the time order of segmented audio waveforms. The process of adding information expressing the relative positional relationship or phase relationship between the series of input data may be referred to as positional encoding.

[0135] The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should reference additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to send and receive during communication.

[0136] The above storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers that the computer can access or in various recording media on the user's computer. In addition, the medium can be distributed across network-connected computer systems, so that computer-readable code can be stored in a distributed manner.

[0137] The steps of a method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0138] While the embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present disclosure can be implemented in other specific forms without altering the technical spirit or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. Memory for storing data to make the creation model lightweight based on artificial intelligence. Includes a processor that performs operations related to the lightweighting of the above-mentioned generation model, The above processor, An artificial intelligence-based generative model lightweight device characterized in that it assigns a randomly initialized score(s) to each weight for a dense network based on an edge pop-up algorithm, finds a random sub-network, sorts the assigned scores in each forward path, and updates the scores using backpropagation while leaving the weights with the preset top k% scores.

2. In paragraph 1, The above processor, An artificial intelligence-based generative model lightweight device characterized in that it leaves the weights having the preset top k% scores, makes other weights 0, calculates the loss of the sub-network in the reverse path, and utilizes the backpropagation.

3. In paragraph 1, The above processor, An artificial intelligence-based lightweight generation model device characterized in that, when calculating the loss of the above sub-network, the image generated through the above sub-network and the actual image are transmitted to an embedding case to calculate the maximum mean discrepancy (MMD) score.

4. In paragraph 3, The above processor, An artificial intelligence-based lightweight generative model device characterized in that it produces the maximum average mismatch score by matching all orders of moments for real and fake samples as two sample sets.

5. In paragraph 4, The above maximum average discrepancy score is, It is characterized by being calculated based on the following <Mathematical Formula 1>. A lightweight device for artificial intelligence-based generative models. [Mathematical Formula 1] Here, ri represents a real sample and fi represents a fake sample.

6. In paragraph 4, The above processor, An artificial intelligence-based generative model lightweight device characterized by using a pre-trained VGG network as a kernel for the above moment matching.

7. In paragraph 6, The above processor, An artificial intelligence-based generative model lightweight device characterized in that it finds SLTs (Strong Lottery Tickets) by repeatedly performing an operation of updating the above maximum average discrepancy score.

8. In paragraph 1, electrically connected to the above processor, An artificial intelligence-based generation model lightweight device further comprising a communication unit that performs communication with an external device that provides data for lightweighting the above generation model.

9. In a method for lightweighting an artificial intelligence-based generative model performed by a device, A step of assigning a randomly initialized score(s) to each weight for the dense network based on the edge pop-up algorithm; Steps to find an arbitrary subnetwork; a step of sorting the assigned scores in each forward path; and A method characterized by including a step of updating a score using backpropagation while leaving weights having a preset top k% score.

10. In paragraph 9, The above updating steps are: A method characterized in that the weights having the preset top k% scores are left, other weights are made 0, the loss of the sub-network is calculated in the reverse path, and the backpropagation is utilized.

11. In paragraph 9, The above updating steps are: A method characterized in that, when calculating the loss of the above sub-network, the image generated through the above sub-network and the actual image are transmitted to an embedding case to calculate the maximum mean discrepancy (MMD) score.

12. In paragraph 11, The above updating steps are: A method characterized in that the maximum average disparity score is calculated by matching all orders of moments for real and fake samples as two sets of samples.

13. In paragraph 12, The above maximum average discrepancy score is, A method characterized in that it is calculated based on the following <Mathematical Formula 1>. [Mathematical Formula 1] Here, ri represents a real sample and fi represents a fake sample.

14. In paragraph 12, The above updating steps are: A method characterized in that a pre-trained VGG network is used as a kernel for the above moment matching.

15. In paragraph 14, The above updating steps are: A method characterized in that it finds SLTs (Strong Lottery Tickets) by repeatedly performing an operation of updating the above maximum average discrepancy score.

Images