Method for compressing large language model and electronic device for executing same

Abstract

A method of compressing a large language model and an electronic device for performing the same are provided. A method of operating an electronic device includes receiving data through a neural network model including at least one transformer block and at least one operation block, and outputting an inference result for the data in the neural network model. The at least one operation block is configured to receive as an input a flag generated based on the data, and selectively perform an operation on the at least one flag based on an index of the flag.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2024-0146832, filed on October 24, 2024, and Korean Patent Application No. 10-2025-0051613, filed on April 21, 2025, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The methods and apparatus according to the embodiments relate to a method for compressing large language models and an electronic device for performing the method. Background Technology

[0004] Large Language Models (LLMs) are deep learning-based neural network models trained on massive datasets. LLMs are specifically designed for understanding and generating text data. They have revolutionized the field of natural language processing and are one of the key technologies enabling computers to understand and process human language. Representative LLMs include Generative Pre-trained Transformers (GPTs) and Transformer-based Bidirectional Encoder Representations (BERTs). Summary of the Invention

[0005] One or more embodiments may at least solve the above-described problems and / or disadvantages, as well as other disadvantages not described above. Furthermore, embodiments are not required to overcome the disadvantages described above, and embodiments may not overcome any of the problems described above.

[0006] According to one aspect of this disclosure, a method of operating an electronic device includes: receiving data through a neural network model including at least one converter block and at least one arithmetic block, and outputting inference results in the neural network model for the data, wherein the at least one arithmetic block is configured to receive tags generated based on the data as input, and selectively perform operations on the at least one tag based on the index of the tag.

[0007] At least one operation block can be configured to perform operations on the initial tag corresponding to the first tag based on the tag index.

[0008] At least one operation block can be configured to be indexed based on the tag, outputting the remaining tags as is, except for the initial tag corresponding to the first tag.

[0009] Operations can include arithmetic operations.

[0010] The operation can be an addition operation, which adds a predetermined value to an initial tag corresponding to the first tag based on the tag index.

[0011] The predetermined value can be determined based on one or more deviations, which represent the difference between the output and input of one or more target converter blocks replaced by at least one operational block.

[0012] According to one aspect of this disclosure, a method of operating an electronic device includes: determining a deviation between the output and input of each of a plurality of converter blocks representing a first neural network model; determining a target converter block among the plurality of converter blocks based on a target number of converter blocks related to the degree of compression of the first neural network model; and obtaining a second neural network model by performing compression on the first neural network model based on the deviation.

[0013] The second neural network model may include a first operational block that replaces the target transformer block with compression, and the first operational block may be configured to perform operations based on the deviation of the target transformer block corresponding to the first operational block.

[0014] The first operation block can be configured to perform an addition operation on the initial marker in the input of the first operation block based on the deviation of the target transformer block replaced by the first operation block, and to pass the result of the addition operation to the next layer.

[0015] Based on at least two target transformer blocks determined by the number of targets as consecutive target transformer blocks, the second neural network model may include a second operational block in which the consecutive target transformer blocks performing the operation are replaced by merging the deviations of the consecutive target transformer blocks.

[0016] Obtaining a second neural network model may include generating the second neural network model by performing structured pruning on at least one target transformer block in the first neural network model, which is determined by the number of targets.

[0017] Obtaining a second neural network model may include generating the second neural network model by performing unstructured pruning on at least one target transformer block in the first neural network model, which is determined by the number of targets.

[0018] The method may further include: determining whether the number of currently determined target transformer blocks has reached the target number; and based on the fact that the number of currently determined target transformer blocks has not reached the target number, determining additional target transformer blocks based on the result of sequentially transforming any one of the remaining transformer blocks other than the currently determined target transformer blocks into a first operation block that performs addition operations only on the initial mark.

[0019] Determining the target transformer block may include: determining whether the number of currently determined target transformer blocks has reached the target number; and when the number of currently determined target transformer blocks has not reached the target number, additionally determining the target transformer block based on the result of sequentially transforming any one of the remaining transformer blocks other than the currently determined target transformer block into a first operation block that performs addition operations only on the initial mark.

[0020] Determining the target transformer block may additionally include identifying the remaining transformer block that has the least impact on the output of the first neural network model when transformed into the first operational block as the target transformer block.

[0021] According to one aspect of this disclosure, an electronic device includes: a memory configured to store instructions; and at least one processor configured to execute the instructions. When the instructions are executed individually and / or jointly by the at least one processor, the electronic device receives data through a neural network model including at least one converter block and at least one arithmetic block, and outputs inference results for the data in the neural network model. The at least one arithmetic module is configured to receive tags generated based on the data as input and selectively perform operations on at least one tag based on the index of the tag.

[0022] At least one operation block can be configured to perform operations on the initial tag corresponding to the first tag based on the tag index.

[0023] At least one operation block can be configured to output the remaining tags as is, except for the initial tag corresponding to the first tag, based on the tag index.

[0024] Operations can include arithmetic operations.

[0025] The operation can be an addition operation, which adds a predetermined value to an initial tag corresponding to the first tag based on the tag index.

[0026] Additional aspects of the embodiments will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0027] The above and / or other aspects will become clearer from the description of certain embodiments with reference to the accompanying drawings, in which:

[0028] Figure 1 This is a diagram illustrating an electronic device according to an embodiment;

[0029] Figure 2 This is a diagram illustrating the architecture of a neural network model according to an embodiment;

[0030] Figure 3This is a diagram illustrating a compressed deep learning model according to an embodiment;

[0031] Figure 4 and Figure 5 This is a diagram illustrating the phenomenon of attention sinking according to an embodiment;

[0032] Figure 6 This is a flowchart illustrating a method of operating an electronic device according to an embodiment;

[0033] Figures 7 to 9 This is a diagram illustrating a simplified target converter block according to an embodiment;

[0034] Figure 10 and Figure 11 This is a diagram illustrating inference using a second neural network model according to an embodiment; and

[0035] Figure 12 and Figure 13 This is a flowchart illustrating a method of operating an electronic device according to an embodiment. Detailed Implementation

[0036] The detailed structural or functional descriptions below are provided as examples only, and various changes and modifications can be made to the embodiments. Accordingly, the embodiments are not to be construed as limiting to this disclosure, and should be understood to include all changes, equivalents, and substitutions within the concept and technical scope of this disclosure.

[0037] Although terms such as first and second are used to describe various components, components are not limited to these terms. These terms should only be used to distinguish one component from another. For example, the first component can be called the second component, or similarly, the second component can be called the first component.

[0038] It should be noted that if the first component is described as "connected", "coupled", or "joined" to the second component, the third component may be "connected", "coupled", or "joined" between the first and second components, although the first component may be directly connected, coupled, or joined to the second component.

[0039] The singular forms “a” and “the” are intended to also include the plural forms unless the context explicitly indicates otherwise. It will also be understood that the terms “comprising” and / or “including” as used herein specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0040] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Unless otherwise defined herein, terms (e.g., those defined in common dictionaries) shall be interpreted as having a meaning that matches the contextual meaning in the relevant field, and shall not be interpreted as having an idealized or overly formalized meaning.

[0041] In the following description, embodiments will be illustrated with reference to the accompanying drawings. When describing embodiments with reference to the accompanying drawings, similar reference numerals refer to similar elements, and redundant descriptions associated with them will be omitted.

[0042] Figure 1 This is a diagram illustrating an electronic device according to an embodiment.

[0043] Reference Figure 1 Electronic device 100 may include a host processor 110, memory 120, and accelerator 130. The host processor 110, memory 120, and accelerator 130 can communicate with each other via buses, on-chip networks (NoC), peripheral component interconnects (PCIe), etc. Figure 1 In the examples, only components relevant to the embodiments described herein are shown as included in the electronic device 100. Therefore, in addition to Figure 1 In addition to the components shown, the electronic device 100 may also include other general-purpose components.

[0044] The host processor 110 can perform overall functions for controlling the electronic device 100. The host processor 110 can control the electronic device 100 overall by executing programs and / or instructions stored in the memory 120. The host processor 110 can be implemented as a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), etc., included in the electronic device 100, but its embodiments are not limited thereto.

[0045] The memory 120 can be hardware used to store data processed and data to be processed in the electronic device 100. Furthermore, the memory 120 can store applications, drivers, etc., to be driven by the electronic device 100. The memory 120 may include volatile memory (e.g., dynamic random access memory (DRAM)) and / or non-volatile memory.

[0046] Electronic device 100 may include an accelerator 130 for computation. Accelerator 130 can handle tasks that, due to the characteristics of the task, would be handled more efficiently by a separate dedicated processor (i.e., accelerator 130) rather than a general-purpose host processor 110. In an embodiment, a large language model (LLM) may be executed in accelerator 130. In this case, one or more processing elements (PEs) included in accelerator 130 may be used. Accelerator 130 may correspond to, for example, a neural processing unit (NPU), tensor processing unit (TPU), digital signal processor (DSP), GPU, neural engine, etc., that performs computations according to a neural network.

[0047] Language models (LMs) can include LLMs as neural network models. An LLM is a neural network model that is a deep learning-based model trained on massive amounts of data. LLMs are specifically designed for understanding and generating text data. To improve performance, LLMs need to include more parameters and can therefore be very large. For example, an LLM can include billions to hundreds of billions of parameters. Due to the large size of LLMs, the hardware resources of the electronic device 100 used to execute the LLM become critical when executing it. For example, the execution of an LLM (e.g., inference) may require hundreds of gigabytes of random access memory (RAM). Consequently, it is difficult to execute LLMs on electronic devices without sufficient hardware resources. Therefore, there are increasing attempts to compress LLMs while minimizing the degradation of inference performance, enabling LLM execution even on electronic devices with limited hardware resources.

[0048] The following will be from Figure 4 This paper begins by describing a method for compressing neural network models. The neural network model can be an LM or an LLM. LM and LLM differ only in scale, and it will be apparent to those skilled in the art that this method for compressing neural networks is equally applicable to both LLM and LM.

[0049] The architecture of a typical LLM will be described below.

[0050] Figure 2 This is a diagram illustrating the architecture of a neural network model according to an embodiment.

[0051] Referring to FIG2, for ease of description, only a transformer block performing decoding is shown. However, this is merely an example to aid in understanding the transformer block and should not be construed as limiting or restricting the scope of other embodiments. For example, even if the neural network model 200 (e.g., LM or LLM) includes a transformer block performing encoding only, the description of this disclosure can be applied in the same manner. For example, even if the neural network model 200 includes both a transformer block performing encoding and a transformer block performing decoding, the description of this disclosure can be applied in the same manner.

[0052] In neural network model 200, input embedding can represent the operation of converting tokens (e.g., words) into vector form in a way that neural network model 200 can understand. To enable the transformer block to process order information, positional information of the tokens (e.g., relative order information) may be required. In neural network model 200, positional embedding can be the operation of adding positional information corresponding to words (or tokens) to a vector. Neural network model 200 can be an operation that trains the input order of words using positional embeddings.

[0053] The neural network model 200 may include multiple transformer blocks 210, 220, and 230. A transformer block may be referred to as a transformer neural network. Transformer block 210 may receive tags that have undergone input embedding and position embedding. Multiple transformer blocks 210, 220, and 230 may be connected in series. For example, a transformer block may receive the output of a prior transformer as input.

[0054] Each of the multiple converter blocks 210, 220, and 230 may include multiple layers. A block that includes attention layer 223 and feedforward layer 224 may be referred to as a converter block. However, this is merely an example to aid in understanding converter blocks and should not be considered as limiting or constraining the scope of other embodiments. For example, converter block 220 may also include normalization layer 221 and linearization layer 222.

[0055] Normalization layer 221 can stabilize training by normalizing the output of preceding layers. Linear layer 222 can perform linear transformations on the input and / or output of the attention layer. For example, linear layer 222 can be used to reconstruct the output of attention layer 223 and match dimensions. Attention layer 223 can use the attention mechanism to identify relationships between input labels. Feedforward layer 224 can perform additional nonlinear transformations after the attention mechanism terminates.

[0056] However, the structure of the converter block described above is merely an example, and this disclosure is not limited thereto. For example, the converter block may include... Figure 2 The diagram shows more or fewer layers, and one or more transformer blocks can be defined as high-level transformer blocks. For example, according to an embodiment, Figure 2 Two or more consecutive transformer blocks shown can be defined as a high-level transformer block.

[0057] In neural network model 200, the output layer (e.g., Figure 2 The prediction can be generated based on the output of the last transformer block.

[0058] The performance of neural network model 200 can be related to the number of transformer blocks. As the number of transformer blocks increases, neural network model 200 can learn more deeply. For example, as the number of transformer blocks increases, neural network model 200 can learn more complex contextual information and learn the correlations between distant tokens in long sentences. As the number of transformer blocks increases, the size of neural network model 200 can also increase. To execute a neural network model 200 with increased scale, more hardware resources in electronic devices may be required. Increased hardware resources can refer to increased costs. Therefore, it may be necessary to compress large-scale artificial intelligence models (such as neural network model 200).

[0059] The following sections will describe existing techniques for compressing artificial intelligence models (e.g., neural network model 200).

[0060] Figure 3 This is a diagram illustrating a compressed deep learning model according to an embodiment.

[0061] Reference Figure 3 The diagram illustrates a neural network model 300 (e.g., LM or LLM) according to an embodiment. The neural network model 300 may include multiple layers (e.g., L0, L1, L2, and L3). Each layer may include multiple nodes (e.g., neurons). Nodes may connect to nodes in the next layer. Nodes may be connected to connection nodes in the next layer via weights. Values ​​input to a node may be multiplied by weights and passed to the next node.

[0062] Pruning can be performed to compress the neural network model 300. Pruning can be a process of removing unnecessary weights (e.g., parameters) from the neural network model 300. Unnecessary weights can be removed from the neural network model 300 through pruning. For example, a compressed neural network model 310 can be obtained after removing unnecessary weights from the neural network model 300.

[0063] Pruning is a widely used technique to improve the efficiency and performance of neural network model 300. By pruning, the size of neural network model 300 can be reduced and the running speed of the neural network can be increased.

[0064] Pruning can include unstructured pruning and structured pruning.

[0065] Unstructured pruning can be a method of removing individual weights from a neural network model 300. For example, in unstructured pruning, unnecessary or unimportant weights in the neural network model 300 can be selectively removed by removing weights with values ​​less than a threshold. A drawback of unstructured pruning is that it introduces complex data access patterns because it removes individual weights one by one. Furthermore, to improve operational speed through unstructured pruning, a very high pruning percentage (e.g., 90%) may be required, which may not be suitable for compressing the neural network model 300.

[0066] Structured pruning can be a method of directly removing entire layers or channels from a neural network model 300. For example, structured pruning can remove weights in a specific pattern (e.g., a 2:4 pattern). When removing weights in a specific pattern, there may be a drawback: the operation speed only increases when the batch size is very large (e.g., 128) or larger. For example, structured pruning can completely remove specific layers or channels (e.g., transformer blocks). Removing specific layers or channels can simplify the complexity of the neural network model 300 and save operational resources. However, when completely removing specific layers or channels, the compressed neural network model 310 may not be able to perform any important functions performed by the removed layers or channels, resulting in performance degradation.

[0067] Therefore, in addition to the pruning methods mentioned above, it may be necessary to compress the neural network model 300.

[0068] Figure 4 and Figure 5 This is a diagram illustrating the phenomenon of attention sinking according to an embodiment.

[0069] Reference Figure 4 The diagram illustrates a neural network model 400 (e.g., LM or LLM) comprising multiple transformer blocks 401, 402, 403, and 420. In this disclosure, for ease of description, components other than the transformer blocks are omitted from the architecture of the neural network model 400. Therefore, those skilled in the art will understand that the neural network model 400 may also include components other than the transformer blocks. Figure 4 Components other than those shown.

[0070] Transformer block 401 can be initially set in neural network model 400. Transformer block 401 can receive tags. When a sentence is input, neural network model 400 can segment the sentence into words, parts of words, or characters as tags. For example, when the sentence "I have a meeting today" is received, neural network model 400 can segment the sentence into "I", "today", "have", "a meeting", and "." as tags.

[0071] These tags can be input into transformer block 401. Among the tags input into transformer block 401, the earliest tag can be called the initial tag. The first tag among the tags input into transformer block 401 can be the initial tag. The first tag can be the initial tag based on the index of the tags input into transformer block 401. For example, when the tag index starts from "0", the tag with index 0 can be the initial tag. For example, in the sentence above, "I" can be the initial tag.

[0072] In the tags input to subsequent transformer blocks 402, 403, and 420, other than transformer block 401 which is the first block set in the neural network model 400, the tag corresponding to the first tag based on the tag index can be the initial tag. For example, in the tags input to transformer block 402, tag 412 can be the tag corresponding to the first tag (i.e., the initial tag).

[0073] For each transformer block, the position of the initial marker can be the same. The position of the initial marker can be the same as the first marker among the markers input to each transformer block. For example, the initial marker for transformer block 401 can be "I", the initial marker for transformer block 402 can be marker 412, and the initial marker for transformer block 403 can be marker 413. The position of the initial marker can be the same as the position of the first marker. The first marker can refer to the first one based on the marker index.

[0074] Initial labels can significantly impact the performance of neural network model 400. This phenomenon, where initial labels significantly affect the performance of neural network model 400, can be termed "attention sink" (paper "Efficient Streaming Language Model with Attention Sink (ICLR 2024)"). Attention sink refers to the phenomenon where neural network model 400 pays intense attention to the initial labels of the input. After several initial transformer blocks, neural network model 400 may become highly attentive to the initial labels. Because neural network model 400 pays more attention to the initial labels, the updated values ​​of the initial labels after passing through transformer blocks may have almost fixed values, regardless of the input. For example, regardless of the sentence input to neural network model 400, the updated values ​​of the initial labels after passing through transformer blocks may remain almost unchanged after several initial transformer blocks.

[0075] When the input h of the k-th transformer block k With the input h of the (k+1)th transformer block k+1 The change between (or the output of the k-th transformer block) is hour, This can refer to the updated value in the k-th transformer block. For example, regardless of the sentence input to neural network model 400, The initial mark can be almost constant.

[0076] Reference Figure 5 Table 500 shows the results of experiments conducted on Llama2-7B (as a neural network model, such as LM or LLM) using data with a batch size of 100. Table 500 can show the similarity between the following values: the value updated in the third transformer block for the zeroth label when the first batch is input, and the value updated in the third transformer block for the zeroth label when the second batch is input. Table 500 can also show the similarity between the following values: the value updated in the third transformer block for the fourth label when the first batch is input, and the value updated in the third transformer block for the fourth label when the second batch is input. The label index can start from 0. The zeroth label in Table 500 can correspond to the initial label.

[0077] Cosine similarity is a method of measuring similarity using the angle between two vectors. In cosine similarity, when two vectors point in roughly the same direction, the cosine similarity can become closer to 1, and when two vectors point in roughly opposite directions, the cosine similarity can become closer to -1. For example, a cosine similarity closer to 1 indicates similarity, while a cosine similarity closer to -1 indicates dissimilarity.

[0078] The method of using Euclidean distance can be a way of measuring similarity by using the straight-line distance between two points (or two vectors). The closer the Euclidean distance is to 0, the more similar the two points (or vectors) are.

[0079] Referring to Table 500, the cosine similarity of the zeroth marker can be 0.99, close to 1, and the Euclidean distance can be 0.06, close to 0. For example, this can indicate that the value updated by the third transformer block for the zeroth marker when the first batch of input is similar to the value updated by the third transformer block for the zeroth marker when the second batch of input is input.

[0080] Referring to Table 500, the cosine similarity of the fourth label can be 0.18, which is not close to 1, and the Euclidean distance can be 2.51, which is not close to 0. For example, this can indicate that the value updated by the third transformer block for the fourth label when the first batch of input is not similar to the value updated by the third transformer block for the fourth label when the second batch of input is input.

[0081] In summary, it can be observed that, based on the "attention sinking" phenomenon, the neural network model pays close attention to the initial label, and the transformer block updates similar values ​​for the initial label regardless of the input. The following section describes a method for simplifying (e.g., compressing) the neural network model by updating only the initial label that has a significant impact on performance, based on the aforementioned characteristics.

[0082] Figure 6 This is a flowchart illustrating a method of operating an electronic device according to an embodiment.

[0083] In the following embodiments, the operations may be performed sequentially, but not necessarily sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Figure 6 The operations shown can be performed by at least one component of the electronic device. For example, the electronic device may include a memory storing instructions. When the instructions are executed individually and / or jointly by at least one processor, the electronic device can perform the following operations.

[0084] In operation 610, the electronic device can determine the number (N) of converter blocks to be simplified. Simplification may refer to the compression of a model (e.g., an LM, LLM, or neural network model). Simplification may refer to model compression, i.e., simplifying the operation of a model (e.g., an LM, LLM, or neural network model).

[0085] Electronic devices can determine the number of transformer blocks to be transformed into computational blocks for a neural network model. The number of transformer blocks to be transformed into computational blocks can be called the target number. This target number can be related to the compression level of the first neural network model.

[0086] Here, the neural network model to be compressed can be referred to as the first neural network model (e.g., the first LM or the first LLM), and the model obtained by compressing the first neural network model can be referred to as the second neural network model (e.g., the second LM or the second LLM). The electronic device can determine the number of converter blocks to be converted into operational blocks among the multiple converter blocks included in the first neural network model. For example, the electronic device can determine the number of converter blocks to be converted into operational blocks as N. The electronic device can determine the target converter block among the multiple converter blocks based on the target number.

[0087] The electronic device can determine the target number based on the hardware resources of the device used to execute the second neural network model. The electronic device can determine the target number based on at least one of the throughput and memory constraints of the device used to execute the second neural network model. Throughput can refer to the amount of work (e.g., the number of data samples) that the device used to execute the second neural network model can process in a given time (e.g., x seconds). Memory constraints can refer to the size of the storage space and / or RAM of the device used to execute the second neural network model.

[0088] For example, when the size of the RAM used to execute the first neural network model is 256 gigabytes (GB) and the size of the RAM used by the electronic device to execute the second neural network model is 128 GB, the electronic device can determine the target number based on memory constraints.

[0089] According to the embodiment, operation 610 can be omitted. The target number can be predetermined, and when the target number is predetermined, operation 610 can be omitted. For example, the target number can be predetermined by the user who wishes to compress the first neural network model.

[0090] In operation 620, the electronic device can determine the deviation of each converter block by initializing the first neural network model.

[0091] Initialization can be an operation that determines the bias of each converter block. Electronic devices can determine the bias that indicates the difference between the output and input of each of the multiple converter blocks included in a first neural network model.

[0092] The electronic device can input a sentence beginning (BOS) tag to the first neural network model. The BOS tag can be a marker that informs the first neural network model of the start of a sentence. The electronic device can initialize the first neural network model by inputting the BOS tag. The electronic device can determine the bias of each transformer block based on the initialized first neural network model.

[0093] The deviation of each converter block can be the change between the initial mark input to the converter block and the output of the converter block corresponding to the initial mark (e.g., Figure 4 of ).

[0094] The deviation of each converter block can be determined by the difference between the initial mark input to the converter block and the output of the converter block corresponding to the initial mark. For example, refer to Figure 4 The deviation of the second converter block 402 can be the difference between the initial mark (e.g., mark 412) input to the second converter block 402 and the output (e.g., mark 413) output by the second converter block 402 corresponding to the initial mark (e.g., mark 412).

[0095] In operation 630, the electronic device can determine whether the number of target converter blocks has reached N.

[0096] The target transformer block can indicate which of the multiple transformer blocks included in the first neural network model is determined to be a compressed transformer block. The electronic device can determine whether the number of currently identified target transformer blocks has reached the target number.

[0097] When the number of target converter blocks is currently determined to be m, the electronic device can determine whether m=N.

[0098] When the number of target converter blocks has been reached, the electronic device can terminate compression. When the number of target converter blocks has not been reached, the electronic device can execute operation 640.

[0099] The electronic device can be operated repeatedly for 640 to 660 times until the number of currently determined target converter blocks reaches the target number.

[0100] In operation 640, the electronic device can determine the target converter block.

[0101] The following describes the case where the number of target transformer blocks currently determined (e.g., m) is 1 or more.

[0102] In operation 640, the currently determined target transformer block can be in a compressed state according to operation 650. According to operation 650, the currently determined target transformer block can be transformed into an operation block. This operation block can use the deviation of the target transformer block transformed into an operation block to perform operations only on the initial mark.

[0103] For example, assuming the number of target transformer blocks currently determined is m, the intermediate neural network model can include m operational blocks (e.g., m is a natural number greater than or equal to 1). The intermediate neural network model is a neural network model generated during the process of compressing the first neural network model into the second neural network model, and at least one target transformer block currently determined can be transformed into an operational block.

[0104] For example, suppose the total number of transformer blocks in the first neural network model is M, and the number of target transformer blocks currently determined is m. The intermediate neural network may include m operational blocks and (Mm) remaining transformer blocks. M is a natural number greater than m.

[0105] In operation 640, the electronic device may additionally determine the target converter block. The electronic device may sequentially compress the remaining converter blocks included in the intermediate neural network model one by one to generate multiple compressed intermediate neural network models. The remaining converter blocks may be converter blocks included in the intermediate neural network and may refer to candidates that can be determined as the target converter block.

[0106] An electronic device can sequentially compress the residual transformer blocks of an intermediate neural network model one by one to generate multiple compressed intermediate neural network models, wherein a different residual transformer block is transformed into an operational block.

[0107] For example, suppose an intermediate neural network comprises m operational blocks and (Mm) residual transformer blocks, where M is a natural number greater than m. The residual transformer blocks can be sequentially compressed one by one to generate (Mm) compressed intermediate neural network models. For example, a first compressed intermediate neural network model could be a first residual transformer block that has been transformed into operational blocks, and a second compressed intermediate neural network model could be a second residual transformer block that has been transformed into operational blocks.

[0108] Electronic devices can determine the target converter block based on multiple compressed intermediate neural network models generated by sequentially transforming the remaining converter blocks one by one into arithmetic blocks. Electronic devices can further determine the target converter block based on the performance of the multiple compressed intermediate neural network models. Electronic devices can identify the remaining converter block that has the least impact on the output (e.g., performance or accuracy) even if it is transformed into an arithmetic block as the target converter block.

[0109] Electronic devices can compare the performance of an intermediate neural network model with that of multiple compressed intermediate neural network models to determine the compressed intermediate neural network model with the smallest performance variation. The remaining transformer block (with the smallest performance variation) in the compressed intermediate neural network model after transformation into computational blocks can be identified as the target transformer block.

[0110] For example, suppose that the compressed intermediate neural network model generated by the second residual converter block, which is transformed into an operational block within the residual converter block, exhibits the least performance variation. The electronic device can then identify the second residual converter block within the residual converter block as the target converter block.

[0111] The following section will describe the case where at least one target converter block has not yet been identified.

[0112] According to the embodiment, there may not be at least one target converter block that is currently determined. For example, when the number of target converter blocks currently determined is m, m can be 0.

[0113] The electronic device can determine the target transformer block from the remaining transformer blocks besides the currently determined target transformer block. Since the number of currently determined target transformer blocks is 0, the remaining transformer blocks can be multiple transformer blocks included in the first neural network. Since the number of currently determined target transformer blocks is 0, the electronic device can sequentially compress multiple transformer blocks included in the first neural network model one by one to generate multiple compressed first neural network models. The electronic device can sequentially compress multiple transformer blocks included in the first neural network model one by one to generate multiple compressed first neural network models, wherein a different transformer block is transformed into a computation block.

[0114] For example, suppose the total number of transformer blocks in the first neural network model is M. M is a natural number greater than or equal to 1. An electronic device can sequentially compress multiple transformer blocks one by one to generate M compressed first neural network models.

[0115] An electronic device can determine a target converter block based on multiple compressed first neural network models generated by sequentially transforming multiple converter blocks into arithmetic blocks one by one. The electronic device can also determine the target converter block based on the performance of the multiple compressed first neural network models. Furthermore, the electronic device can identify the converter block that has the least impact on the output (e.g., performance or accuracy) even when transformed into an arithmetic block as the target converter block.

[0116] An electronic device can compare the performance of a first neural network model with that of multiple compressed first neural network models to determine the compressed first neural network model with the smallest performance variation. The transformer block in the compressed first neural network model that, when transformed into computational blocks, exhibits the smallest performance variation can be identified as the target transformer block.

[0117] For example, suppose that the compressed first neural network model generated by the second converter block, which is transformed into an operational block, has the least performance change. An electronic device can identify the second converter block among multiple converter blocks as the target converter block.

[0118] In operation 650, the electronic devices can simplify the target converter block.

[0119] The electronic device can compress the target converter block determined in operation 640. The electronic device can transform the target converter block into a computation block through compression. The electronic device can replace the target converter block with a computation block through compression.

[0120] Compression may include transforming a converter block into an arithmetic block. The arithmetic block may perform operations based on the deviation of the transformed converter block, but only against an initial mark. An electronic device may transform a target converter block into an arithmetic block, which performs operations based on the deviation of the target converter block, but only against an initial mark. This operation may include arithmetic operations. This operation may include addition, adding a predetermined value to an initial mark. The predetermined value may be the deviation of the transformed converter block.

[0121] According to embodiments, in addition to transforming into operational blocks, structured pruning and / or unstructured pruning can be performed to identify and simplify the target transformer block. Hereinafter, a method using at least one of structured pruning, unstructured pruning, and compressed transformation into operational blocks will be described.

[0122] Structured pruning can include pruning methods such as removing weights in a specific pattern and completely removing the target transformer block, as referenced above. Figure 3 As mentioned above, unstructured pruning can include methods such as removing individual weights, as described in the reference above. Figure 3 The term "transform into a computational block" can refer to transforming a converter block into a computational block that performs operations only on an initial mark based on the deviation of the corresponding converter block. Electronic devices can determine the target converter block based on a compression method including at least one of structured trimming, unstructured trimming, and transform into a computational block.

[0123] In the following text, it is assumed that the number of target transformer blocks (e.g., m) is 1 or more, and that structured pruning, unstructured pruning, and transformation into operational blocks are used as compression methods. However, it will be apparent to those skilled in the art that the following description also applies when the compression method includes at least one of structured pruning, unstructured pruning, and transformation into operational blocks.

[0124] In operation 640, the currently determined target converter block can be in a compressed state according to operation 650. According to operation 650, the currently determined target converter block can be in a compressed state using the target compression method used to determine the target converter block.

[0125] For example, assuming the number of target transformer blocks currently determined is m, the intermediate neural network model can include m (e.g., m is a natural number greater than or equal to 1) compressed target transformer blocks. Compressed target transformer blocks can be compressed by performing structured pruning, unstructured pruning, or transforming them into operational blocks. Compressed target transformer blocks can be compressed using a target compression method used to determine the target transformer blocks. The intermediate neural network model is a neural network model generated during the process of compressing a first neural network model into a second neural network model, and at least one currently determined target transformer block can be in a state of compression using the target compression method used to determine the target transformer blocks.

[0126] For example, assuming the total number of transformer blocks in the first neural network model is M, then the number of target transformer blocks currently determined is m. The intermediate neural network can include m compressed target transformer blocks and (M - m) remaining transformer blocks. M is a natural number greater than m.

[0127] In operation 640, the electronic device may additionally determine the target transformer block. The electronic device may sequentially compress the remaining transformer blocks included in the intermediate neural network model one by one according to a compression method to generate multiple compressed intermediate neural network models. The remaining transformer blocks may be transformer blocks included in the intermediate neural network, and may refer to candidates that can be determined as the target transformer block. The electronic device may sequentially compress the remaining transformer blocks one by one according to a compression method to generate multiple compressed intermediate neural network models, wherein each remaining transformer block is compressed using a different compression method.

[0128] For example, suppose an intermediate neural network consists of m operational blocks and (Mm) residual transformer blocks, where M is a natural number greater than m. The residual transformer blocks can be compressed sequentially according to a compression method to generate 3 (Mm) compressed intermediate neural network models. Since the compression method includes three approaches (e.g., structured pruning, unstructured pruning, and transformation into operational blocks), 3 (Mm) compressed intermediate neural network models can be generated. For example, the first group of (Mm) compressed intermediate neural network models could be the last residual transformer block that has undergone structured pruning. For example, the second group of (Mm) compressed intermediate neural network models could be the last residual transformer block that has undergone unstructured pruning. For example, the third group of (Mm) compressed intermediate neural network models could be the last residual transformer block that has been transformed into operational blocks.

[0129] Electronic devices can determine a target converter block based on multiple compression intermediate neural network models generated by sequentially compressing the remaining converter blocks one by one according to a compression method. Electronic devices can also determine the target converter block based on the performance of the multiple compression intermediate neural network models. Finally, electronic devices can identify the target compression method and the target converter block that have the least impact on the output (e.g., performance or accuracy).

[0130] Electronic devices can compare the performance of an intermediate neural network model with that of multiple compressed intermediate neural network models to determine the compressed intermediate neural network model with the smallest performance variation. The compressed remaining transformer blocks in the compressed intermediate neural network model with the smallest performance variation can be identified as the target transformer blocks, and the method used to compress the remaining transformer blocks can be identified as the target compression method.

[0131] For example, suppose that the performance change of the compressed intermediate neural network model generated by unstructured pruning of the second residual converter block among the residual converter blocks is minimal. An electronic device can identify the second residual converter block among the residual converter blocks as the target converter block and determine unstructured pruning as the target compression method for that target converter block.

[0132] In the following section, a scenario where at least one target transformer block has not yet been determined will be described. It is assumed that structured pruning, unstructured pruning, and transformation into operational blocks are used as compression methods. However, it will be apparent to those skilled in the art that the following description also applies when the compression method includes at least one of structured pruning, unstructured pruning, and transformation into operational blocks.

[0133] According to the embodiment, there may not be at least one target converter block that is currently determined. For example, when the number of target converter blocks currently determined is m, m can be 0.

[0134] Since the number of target converter blocks currently determined is 0, the electronic device can sequentially compress multiple converter blocks included in the first neural network model one by one according to the compression method to generate multiple compressed first neural network models. The electronic device can sequentially compress multiple converter blocks included in the first neural network model one by one according to the compression method to generate multiple compressed first neural network models, wherein the converter blocks are compressed using different compression methods.

[0135] For example, suppose the total number of transformer blocks in the first neural network model is M. M is a natural number greater than or equal to 1. An electronic device can sequentially compress multiple transformer blocks one by one according to a compression method to generate 3M compressed first neural network models. Since the compression method includes three approaches (e.g., structured pruning, unstructured pruning, and transformation into operational blocks), 3M compressed first neural network models can be generated. For example, the first group of M compressed first neural network models could be the last transformer block among the transformer blocks that has undergone structured pruning. For example, the second group of M compressed first neural network models could be the last transformer block among the transformer blocks that has undergone unstructured pruning. For example, the third group of M compressed first neural network models could be the last transformer block among the transformer blocks that has been transformed into operational blocks.

[0136] An electronic device can determine a target converter block based on multiple compressed first neural network models generated by sequentially compressing multiple converter blocks according to a compression method. The electronic device can also determine the target converter block based on the performance of the multiple compressed first neural network models. The electronic device can identify the converter block that has the least impact on the output (e.g., performance or accuracy) even after compression as the target converter block.

[0137] An electronic device can compare the performance of a first neural network model with the performance of multiple compressed first neural network models to determine the compressed first neural network model with the smallest performance variation. The compressed transformer block in the compressed first neural network model with the smallest performance variation can be identified as the target transformer block, and the method of compressing the transformer block can be identified as the target compression method.

[0138] For example, suppose that the performance change of the compressed first neural network model generated by unstructured pruning of the second converter block among multiple converter blocks is minimal. An electronic device can identify the second converter block among multiple converter blocks as the target converter block and determine the unstructured pruning as the target compression method.

[0139] In operation 650, the electronic devices can simplify the target converter block.

[0140] The electronic device can use a target compression method to compress the target converter block determined in operation 640. The target compression method is a compression method used to determine the target converter block and may include structured pruning, unstructured pruning, or transformation into a computational block.

[0141] For example, an electronic device can use a target compression method (structured pruning) to compress a target converter block. Alternatively, an electronic device can use a target compression method (unstructured pruning) to compress a target converter block. Or, an electronic device can compress a target converter block by transforming it into an operational block.

[0142] Please refer to later Figure 11 Describe a second neural network model that uses at least one of structured pruning, unstructured pruning, and transformation into operational blocks as compression methods.

[0143] Figures 7 to 9 This is a diagram illustrating a simplified target converter block according to an embodiment.

[0144] Reference Figure 7 The diagram illustrates only a portion of the multiple transformer blocks included in a first neural network model 700 (e.g., a first LM or a first LLM). For example, only a few transformer blocks are shown. Figure 4 The tenth to fourteenth transformer blocks in the neural network model. However, this is merely an example, and those skilled in the art will understand that the description of this disclosure can also be applied to neural network models with fewer or more than 20 transformer blocks.

[0145] The first neural network model 700 can be in an initialization state. The electronic device can determine the deviation of each converter block through the initialization of the first neural network model 700. For example, the first neural network model can be in a state where it receives BOS tags to determine the deviation of each converter block. The deviation can be the change between the initial tag input to the converter block and the output of the converter block corresponding to the initial tag. For example, b10 is the deviation of the tenth converter block, which can be the change between the initial tag input to the tenth converter block and the output of the tenth converter block corresponding to the initial tag.

[0146] According to an embodiment, the bias can be updated. The electronic device can perform updates (e.g., fine-tuning) on ​​the first neural network model. For example, the electronic device can endow the first neural network model with additional capabilities by adding smaller parameters relative to the size of the first neural network model using methods such as Low-Rank Adaptive (LoRA) and Data-Efficient Low-Rank Adaptive (DoRA). The electronic device can perform updates by fine-tuning the bias and adding parameters.

[0147] Electronic devices can identify target converter blocks. For example, electronic devices can identify converter block 701 as the target converter block. Electronic devices can base their decisions on... Figure 6 Determine the target transformer block. For example, an electronic device may sequentially simplify (e.g., model compression) multiple transformer blocks and determine the transformer block that has the least impact on the output of the first neural network model 700 as the target transformer block.

[0148] The electronic device can perform compression on the converter block 701. For example, the electronic device can transform the converter block 701 into an arithmetic block 711. The arithmetic block 711 can perform operations based on the deviation of the corresponding converter block 701. For example, the arithmetic block 711 can perform arithmetic operations based on the deviation of the corresponding converter block 701 (e.g., b12). The arithmetic operation can be addition.

[0149] Operation block 711 can output the remaining tags, excluding the initial tag corresponding to the first tag, as is based on the tag index. Operation block 711 can also bypass this operation and not perform the operation on the remaining tags, excluding the initial tag corresponding to the first tag, based on the tag index.

[0150] Operation block 711 can perform an operation that adds 0 and the remaining tags except for the initial tag corresponding to the first tag, based on the tag index. Operation block 711 can perform an operation that adds 0 and the remaining tags except for the initial tag corresponding to the first tag, based on the tag index, and outputs the remaining tags as is.

[0151] Operation block 711 can transfer the result of the operation to the next layer. The next layer may include at least one of the following: the next operation block, the next transformer block, and the output layer.

[0152] Reference Figure 7 The diagram illustrates an intermediate neural network model 710 in which transformer block 701 is replaced by operation block 711. In the intermediate neural network model 710, operation block 711 can receive an initial label (e.g., y0). Operation block 711 can perform an addition of the deviation (e.g., b12) corresponding to transformer block 701 and the initial label (e.g., y0). Operation block 711 can perform a pass-through to the next transformer block without performing any operation on labels other than the initial label in the input.

[0153] The electronic equipment can determine whether the target number of converter blocks has been reached. If the target number of converter blocks has not been reached, the electronic equipment can re-determine the target number of converter blocks. The electronic equipment can be based on... Figure 6 The target converter block is determined. For example, the electronic device can determine converter block 713 as the target converter block in the intermediate neural network model 710 where converter block 701 is replaced by operation block 711. The electronic device can then transform converter block 713 into operation block 721.

[0154] Reference Figure 7 The diagram illustrates an intermediate neural network model 720 in which transformer block 713 is transformed into arithmetic block 721. Arithmetic block 721 can perform addition of the deviation (e.g., b14) corresponding to transformer block 713 and the initial label of arithmetic block 721.

[0155] The electronic device can repeat the above operations until the number of target converter blocks reaches the target number. When the number of target converter blocks reaches the target number, the electronic device can obtain a second neural network model that transforms the target converter blocks determined according to the target number into computation blocks. With the compression operation of the target converter blocks, the electronic device can obtain a second neural network model compressed from the first neural network model.

[0156] Reference Figure 8 The diagram illustrates only a portion of the multiple transformer blocks included in the first neural network model 800. For example, only a few are shown. Figure 4 The tenth to fourteenth transformer blocks in the neural network model. However, this is merely an example, and those skilled in the art will understand that the description of this disclosure can also be applied to neural network models with fewer or more than 20 transformer blocks.

[0157] According to an embodiment, at least some of the target converter blocks determined based on the target number can be consecutive. For example, among five target converter blocks, three target converter blocks (e.g., the eleventh, twelfth, and thirteenth converter blocks) can be consecutive. The electronic device can transform consecutive target converter blocks into arithmetic blocks that perform operations by incorporating the deviations of consecutive target converter blocks. The electronic device can replace consecutive target converter blocks with arithmetic blocks that perform addition by incorporating the deviations of consecutive target converter blocks.

[0158] Referring to a second neural network model 810 simplified (e.g., model compression) from the first neural network model 800, the electronic device can transform converter blocks 801, 802, and 803 into an operation block 811 that performs operations based on b11, b12, and b13. The operation block 811 can perform arithmetic operations (e.g., addition) based on b11, b12, and b13 on the initial label in its input. For example, the operation block 811 can perform the addition of b11+b12+b13 on only the initial label in its input. To distinguish between the operation block transformed from a single target converter block and the operation block 811 transformed from consecutive target converter blocks, they can be referred to as the first operation block and the second operation block, respectively.

[0159] Reference Figure 9 The diagram illustrates only a portion of the multiple transformer blocks included in the first neural network model 900. For example, only a few are shown. Figure 4 The tenth to fourteenth transformer blocks in the neural network model. However, this is merely an example, and those skilled in the art will understand that the description of this disclosure can also be applied to neural network models with fewer or more than 20 transformer blocks.

[0160] According to an embodiment, the first neural network model 900 can be compressed into the second neural network model 910 by performing at least one of unstructured pruning, structured pruning, and transformation into an operational block on the target transformer block.

[0161] For example, transformer block 901 is identified as the target transformer block because it has the least impact on the output of the first neural network model 900 when transformed into an operational block. Similarly, transformer block 903 is identified as the target transformer block because it has the least impact on the output of the first neural network model 900 when performing structured pruning (e.g., removing transformer blocks). Finally, transformer block 905 is identified as the target transformer block because it has the least impact on the output of the first neural network model 900 when performing unstructured pruning.

[0162] When a target converter block is determined using two or more compression methods, the electronic device can compress the corresponding target converter block based on the determined compression method.

[0163] For example, an electronic device can be compressed by transforming converter block 901 into operational block 911. An electronic device can be compressed by performing structured pruning on converter block 903. An electronic device can be compressed by performing unstructured pruning on converter block 905.

[0164] Referring to a second neural network model 910, the second neural network model 910 may include an operational block 911 and a transformer block 915 that performs unstructured pruning on it. Inference using the second neural network model 910 (e.g., a second LM or a second LLM) will be described below.

[0165] Figure 10 and Figure 11 This is a diagram illustrating inference using a second neural network model according to an embodiment.

[0166] Reference Figure 10 The diagram illustrates a second neural network model 1000 (e.g., LM or LLM). The second neural network model 1000 can represent, according to the reference above... Figures 6 to 9 The described method is a model that compresses the first neural network model. Figure 10 The second neural network model 1000 in the example is used to describe the reasoning of the compressed neural network model and should not be construed as a limitation on other embodiments.

[0167] The second neural network model 1000 can be obtained or derived from a first neural network model comprising multiple transformer blocks. The second neural network model 1000 can be obtained by transforming at least one target transformer block, determined based on a target number related to the compression degree of the first neural network model, into one or more computational blocks 1015 and 1030. For example, the transformation process to obtain the second neural network model 1000 may include: identifying and compressing certain transformer blocks from the first neural network model based on a target compression ratio or a target number of blocks (defining the required model size or computational efficiency). To this end, at least one target transformer block selected from the first neural network model is replaced or transformed into one or more computational blocks 1015 and 1030. These computational blocks 1015 and 1030 can be lightweight computational modules that approximate or replace the functionality of the original transformer blocks with lower complexity.

[0168] The second neural network model 1000 may include one or more transformer blocks (e.g., unchanged transformer blocks) 1005, 1010, 1020, 1025, 1035, and 1040, and one or more operational blocks (e.g., transformed operational blocks) 1015 and 1030. The one or more transformer blocks 1005, 1010, 1020, 1025, 1035, and 1040 may be transformer blocks included in a first neural network model, which is the model before the second neural network model 1000 is compressed, and will not be transformed into the one or more operational blocks 1015 and 1030. The one or more operational blocks 1015 and 1030 may be blocks replaced by blocks among the plurality of transformer blocks included in the first neural network model that are determined to be the target transformer block.

[0169] One or more operation blocks 1015 and 1030 can perform operations on a predetermined value of an initial marker based on inputs to the operation blocks. For example, one or more operation blocks 1015 and 1030 can perform simplified calculations compared to a converter block. The operation blocks can apply arithmetic operations, such as applying an additive bias to the initial marker. One or more operation blocks 1015 and 1030 can perform addition, adding the predetermined value to the initial marker corresponding to the first marker based on the index of the input marker. One or more operation blocks 1015 and 1030 can perform addition that simply adds the predetermined value to the initial marker. The predetermined value can be determined based on one or more biases replaced by at least one operation block. The biases can be determined based on the difference between the input and output of the target converter block.

[0170] One or more operation blocks 1015 and 1030 may include at least one of the first operation block 1015 and the second operation block 1030.

[0171] The first operation block 1015 may include a predetermined value of the deviation of one of the at least one target converter blocks determined in the first neural network model. For example, the first operation block 1015 may be a block replaced by a third converter block among the at least one target converter blocks determined in the first neural network model, and may include a predetermined value (e.g., b3) based on the deviation of the third converter block. The first operation block 1015 may perform operations such as Equation 1.

[0172] [Formula 1]

[0173] .

[0174] Assume the tag index starts from 0. The initial tag index can be 0. h l+1 This can represent the output of the operation block located in the first position. h l+1It can refer to the input of the (l+1)th block (e.g., an arithmetic block or a transformer block). k can represent the index of the tag. h l [k] can refer to the value of the k-th flag, which is input to the operation block located in the first position. l It is a predetermined value of the operation block located in the first position, and can represent the deviation of replacing the target transformer block (e.g., the transformer block located in the first position in the first neural network model) with the operation block located in the first position.

[0175] When k is 0 (e.g., when it is the initial mark), the operation block performs addition, adding a predetermined value (e.g., b). l The first operation block 1015 can be a compressed third transformer block where l can be 3. The first operation block 1015 can perform addition, adding a predetermined value (e.g., b3) only to the initial mark.

[0176] The second operation block may include predetermined values ​​based on the deviations of two or more consecutive target converter blocks among at least one target converter block determined in the first neural network model. For example, the second operation block 1030 may include predetermined values ​​(e.g., b6 + b7 + b8) based on the deviations of the sixth to eighth converter blocks among at least one target converter block determined in the first neural network model.

[0177] An electronic device can acquire data for inference using a second neural network model 1000. The data may include content such as at least one of images, text, video, audio, and code. For example, the data may include multiple sentences input to the second neural network model 1000 for translation. For example, the data may include images input to the second neural network model 1000 to obtain image descriptions. However, the above data are merely examples for ease of description and should not be construed as limiting other embodiments.

[0178] An electronic device can input data into a second neural network model 1000, which includes one or more transformer blocks 1005, 1010, 1020, 1025, 1035, and 1040, and one or more operation blocks 1015 and 1030. The second neural network model 1000 can receive data. In the second neural network model 1000, inference results from the data can be output. Compared to the first neural network model, the second neural network model 1000 can output inference results from the data using fewer resources. Tags can be generated based on the data input to the second neural network model 1000. For example, data can be processed into tags and input into the transformer blocks. The position of the tag can be identified by an index indicating the tag position. One or more operation blocks 1015 and 1030 can receive tags generated based on the data as input and selectively perform operations based on the tag indices.

[0179] The second neural network model 1000 can output inference results faster than the first neural network model. The second neural network model 1000 can output inference results with higher accuracy than neural network models that perform structured pruning and / or unstructured pruning compression.

[0180] Reference Figure 11 The second neural network model 1100 is shown. The second neural network model 1100 can be represented according to the above reference. Figures 6 to 9 The described method is a neural network model that compresses the first neural network model. Figure 11 The second neural network model 1100 in the description is used to describe the reasoning of the compressed neural network model and should not be construed as a limitation on other embodiments.

[0181] The second neural network model 1100 may be obtained or derived from the first neural network model, which includes multiple transformer blocks. The second neural network model 1100 may include target transformer blocks 1115, 1125, and 1135, wherein at least one target transformer block, determined based on the number of targets related to the compression degree of the first neural network model, is compressed.

[0182] The second neural network model 1100 may include one or more transformer blocks 1105, 1110, 1120, 1130, 1040, 1045, and 1150, and one or more compression target transformer blocks 1115, 1125, and 1135. The one or more transformer blocks 1105, 1110, 1120, 1130, 1040, 1045, and 1150 may be original transformer blocks that have not been transformed or converted into one or more compression target transformer blocks 1115, 1125, and 1135 included in the first neural network model, wherein the first neural network model is the model before compression of the second neural network model 1100. The one or more compression target transformer blocks 1115, 1125, and 1135 may include at least one of a computation block 1115, a structured-pruned transformer block 1125, and an unstructured-pruned transformer block 1135.

[0183] Reference Figure 11 One or more compression target transformer blocks 1115, 1125, and 1135 may include computation block 1115, structured-pruned transformer block 1125, and unstructured-pruned transformer block 1135. However, this is merely an example for describing the second neural network model 1100 and should not be construed as limiting other embodiments. For example, it will be apparent to those skilled in the art that the description of this disclosure can also be applied to cases that only include computation block 1115 and structured-pruned transformer block 1125.

[0184] At least one arithmetic block 1115 may be a target transformer block, which is transformed into arithmetic block 1115 of a target compression scheme. At least one arithmetic block 1115 may perform an operation based on a predetermined value of an initial marker in the input to the arithmetic block. At least one arithmetic block 1115 may perform addition, adding only the predetermined value and the initial marker. A detailed description of the addition performed by this arithmetic block has been referenced above. Figure 7 and Figure 10 Provided, therefore omitted here.

[0185] At least one arithmetic block 1115 may include at least one of a first arithmetic block and a second arithmetic block. Detailed descriptions of the first and second arithmetic blocks have been referenced above. Figure 10 Provided, therefore omitted here. Although the second neural network model 1100 does not show a second operational block, this is merely an example and should not be construed as a limitation on other embodiments. For example, the second neural network model may include at least one of a first operational block, a second operational block, a structured-pruned transformer block 1125, and an unstructured-pruned transformer block 1135.

[0186] The second neural network model 1100 may include at least one of a structured-pruned transformer block 1125 and an unstructured-pruned transformer block 1135. The structured-pruned transformer block 1125 may be a compressed target transformer block based on structured pruning (a target compression method). The unstructured-pruned transformer block 1135 may be a compressed target transformer block based on unstructured pruning (a target compression method).

[0187] Structured pruning can be a method of directly removing an entire layer or channel, or a method of removing weights according to a specific pattern. Unstructured pruning is a method of removing individual weights, and can be a method of removing weights whose values ​​are less than a threshold. Detailed descriptions of structured and unstructured pruning have been referenced above. Figure 3 Provided, therefore omitted here.

[0188] Referenced above Figure 6 The method for determining a target transformer block to perform at least one of structured pruning, unstructured pruning, and transformation into a first neural network model, as well as a method for determining a target compression method, is described, and therefore its detailed description will be omitted.

[0189] An electronic device can acquire data for inference using a second neural network model 1100. This data may include content such as at least one of images, text, video, audio, and code. For example, the data may include multiple sentences input to the second neural network model 1100 for translation. For example, the data may include images input to the second neural network model 1100 to obtain image descriptions. However, the above data is merely an example for ease of description and should not be construed as limiting other embodiments.

[0190] The electronic device may include one or more converter blocks 1105, 1110, 1120, 1130, 1040, 1045, and 1150, and one or more compression target converter blocks. The one or more compression target converter blocks may include at least one of a computation block 1115, a structured-pruned converter block 1125, and an unstructured-pruned converter block. The computation block 1115 may include at least one of a first computation block and a second computation block.

[0191] An electronic device can input data into a second neural network model 1100, which includes one or more converter blocks 1105, 1110, 1120, 1130, 1040, 1045, and 1150, and one or more compression target converter blocks 1115, 1125, and 1135. The second neural network model 1100, including one or more converter blocks 1105, 1110, 1120, 1130, 1040, 1045, and 1150, and one or more compression target converter blocks 1115, 1125, and 1135, can receive data.

[0192] The second neural network model 1100 can output inference results for the data. Compared to the first neural network model, the second neural network model 1100 can output inference results for the target input using fewer resources. The second neural network model 1100 can output inference results faster than the first neural network model. Compared to neural network models that are compressed by performing structured pruning or unstructured pruning, the second neural network model 1100 can output inference results with higher accuracy.

[0193] Figure 12 and Figure 13 This is a flowchart illustrating a method of operating an electronic device according to an embodiment.

[0194] Reference Figure 12 The diagram shows a flowchart illustrating the operation of an electronic device for performing compression of a first neural network model.

[0195] In the following embodiments, the operations may be performed sequentially, but not necessarily sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Figure 12 The operations shown can be performed by at least one component of the electronic device. For example, the electronic device may include a memory storing instructions. When the instructions are executed individually and / or jointly by at least one processor, the electronic device can perform the following operations.

[0196] In operation 1210, the electronic device can determine the deviation of each of the multiple transformer blocks in the first neural network model. The deviation can represent the difference between the output and input of each transformer block.

[0197] In operation 1220, the electronic device can determine the target converter block among a plurality of converter blocks based on the number of targets (e.g., the number of targets or the target compression level) related to the compression degree of the first neural network model.

[0198] In operation 1230, the electronic device can obtain a second neural network model by performing compression (e.g., structured pruning or unstructured pruning) on ​​the first neural network model based on the bias.

[0199] Operations 1210 to 1230 have been referenced above. Figures 1 to 11 A detailed description has been provided, therefore its detailed description will be omitted.

[0200] Reference Figure 13 The diagram shows a flowchart of inference using a neural network model (e.g., a second neural network model).

[0201] In the following embodiments, the operations may be performed sequentially, but not necessarily sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Figure 13 The operations shown can be performed by at least one component of the electronic device. For example, the electronic device may include a memory storing instructions. When the instructions are executed individually and / or jointly by at least one processor, the electronic device can perform the following operations.

[0202] In operation 1310, the electronic device can cause a neural network model, including at least one converter block and at least one computation block, to receive data. The electronic device can input data into the neural network model, including at least one converter block and at least one computation block.

[0203] In operation 1320, the electronic device can output the inference results of the neural network model on the data.

[0204] At least one operation block can receive tags generated based on data as input. At least one operation block can selectively perform operations on at least some of the tags based on the tags' indices.

[0205] Operations 1310 and 1320 have been referenced above. Figures 1 to 11 A detailed description has been provided, therefore its detailed description will be omitted.

[0206] Based on the phenomenon that neural network models pay close attention to the initial label, this disclosure provides a method for compressing neural network models, which maintains the performance of the neural network model by performing operations on the initial label even when most of the operations of the transformer block are omitted.

[0207] This disclosure allows compressed neural network models to maintain performance while compressing their size, enabling them to run on a variety of devices with limited hardware resources, such as smartphones, head-mounted displays (HMDs), personal computers (PCs), and tablet PCs.

[0208] The embodiments described herein can be implemented using hardware components, software components, and / or combinations thereof. The processing device can be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), DSPs, microcomputers, FPGAs, programmable logic units (PLUs), microprocessors, or any other device capable of responding to and executing instructions in a defined manner. The processing device can run an operating system and one or more software applications running on the operating system. The processing device can also access, store, manipulate, process, and create data in response to the execution of software. For simplicity, the singular is used to describe the processing device; however, those skilled in the art will recognize that the processing device can include multiple processing elements and / or various types of processing elements. For example, the processing device can include multiple processors, or a single processor and a single controller. Furthermore, different processing configurations are also possible, such as parallel processors.

[0209] Software can include computer programs, code, instructions, or some combination thereof, to independently or uniformly instruct or configure a processing device to operate as needed. Software and data can be stored in any type of machine, component, physical or virtual device, or computer storage medium or device capable of providing instructions or data to or being interpreted by the processing device. Software can also be distributed across network-coupled computer systems, enabling it to be stored and executed in a distributed manner. Software and data can be stored on one or more non-transitory computer-readable recording media.

[0210] The methods according to the above embodiments can be recorded in a non-transitory computer-readable medium including program instructions for implementing the various operations of the above embodiments. The medium may also include program instructions, data files, data structures, etc., alone or in combination. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of the embodiments, or they may be of types known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include: magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as CD-ROMs and / or DVDs; magneto-optical media, such as optical discs; and hardware devices specifically configured for storing and executing program instructions, such as read-only memory (ROM), RAM, flash memory, etc. Examples of program instructions include both machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0211] The aforementioned hardware device can be configured to act as one or more software modules to perform the operations of the above embodiments, or vice versa.

[0212] As described above, although embodiments have been described with reference to the limited accompanying drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved if the described techniques are performed in a different order and / or if the components in the described system, architecture, device, or circuit are combined in a different manner, or replaced or supplemented by other components or their equivalents.

[0213] Therefore, other embodiments, other examples and equivalents of the claims are within the scope of the appended claims.

Claims

1. A method for operating an electronic device, the method comprising: Data is received through a neural network model that includes at least one transformer block and at least one computation block; as well as Output the inference results of the neural network model for the given data. The at least one arithmetic block is configured to receive a tag generated based on the data as input, and to selectively perform operations on at least one of the tags based on the index of the tag.

2. The method according to claim 1, wherein, The at least one operation block is configured to perform operations on the initial tag corresponding to the first tag based on the index of the tag.

3. The method according to claim 1, wherein, The at least one operation block is configured to output the remaining tags, excluding the initial tag corresponding to the first tag, as is, based on the index of the tag.

4. The method according to claim 1, wherein, The operations include arithmetic operations.

5. The method according to claim 4, wherein, The operation is an addition operation, which adds a predetermined value and an initial tag corresponding to the first tag based on the index of the tag.

6. The method according to claim 5, wherein, The predetermined value is determined based on one or more deviations, which represent the difference between the output and input of one or more target converter blocks replaced by the at least one computational block.

7. A method of operating an electronic device, the method comprising: Determine the deviation between the output and input of each of the multiple transformer blocks representing the first neural network model; Based on the target number of converter blocks related to the compression degree of the first neural network model, a target converter block is determined among the plurality of converter blocks; as well as A second neural network model is obtained by compressing the first neural network model based on the deviation.

8. The method according to claim 7, wherein, The second neural network model includes a first computation block that replaces the target transformer block with compression, and The first operation block is configured to perform operations based on the deviation of the target converter block corresponding to the first operation block.

9. The method according to claim 8, wherein, The first arithmetic block is configured to perform an addition operation on the initial marker in the input of the first arithmetic block based on the deviation of the target converter block replaced by the first arithmetic block, and to pass the result of the addition operation to the next layer.

10. The method according to claim 7, wherein, Based on at least two target transformer blocks determined by the target quantity as consecutive target transformer blocks, the second neural network model includes a second operational block in which the consecutive target transformer blocks performing the operation are replaced by merging the deviations of the consecutive target transformer blocks.

11. The method according to claim 7, wherein, Obtaining the second neural network model includes: generating the second neural network model by performing structured pruning on at least one of the target transformer blocks in the first neural network model, which is determined by the number of targets.

12. The method according to claim 7, wherein, Obtaining the second neural network model includes: generating the second neural network model by performing unstructured pruning on at least one of the target transformer blocks in the first neural network model, which is determined by the number of targets.

13. The method of claim 7, further comprising: Determine the number of targets in the first neural network model that are related to the compression level of the first neural network model. Determining the target quantity includes: determining the target quantity based on the hardware resources of the second neural network model obtained by the target device performing compression on the first neural network model.

14. The method according to claim 7, wherein, Determining the target converter block includes: Determine whether the currently determined target number of converter blocks has reached the target number; and Since the number of target converter blocks currently determined has not reached the target number, additional target converter blocks are determined based on the result of sequentially transforming any one of the remaining converter blocks other than the currently determined target converter blocks into a first operation block that performs addition operations only on the initial mark.

15. The method according to claim 14, wherein, Determining the target transformer block further includes: identifying the remaining transformer block that has the least impact on the output of the first neural network model when transformed into the first operational block as the target transformer block.

16. An electronic device comprising: The memory is configured to store instructions; as well as At least one processor is configured to execute the instructions. When the instructions are executed individually or jointly by the at least one processor, the electronic device performs the following operations: Data is received by a neural network model comprising at least one transformer block and at least one computation block; and Output the inference results of the neural network model for the data, and The at least one computation block is configured to receive a tag generated based on the data as input, and to selectively perform operations on at least one of the tags based on the index of the tag.

17. The electronic device according to claim 16, wherein, The at least one operation block is configured to perform operations on the initial tag corresponding to the first tag based on the index of the tag.

18. The electronic device according to claim 16, wherein, The at least one operation block is configured to output the remaining tags, excluding the initial tag corresponding to the first tag, as is based on the index of the tag.

19. The electronic device according to claim 16, wherein, The operations include arithmetic operations.

20. The electronic device according to claim 19, wherein, The operation is an addition operation, which adds a predetermined value and an initial tag corresponding to the first tag based on the index of the tag.

Images