Method and system for lightening neural network in consideration of connection relationship

Abstract

Disclosed are a method and a system for lightening a neural network in consideration of a connection relationship. The lightening method according to an embodiment may comprise the steps of: receiving a neural network model including attention; and generating a lightened neural network model by removing neurons of the same index of a query layer and a key layer included in the attention.

Description

Method and system for lightweighting neural networks considering connectivity

[0001] Embodiments of the present invention relate to a method and system for lightweighting a neural network considering connection relationships.

[0002] In the prior art, there have been various research approaches to lighten and accelerate over-parameterized transformer models.

[0003] For example, token merging technology can reduce computational load by combining similar input patches, and head pruning technology can eliminate independent heads, resulting in substantial weight reduction and acceleration. However, both token merging and head pruning technologies suffer from significant performance degradation.

[0004] As another example, neuron pruning techniques have the advantage of not causing any performance degradation compared to the upper-level lightweight techniques, but have the disadvantage of not providing any practical acceleration or lightweighting benefits for the transformer model.

[0005] Fully connected layers, which dominate multi-head self-attention (MHSA), unlike convolutional operations, do not have receptive fields, making it difficult to distinguish between structured and unstructured pruning. Nevertheless, existing technologies force the use of specific devices or libraries to achieve acceleration and weight reduction by masking unnecessary weights to zero.

[0006] In addition, the pruning of conventional transformers has been developed into head pruning, which removes some of the heads of multi-head self-attention (MHSA) depending on the target to be lightweighted, and neuron level sparsity, which masks the neurons themselves.

[0007] Since neither of the two methods is a lightweighting technique that takes the characteristics of the transformer into account, important information is lost during lightweighting, so learning after lightweighting is essential.

[0008] Head pruning, in contrast, involves removing some heads from the MHSA, a key component of the transformer, for acceleration purposes. It is not a lightweighting technique based on the specific operating principles or utility of MHSA. For this reason, additional learning is required after head pruning to compensate for the lost information, and it is extremely difficult to restore the performance of the original model due to the information lost through head pruning.

[0009] Neuron-level sparsity is a technique for removing relatively insignificant neurons. However, previous studies on neuron-level sparsity lack consideration of the influence and role of each neuron, making them inefficient in terms of performance and weight reduction. Furthermore, additional training is required to restore performance after each weight reduction technique.

[0010] A lightweight method and system can be provided that can lightweight a neural network including attention by considering the connection relationship.

[0011] A lightweight method and system can be provided by minimizing loss of attention scores.

[0012] The above lightweight method and system can provide a neural network model including lightweight attention.

[0013] An inference method and system using a neural network model including the above lightweight attention can be provided.

[0014] A method for reducing a neural network model including attention by a computer device including at least one processor, the method comprising: receiving a neural network model including attention; and generating a reduced neural network model by removing, by the at least one processor, neurons of the same index in a query layer and a key layer including the attention.

[0015] According to one aspect, the step of generating the lightweight neural network model may include: selecting a neuron to be removed from the query layer or the key layer; removing the selected neuron from the query layer or the key layer; and, if the neuron to be removed is selected from the query layer, removing a neuron of the key layer having the same index as the selected neuron from the key layer, and if the neuron to be removed is selected from the key layer, removing a neuron of the query layer having the same index as the selected neuron from the query layer.

[0016] According to another aspect, the lightweight method may further include a step of removing at least one neuron of a value layer including the attention, and the number and index of neurons removed from the value layer may be determined independently from neurons removed from the query layer and the key layer.

[0017] According to another aspect, the step of generating the lightweight neural network model may be characterized in that, when the attention is multi-head self-attention (MHSA), the same amount of neurons are removed for each head included in the multi-head self-attention to generate the lightweight neural network model.

[0018] According to another aspect, the attention is multi-head self-attention, and the output of each of the query layer, the key layer, and the value layer of the multi-head self-attention is reshaped into a matrix having the number of heads and the number of neurons of each head as rows and columns, and the step of generating the lightweight neural network model may be characterized by including the step of determining the index by considering the connection relationship with the operator after the reshaping; and the step of removing the neuron corresponding to the determined index.

[0019] According to another aspect, the step of generating the lightweight neural network model may further include a step of adjusting the reshaping layer and the operator that perform the reshaping based on the determined index.

[0020] According to another aspect, the step of determining the index may be characterized by including the step of determining one of a plurality of optimization policies; and the step of determining the index according to the determined policy.

[0021] According to another aspect, the plurality of optimization policies may be characterized by including two or more of (1) a None policy that removes the same number of neurons for each head of the multi-head self-attention, (2) a Sum policy that determines an index for removing a neuron based on the sum of the importances of neurons of the same index of heads included in the multi-head self-attention, (3) an Average policy that determines an index for removing a neuron based on the average of the importances of neurons of the same index of the heads, and (4) a Count policy that counts the number of neurons to be removed from the heads based on importance by index and determines an index for removing a neuron based on the number of neurons counted by index.

[0022] According to another aspect, the step of determining the index may be characterized by determining the index for removing a neuron based on the sum of the importance of neurons of the same index of heads included in the multi-head self-attention.

[0023] According to another aspect, the step of determining the index may be characterized by determining the index for removing a neuron based on the average of the importance of neurons of the same index of heads included in the multi-head self-attention.

[0024] According to another aspect, the step of determining the index may be characterized by counting the number of neurons to be removed from the heads included in the multi-head self-attention based on importance for each index, and determining an index for removing a neuron based on the number of neurons counted for each index.

[0025] According to another aspect, the step of generating the lightweight neural network model may include: calculating a sum between matrix products of outputs from neurons of the same index of the query layer and the key layer; decomposing the calculated sum by the at least one processor to generate a target matrix; selecting a neuron to be removed from the attention based on a similarity of the matrix products with the target matrix by the at least one processor; and generating the lightweight neural network model by removing the selected neuron from the attention.

[0026] According to another aspect, the step of generating the target matrix may be characterized by generating the target matrix through singular value decomposition of the calculated sum.

[0027] According to another aspect, the target matrix may be characterized as being composed of eigenvectors corresponding to the smallest singular value.

[0028] According to another aspect, the step of selecting the neuron may be characterized by selecting the neuron to be removed by determining whether to prune the neurons of the query layer and the neurons of the key layer used in the calculation of each of the matrix products based on the similarity of the matrix products.

[0029] According to another aspect, the step of selecting the neuron may be characterized by selecting the neuron of the query layer and the neuron of the key layer corresponding to the matrix product having the highest similarity with the square matrix as the neuron to be removed through pruning.

[0030] According to another aspect, the similarity may be characterized as including cosine similarity.

[0031] According to another aspect, it may be characterized in that neurons of the same index in the query layer and the key layer are removed.

[0032] According to another aspect, the step of calculating the sum, the step of generating the target matrix, the step of selecting the neurons, and the step of generating the lightweight neural network model may be characterized in that they are repeatedly performed a number of times determined based on the performance of the lightweight neural network model.

[0033] According to another aspect, the step of calculating the sum, the step of generating the target matrix, the step of selecting the neurons, and the step of generating the lightweight neural network model may be characterized in that they are repeatedly performed while the performance of the lightweight neural network model satisfies a performance range set by a user without retraining the lightweight neural network model.

[0034] A neural network model including lightweight attention is provided according to the above method.

[0035] A method for inference of a computer device including at least one processor is provided, comprising a step of processing inference for input data using a neural network model including attention from which neurons of the same index in a query layer and a key layer are removed.

[0036] According to one side, when the attention is multi-head self-attention (MHSA), it can be characterized in that the same amount of filters are removed for each head included in the multi-head self-attention.

[0037] According to another aspect, the attention may be characterized in that it is lightweight by removing selected neurons based on the similarity between the matrix products generated by decomposing the sum of the matrix products of the outputs through the neurons of the same index of the query layer and the key layer and the square matrix.

[0038] A computer program stored on a computer-readable recording medium is provided for executing the above method on a computer device in combination with a computer device.

[0039] A computer-readable recording medium having recorded thereon a program for executing the above method on a computer device is provided.

[0040] A computer device is provided, comprising at least one processor implemented to execute instructions readable from a computer device, characterized in that the computer device receives a neural network model including attention by the at least one processor, and generates a lightweight neural network model by removing neurons of the same index of a query layer and a key layer including the attention.

[0041] A computer device is provided, comprising at least one processor configured to execute instructions readable from a computer device, characterized in that the at least one processor processes inference on input data using a neural network model including attention in which neurons of the same index in a query layer and a key layer are removed.

[0042] A lightweight method and system can be provided that can lightweight a neural network including attention by considering the connection relationship.

[0043] In addition, a lightweight method and system can be provided by minimizing loss of attention scores.

[0044] In addition, a neural network model including lightweight attention can be provided through the above lightweight method and system.

[0045] In addition, an inference method and system using a neural network model including the above lightweight attention can be provided.

[0046] In addition, a lightweight method and system for neural networks that consider the connection relationship of attention can complement the limitations of existing neuron pruning, which does not affect speed and model size, while preserving the advantages of neuron pruning.

[0047] Additionally, it can reduce the training or inference cost of very large models such as ChatGPT, and can create more powerful models than existing models by acting as a regularization term.

[0048] In addition, by presenting neuron-level sparsity that takes into account the characteristics of the neural network, a lightweight method and system that can preserve existing performance to a certain extent without learning are provided, thereby eliminating the need for relearning or minimizing relearning.

[0049] FIG. 1 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.

[0050] FIG. 2 is a drawing illustrating an example of a lightweighting process according to one embodiment of the present invention.

[0051] FIG. 3 is a drawing illustrating another example of a lightweighting process according to one embodiment of the present invention.

[0052] FIG. 4 is a diagram illustrating an example of an optimization process of MHSA considering a reshape operator in one embodiment of the present invention.

[0053] FIGS. 5 and 6 are drawings illustrating an example of applying the same operator to each neuron of a head in one embodiment of the present invention.

[0054] FIG. 7 is a diagram for explaining a sum policy and an average policy in one embodiment of the present invention.

[0055] FIG. 8 is a drawing for explaining a count policy in one embodiment of the present invention.

[0056] Figure 9 is a flowchart illustrating an example of a lightweighting method according to one embodiment of the present invention.

[0057] Figure 10 is a flowchart illustrating an example of an inference method according to one embodiment of the present invention.

[0058] FIG. 11 is a diagram showing an example of the result of matrix multiplication of outputs from neurons of the same index as the query and key in one embodiment of the present invention.

[0059] FIG. 12 is a diagram illustrating an example of a process for calculating a square matrix in one embodiment of the present invention.

[0060] FIG. 13 is a diagram illustrating an example of a process for selecting a neuron by comparing the matrix product between outputs of neurons of the same index and a square matrix in one embodiment of the present invention.

[0061] Figure 14 is a flowchart illustrating another example of a lightweighting method according to one embodiment of the present invention.

[0062] FIG. 15 is a flowchart illustrating another example of an inference method according to one embodiment of the present invention.

[0063] Hereinafter, the detailed description will be given with reference to the attached drawings.

[0064] The lightweight system according to embodiments of the present invention can be implemented by at least one computer device, and the lightweight method according to embodiments of the present invention can be performed by at least one computer device included in the lightweight system. The computer device can have a computer program according to an embodiment of the present invention installed and run, and the computer device can perform the lightweight method according to embodiments of the present invention under the control of the run computer program. The above-described computer program can be stored in a computer-readable recording medium so as to be coupled to the computer device and cause the computer device to execute the lightweight method.

[0065] FIG. 1 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. As illustrated in FIG. 1, the computer device (100) may include a memory (110), a processor (120), a communication interface (130), and an input / output interface (I / O interface) 140. The memory (110) is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as the disk drive may be included in the computer device (100) as a separate permanent storage device distinct from the memory (110). In addition, an operating system and at least one program code may be stored in the memory (110). These software components may be loaded into the memory (110) from a computer-readable recording medium separate from the memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (110) via a communication interface (130) other than a computer-readable recording medium. For example, the software components may be loaded into the memory (110) of the computer device (100) based on a computer program installed by files received via a network (Network, 160).

[0066] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via the memory (110) or the communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a storage device such as the memory (110).

[0067] The communication interface (130) may provide a function for the computer device (100) to communicate with other devices via a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer device (100) according to program codes stored in a recording device such as a memory (110) may be transmitted to other devices via the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (100) via the communication interface (130) of the computer device (100) via the network (160). Signals, commands, data, etc. received via the communication interface (130) may be transmitted to the processor (120) or the memory (110), and files, etc. may be stored in a storage medium (the aforementioned permanent storage device) that the computer device (100) may further include.

[0068] The input / output interface (140) may be a means for interfacing with an input / output device (I / O device, 150). For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface (140) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (150) may also be configured as a single device with the computer device (100).

[0069] Additionally, in other embodiments, the computer device (100) may include fewer or more components than those illustrated in FIG. 1. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (100) may be implemented to include at least some of the input / output devices (150) described above, or may further include other components such as a transceiver, a database, etc.

[0070] Embodiments of the present invention relate to a method and system for reducing the weight of a neural network that includes attention. Here, reducing the weight may involve reducing the size of the attention and / or accelerating the attention. The invention will be described below primarily with reference to self-attention in a transformer model as an example of attention; however, the embodiments of the present invention are not limited thereto.

[0071] Self-attention consists of three weights: query, key, and value. In this self-attention, an attention map is generated using the query and key, which indicates the probability of which token to focus on among the given tokens, and the generated attention map is then multiplied by the value and a matrix. At this time, the output of the key layer is transposed and matrix-multiplied with the output of the query layer during the process of generating the attention map. In conventional pruning techniques, the filters that serve as weights of the neurons included in the query layer and / or key layer are masked to 0, so that even though the output value corresponding to the neuron with the masked filter is 0, the matrix multiplication is performed between the 0 output and the output of the key layer or query layer. Although the attention map generated in this way is significantly distorted compared to the attention map of the original model, it only causes a decrease in performance without improving the speed or model size.

[0072] FIG. 2 is a diagram illustrating an example of a lightweighting process according to an embodiment of the present invention. In order to solve the problems of the conventional pruning technique described above, in the lightweighting process according to the present embodiment, neurons of the same index in the query layer and the key layer included in self-attention can be removed, as shown in the first dotted box (210). Here, the query layer and the key layer can correspond to layers for calculating the query and the key among the basic components of self-attention. Therefore, the shape of the attention map does not change as the out features of the same index in the query and the key are removed. Unlike the conventional pruning technique that masks the filters of the neurons in the query layer and the key layer to 0, this technique actually removes the neurons in the query layer and the key layer, thereby obtaining advantages in terms of speed and model size.

[0073] When neurons of different indices are removed, matrix multiplication is performed between the outputs of neurons of different indices, and thus the attention performance is not preserved. Therefore, as explained above, by removing neurons of the same index, the performance degradation during attention pruning can be minimized.

[0074] Meanwhile, the output of the value layer shown in the second dotted box (220) can be independently pruned because it is matrix-multiplied again with the attention map generated through matrix multiplication between the outputs of the query layer and the key layer. For example, when lightweighting self-attention, after receiving the first lightweighting degree for the query layer and the key layer and the second lightweighting degree for the value layer from the user, the query layer and the key layer can be pruned by removing neurons of the same index according to the first lightweighting degree, and the value layer can be pruned independently by removing neurons according to the second lightweighting degree. The value layer may correspond to a layer for calculating values ​​among the basic components of self-attention.

[0075] This embodiment describes pruning, which removes neurons based on out-of-features of a fully connected layer, rather than a technique that removes neurons based on in-features of a fully connected layer of self-attention, but is not limited thereto. For example, the lightweight system according to this embodiment may also remove neurons with the same index in the query layer and key layer based on in-features of a fully connected layer.

[0076] In embodiments of the present invention, self-attention is defined as a basic component of query, key, and value, and the feed forward network behind self-attention is not specifically described because there is no need to consider connectivity.

[0077] FIG. 3 is a diagram illustrating another example of a lightweighting process according to an embodiment of the present invention. In the case of multi-head self-attention (MHSA), the process can be basically performed in the same manner as the example of the lightweighting process according to the embodiment of FIG. 2. However, in most implementations, queries, keys, and values ​​are implemented and calculated regardless of the head, as shown in FIG. 4, to accelerate operations. In this case, since the part that divides by head after calculating each query, key, and value (for example, the reshape operator that changes each calculated query, key, and value into the form of a feature (or neuron) for each head x number of heads (for example, the reshape layer (2, 4) in FIG. 4), etc.), the same amount of neurons must be removed for all heads. The embodiment of FIG. 3 shows an example in which not only the neurons of the same index are removed for each head of the query layer and the key layer as in the embodiment of FIG. 2, but also the same number of neurons are removed for each head. As a more specific example, the embodiment of FIG. 3 shows an example in which the same number of neurons are removed by removing one neuron each from the query layer and the key layer from Head#1, and one neuron each from the query layer and the key layer from Head#2.

[0078] FIG. 4 is a diagram illustrating an example of an optimization process of MHSA considering a reshape operator in one embodiment of the present invention.

[0079] In this embodiment, in order to accelerate MHSA consisting of multiple self-attentions, most MHSAs are implemented with a structure in which the output of one linear layer having the number of heads X the number of neurons in each head as an out feature is reshaped into the heads and the number of neurons in each head, as in the embodiment of Fig. 4, so that each head is calculated simultaneously in a form like a batch, thereby optimizing the computation speed. For example, the reshape operator can reshape the output of one linear layer having the number of heads X the number of neurons in each head as an out feature into a matrix having the number of heads and the number of neurons in each head as rows and columns.

[0080] In this embodiment, the None policy, the Average policy, the Sum policy, and the Count policy are proposed as optimization policies to maintain the above structure and accelerate the MHSA lightweighting.

[0081] In the case of the null policy, as described in the embodiment of FIG. 4, it is a technique that can be applied to most general MHSAs, which is a technique for making MHSA lightweight by removing the same number of neurons for each head so that the structure of MHSA using the reshape operator can be preserved. In this case, since there is no additional operator for each neuron of the head after reshaping in MHSA, and the output of MHSA is flattened for the feedforward network after MHSA, the null policy that removes the same number of neurons for each head can be utilized. For example, the output of MHSA can have the form of (batch, number of heads, tokens, neurons). In this case, for the feedforward network after MHSA, the output can be flattened to have the form of (batch, tokens, number of heads Х neurons).

[0082] On the other hand, if the output of MHSA in the form of (batch, number of heads, tokens, neurons) passes through the operator before the flattening, it is necessary to lighten it by considering the index.

[0083] FIGS. 5 and 6 are diagrams illustrating examples of applying the same operator to each neuron of a head according to one embodiment of the present invention. If a specific operator needs to be applied to each neuron of the heads after reshaping, the indices of the neurons to be removed for each head need to be unified. In the case of the null policy, as explained above, since it targets the case where the same operator is not applied to each neuron after the reshape operator, it refers to the case where only the number of neurons to be removed is unified regardless of the location of the neuron to be removed, and most MHSAs can respond through the null policy. In other words, if a specific operator is not applied to each neuron of the heads after the reshape operator, the same number of neurons for each head can be removed through the null policy.

[0084] Conversely, when a specific operator is applied after the reshape operator, the indices of the neurons to be removed can be unified for each head to unify the locations of the neurons to be removed. These specific operators may include operators that calculate the remainder by arranging them based on one axis of the reshape, such as elementwise multiplication or linear layers for projection to other dimensions. Here, one axis can mean the axis of neurons arranged according to the same index for each head through the reshape.

[0085] FIG. 7 is a diagram for explaining a sum policy and an average policy in one embodiment of the present invention.

[0086] As explained above, the reshape layer can reshape the output of a linear layer whose out-features are the number of heads X the number of neurons in each head, into a matrix whose rows and columns are the number of heads and the number of neurons in each head. At this time, the elements of one column of the reshape matrix can correspond to the outputs of neurons of the same index in the heads. For example, in Fig. 7, neurons 0, 4, and 8 can be neurons of the same index in three heads.

[0087] The sum policy may refer to a policy that unifies and determines the index of a neuron to be removed from a head based on the sum of the importances of neurons of the same index in the heads. For example, when using the sum policy in the embodiment of FIG. 7, the index for neurons 0, 4, and 8 of the same index in the heads may be determined based on the sum of the importances, and then the neurons of the determined index may be removed from the heads.

[0088] Additionally, the average policy may refer to a policy that unifies and determines the index of a neuron to be removed from a head based on the average importance of neurons of the same index in the heads. For example, in the embodiment of FIG. 7, when using the average policy, the index for neurons 0, 4, and 8 of the same index in the heads may be determined based on the average importance, and then the neurons of the determined index may be removed from the heads.

[0089] When using the sum policy, the importance of the corresponding neuron is greater than that of the general layers, so the layer that has been relatively changed to the reshape operator is judged to be more important, so it is relatively less lightweight than when using the average policy. However, when using the average policy, the importance of the neurons is measured equally regardless of the number of heads, so more neurons can be removed compared to when using the sum policy. In addition, since the number of neurons in each head decreases as neurons are removed, both the reshape layer and the operator after the reshape layer need to be adjusted. For example, FIGS. 5 and 6 above show that the input and output of the reshape layer can be adjusted from 3X4 to 3X3 due to the removal of the neurons in the head. The operator after the reshape layer can also be adjusted similarly.

[0090] FIG. 8 is a diagram illustrating a count policy according to an embodiment of the present invention. While the numbers inside the circles representing neurons in the embodiments of FIGS. 5 to 7 represent the indices of the neurons, in the embodiment of FIG. 8, the numbers inside the circles representing neurons are assumed to represent the importance of the corresponding neurons. At this time, in the case of the count policy, similar to the null policy, each neuron can be independently represented regardless of the head and without changing the importance of each neuron. For example, for a given degree of weight reduction of the layer (for example, 6 neurons in the embodiment of FIG. 8), when neurons are removed according to importance, the number of neurons removed for each head with the same index can be counted, and the index of the axis with the largest number of neurons removed can be selected as the index for neuron removal. Thereafter, neurons with the determined indices can be removed from the heads.

[0091] Fig. 9 is a flowchart illustrating an example of a lightweight method according to an embodiment of the present invention. The lightweight method according to the present embodiment can be performed by the computer device (100) described through Fig. 1. At this time, the processor (120) of the computer device (100) can be implemented to execute control instructions according to the code of the operating system included in the memory (110) or the code of at least one computer program. Here, the processor (120) can control the computer device (100) so that the computer device (100) performs steps (910 to 930) included in the method of Fig. 9 according to the control instructions provided by the code stored in the computer device (100).

[0092] In step (910), the computer device (100) may receive a neural network model including attention. For example, the neural network model may be a transformer model, and the attention may be self-attention included in the transformer model, but is not limited thereto.

[0093] In step (920), the computer device (100) can generate a lightweight neural network model by removing neurons of the same index in the query layer and key layer that the attention includes. As described above, if neurons of different indices are removed, the attention performance is not preserved as matrix multiplication is performed between the outputs of neurons of different indices. On the other hand, by removing neurons of the same index as in the present embodiment, the performance degradation during attention pruning can be minimized.

[0094] This step (920) may include steps (921 to 923) as illustrated in FIG. 9.

[0095] In step (921), the computer device (100) may select neurons to be removed from the query layer or the key layer. For example, the computer device (100) may measure the importance of each neuron in the query layer and / or the key layer of attention, and select neurons to be removed based on the measured importance. For example, the computer device (100) may receive from the user a first degree of weight reduction for the query layer and the key layer and a second degree of weight reduction for the value layer when weight reduction of self-attention is performed. Thereafter, the computer device (100) may prune the query layer and the key layer by removing neurons of the same index according to the first degree of weight reduction and the measured importance. In some embodiments, when a neuron to be removed is selected in one of the query layer and the key layer, neurons of the same index are removed in the other layer, so the importance of each neuron may be measured only in one of the query layer and the key layer. Additionally, the computer device (100) can prune the value layer independently from the query layer and the key layer by removing neurons from the value layer based on the second weight reduction degree and the measured importance. For example, the number and index of neurons removed from the value layer can be determined independently from the neurons removed from the query layer and the key layer.

[0096] In step (922), the computer device (100) may remove the selected neuron from the query layer or the key layer. For example, if a neuron is selected from the query layer, the neuron may be removed from the query layer, and if a neuron is selected from the key layer, the neuron may be removed from the key layer.

[0097] In step (923), the computer device (100) can remove a neuron of the key layer having the same index as the selected neuron from the key layer if the neuron to be removed is selected from the query layer, and can remove a neuron of the query layer having the same index as the selected neuron from the query layer if the neuron to be removed is selected from the key layer. In addition, as described above, the computer device (100) can adjust the reshaping layer and operator that performs the reshaping based on the determined index.

[0098] In step (930), the computer device (100) can remove at least one neuron of the value layer included in the attention. At this time, the computer device (100) can prune the value layer independently from the query layer and the key layer. For example, the number and index of neurons removed from the value layer can be determined independently from the neurons removed from the query layer and the key layer. When weighting the attention, the computer device (100) can receive from the user a first weighting degree for the query layer and the key layer and a second weighting degree for the value layer, respectively. Thereafter, the computer device (100) can prune the query layer and the key layer by removing neurons of the same index according to the first weighting degree, and can independently prune the value layer by removing neurons according to the second weighting degree.

[0099] Additionally, as previously described, a case in which attention is MHSA may be considered. In this case, the computer device (100) may remove neurons by considering the connection relationship between neurons in the query layer and the key layer at step (920), while simultaneously removing an equal number of neurons for each head included in the multi-head self-attention.

[0100] For example, the output of each of the query layer, key layer, and value layer of the MHSA can be reshaped into a matrix having the number of heads and the number of neurons in each head as rows and columns through the reshape operator described above. If there is no specific operator after the reshape operator for reshaping, the same number of neurons can be simply removed from each head. However, if there is an additional operator after the reshape operator, neurons with the same index need to be removed for each head. Therefore, the computer device (100) can determine an index by considering the connection relationship with the operator after the reshape layer in step (920), and can remove the neuron corresponding to the determined index.

[0101] According to an embodiment, the computer device (100) may determine one of a plurality of optimization policies to determine an index. At this time, the plurality of optimization policies may include two or more of (1) a None policy that removes the same number of neurons for each head of the multi-head self-attention, (2) a Sum policy that determines an index for removing a neuron based on the sum of the importances of the neurons of the same index in the heads included in the multi-head self-attention, (3) an Average policy that determines an index for removing a neuron based on the average of the importances of the neurons of the same index in the heads included in the multi-head self-attention, and (4) a Count policy that counts the number of neurons to be removed from the heads included in the multi-head self-attention based on importance by index and determines an index for removing a neuron based on the number of neurons counted by index. In addition, the computer device (100) may determine an index according to the determined policy.

[0102] As a more specific example, the computer device (100) may determine an index for removing a neuron based on the sum of the importance of neurons of the same index of heads included in the multi-head self-attention, in order to determine the index according to the sum policy.

[0103] Additionally, the computer device (100) can determine an index for removing a neuron based on the average of the importance of neurons of the same index of heads included in the multi-head self-attention to determine an index according to an average policy.

[0104] In addition, the computer device (100) can count the number of neurons to be removed from heads included in the multi-head self-attention according to the importance of each neuron for each index in order to determine an index according to a count policy, and determine an index for removing a neuron based on the number of neurons counted for each index.

[0105] Once the index is determined, the computer device (100) can remove neurons of the same index by removing neurons of the determined index from each head.

[0106] The inference system according to embodiments of the present invention can be implemented by at least one computer device, and the inference method according to embodiments of the present invention can be performed by at least one computer device included in the inference system. The computer device can have a computer program according to an embodiment of the present invention installed and run, and the computer device can perform the inference method according to embodiments of the present invention under the control of the run computer program. The above-described computer program can be stored in a computer-readable recording medium so as to be coupled to the computer device and cause the computer device to execute the inference method.

[0107] FIG. 10 is a flowchart illustrating an example of an inference method according to an embodiment of the present invention. The inference method according to the present embodiment can be performed by the computer device (100) described through FIG. 1. At this time, the processor (120) of the computer device (100) can be implemented to execute control instructions according to the code of the operating system included in the memory (110) or the code of at least one computer program. Here, the processor (120) can control the computer device (100) so that the computer device (100) performs steps (1010 to 1030) included in the method of FIG. 10 according to the control instructions provided by the code stored in the computer device (100).

[0108] For example, a transformer model including lightweight self-attention according to the embodiments described above may be provided, and a user who receives the transformer model may process inference on data using the provided transformer model.

[0109] In step (1010), the computer device (100) may load a neural network model including attention with neurons of the same index in the query layer and the key layer removed. For example, the computer device (100) may load program code for the neural network model into the memory (110).

[0110] In step (1020), the computer device (100) may receive data. Here, the data may be input into a neural network model for inference using the neural network model. Depending on the embodiment, the order of steps (1010) and (1020) may be changed.

[0111] In step (1030), the computer device (100) may process inferences on the input data using a neural network model. For example, the computer device (100) may input the data received in step (1020) into a neural network model to process inferences on the data.

[0112] In addition, embodiments of the present invention can provide a lightweight method and system that can reduce self-attention while preserving the existing performance of a pre-trained neural network to a certain extent based on neuron level sparsity that takes into account the characteristics of the neural network, and that can minimize or eliminate the need for re-training.

[0113] Transformer-based models typically consist of multi-head self-attention (MHSA) and a feed-forward network. The MHSA comprises multiple self-attention layers, each of which comprises a query, a key, a value, and two feed-forward networks. The query layer, key layer, and value layer, described below, correspond to the layers for calculating the query, key, and value, respectively, among the basic components of self-attention. Among these self-attention components, the value and feed-forward networks aim to project the inputs of the corresponding layers onto other dimensions, and pruning can mean reducing the dimensionality to which the projection is intended. On the other hand, an attention map created by utilizing the transpose of the query output and the key output (for example, the matrix product of the transpose of the query output and the key output) can play a role in indicating which part of the given input to focus on. These operations consist of linear combinations, and can have a relatively small impact on the size and value of the attention map according to weight reduction compared to dimensionality reduction through pruning. In the embodiments of the present invention, since the goal is to achieve stable performance without learning, the focus is on weight reduction of the query and key. In other words, the focus is on a technique that can be weighted without learning, but additional learning is not completely ruled out, as additional learning can enable further weight reduction and acceleration with only a small performance degradation.

[0114] Self-attention, as is generally well known, can be expressed as in mathematical equation 1 below.

[0115]

[0116] Here, Q, K, and V can be matrices of queries, keys, and values ​​given as input. This mathematical expression 1 can represent the process of calculating attention weights. In mathematical expression 1, the inner product operation QK T can mean the inner product of the query matrix Q and the transpose of the key matrix K. The inner product operation can be an operation that calculates the similarity through multiplication and addition between the elements of a vector. The inner product operation can be used to measure how similar each element of the vector is. The result of the inner product operation is a value representing the similarity, and in attention calculation, it can play a role in measuring the similarity between the query and the key. Using this similarity, the attention weight can be calculated, and the process of adjusting the value matrix V through a weighted sum can follow. The similarity obtained through the inner product operation can be normalized through the softmax function, and the normalized weight can be multiplied by the value matrix V to calculate the final attention output. The attention output calculated in this way can be passed to the next stage of the transformer model as a weight representing the importance of the input. Therefore, the inner product operation in the attention included in the transformer model is used as a core operation that calculates the similarity between the query and the key, and through this, the attention weight and the final output can be determined.

[0117] At this time, weight reduction through query and key pruning inevitably causes distortion, but to achieve the goal of performance degradation, the least influential neurons must be identified and removed. In embodiments of the present invention, this series of processes is attempted to be solved through the characteristics of matrix multiplication and Singular Value Decomposition (SVD).

[0118] QK for each element of the matrix multiplied by the matrix (i,j) T person can be expressed as in mathematical formula 2 below.

[0119]

[0120] Here, is the element at position (i, α) of the query matrix Q, can mean the element at position (α, j) of the key matrix K, and dim can mean the dimension of the matrix.

[0121] At this time, when viewed from the perspective of the entire matrix, mathematical expression 2 can be developed as mathematical expression 3 below.

[0122]

[0123] This mathematical expression 3 is expressed as a sum of matrix products of outputs from neurons of the same index as the query and the key. In embodiments of the present invention, the importance of a corresponding neuron (a neuron of the same index) can be measured by utilizing a matrix as a sum of matrix products of outputs from neurons of the same index as the query and the key. By measuring the importance of a neuron by utilizing the sum of matrix products, the importance of each neuron can be measured by considering the connection between the query and the key.

[0124] FIG. 11 is a diagram showing an example of the result of matrix multiplications of outputs from neurons of the same index as the query and the key, according to an embodiment of the present invention. As described above, an attention map can be generated through matrix multiplication between the matrix as the output of the query and the transpose matrix of the output of the key. At this time, the weight reduction in the query and the key cannot help but affect the attention map. Therefore, in embodiments of the present invention, the matrix of the neuron that is most similar to the least influential or relatively less important part (the square matrix to be described later) can be removed. The embodiment of FIG. 11 shows an example of generating index-specific attention maps, such as the 1-1 matrix to the 1-5 matrix, through matrix multiplications of outputs from neurons of the same index as the query and the key. At this time, the transpose matrix of the output from the neuron of the key can be used for matrix multiplication.

[0125] FIG. 12 is a diagram illustrating an example of a process for calculating a square matrix according to an embodiment of the present invention. The embodiment of FIG. 12 shows an example in which the 2-1 to 2-6 matrices are generated as square matrices made of singular vectors by performing SVD on the calculated for a given layer of self-attention. In this case, it is assumed that the square matrix made of eigenvectors corresponding to the lowest singular value is the 2-6 matrix.

[0126] FIG. 13 is a diagram illustrating an example of a process for selecting neurons by comparing matrix multiplication between outputs of neurons of the same index with square matrices in one embodiment of the present invention. The embodiment of FIG. 13 shows an example of calculating the cosine similarity between each of the index-specific attention maps, which are the 1-1 to 1-5 matrices obtained through the embodiment of FIG. 11, and the 2-6 matrix obtained through the embodiment of FIG. 12. At this time, the higher the cosine similarity of each of the 1-1 to 1-5 matrices with the 2-6 matrix, the lower the importance of the neuron used to calculate the matrix multiplication. The neuron of the matrix with the highest cosine similarity with the 2-6 matrix among the 1-1 to 1-5 matrices may be selected as the neuron most similar to the least influential or relatively less important part (the square matrix to be described later). For example, if n neurons are set to be pruned in each of the query layer and the key layer, the neuron corresponding to the n matrix product in order of highest cosine similarity may be selected as the neuron to be removed. Here, the number of neurons set to be pruned may be determined by the user.

[0127] Additionally, the transformer model can be lightened by repeating the method according to the embodiments of FIGS. 11 to 13 until the user can obtain the desired performance without learning for the transformer model including the lightened self-attention, or until the preset performance is obtained.

[0128] Fig. 14 is a flowchart illustrating another example of a lightweighting method according to an embodiment of the present invention. The lightweighting method according to the present embodiment can be performed by the computer device (100) described through Fig. 1. At this time, the processor (120) of the computer device (100) can be implemented to execute control instructions according to the code of the operating system included in the memory (110) or the code of at least one computer program. Here, the processor (120) can control the computer device (100) so that the computer device (100) performs steps (1410 to 1430) included in the method of Fig. 14 according to the control instructions provided by the code stored in the computer device (100).

[0129] In step (1410), the computer device (100) may calculate a sum between matrix products (e.g., index-specific attention maps) of outputs from neurons of the same index in the query layer and the key layer included in the attention of the neural network model. For example, the computer device (100) may calculate a sum between matrix products between the output of a filter of the query layer and the transpose of the output of a neuron of the same index in the key layer by calculating the matrix products. In one embodiment, the neural network model may include a transformer model, and the attention may include self-attention of the transformer model.

[0130] In step (1420), the computer device (100) may generate a target matrix by decomposing the calculated sum. For example, the computer device (100) may generate the target matrix through singular value decomposition of the calculated sum, wherein the target matrix may include a square matrix composed of eigenvectors corresponding to the smallest singular value.

[0131] In step (1430), the computer device (100) may select neurons to be removed from attention based on the similarity of matrix products with the target matrix. For example, the computer device (100) may select neurons in the query layer and neurons in the key layer corresponding to the matrix product with the highest similarity with the target matrix as neurons to be removed through pruning. Here, the similarity may include cosine similarity. In this case, the computer device (100) may select neurons to be removed by determining whether to prune neurons in the query layer and neurons in the key layer used in the calculation of each of the matrix products based on the similarity of the matrix products.

[0132] In step (1440), the computer device (100) may remove neurons selected from the attention. For example, the computer device (100) may remove neurons of the same index in the query layer and the key layer of the attention together.

[0133] In step (1450), the computer device (100) may repeatedly perform steps (1410) to (1440). For example, the computer device (100) may repeatedly perform steps (1410) to (1440) a preset number of times. Here, the preset number of times may include a number of times determined based on the performance of the neural network model including the attention from which the selected neuron has been removed. As another example, the computer device (100) may repeatedly perform steps (1410) to (1440) while the performance of the lightweight neural network model satisfies a performance range set by the user without retraining the lightweight neural network model. As another example, the computer device (100) may repeatedly perform the attention reduction until the performance is similar to the performance of the neural network model requested by the user (for example, the performance with a difference less than a threshold value). A lightweight neural network model may be generated through attention configured through the removal of neurons one or more times.

[0134] The inference system according to embodiments of the present invention can be implemented by at least one computer device, and the inference method according to embodiments of the present invention can be performed by at least one computer device included in the inference system. The computer device can have a computer program according to an embodiment of the present invention installed and run, and the computer device can perform the inference method according to embodiments of the present invention under the control of the run computer program. The above-described computer program can be stored in a computer-readable recording medium so as to be coupled to the computer device and cause the computer device to execute the inference method.

[0135] FIG. 15 is a flowchart illustrating another example of an inference method according to an embodiment of the present invention. The inference method according to the present embodiment can be performed by the computer device (100) described through FIG. 1. At this time, the processor (120) of the computer device (100) can be implemented to execute control instructions according to the code of the operating system included in the memory (110) or the code of at least one computer program. Here, the processor (120) can control the computer device (100) so that the computer device (100) performs steps (1510 to 1530) included in the method of FIG. 15 according to the control instructions provided by the code stored in the computer device (100).

[0136] For example, a transformer model including lightweight self-attention according to the embodiments described above may be provided, and a user who receives the transformer model may process inference on data using the provided transformer model.

[0137] In step (1510), the computer device (100) may load a neural network model including lightweight attention by removing neurons selected based on the similarity between the target matrix generated by decomposing the sum of matrix products of outputs from neurons of the same index in the query layer and the key layer and the matrix products. For example, the computer device (100) may load program code for the neural network model into the memory (110).

[0138] Here, the target matrix may include a square matrix composed of eigenvectors corresponding to the smallest singular value obtained through singular value decomposition of the sum described above. Furthermore, attention can be reduced by removing neurons in the query layer and the key layer corresponding to the matrix product with the highest similarity between the target matrices. This reduction in self-attention has been previously described in detail.

[0139] At step (1520), the computer device (100) may receive data. Here, the data may be input into a neural network model for inference using the neural network model. Depending on the embodiment, the order of steps (1510) and (1520) may be changed.

[0140] In step (1530), the computer device (100) may process inferences on the input data using a neural network model. For example, the computer device (100) may input the data received in step (1520) into a neural network model to process inferences on the data.

[0141] As described above, according to embodiments of the present invention, a method and system for reducing the weight of a neural network including attention by considering the connection relationship can be provided. In addition, a method and system for reducing the weight by minimizing the loss of attention scores can be provided. In addition, a neural network model including the reduced attention can be provided through the above-mentioned method and system for reducing the weight. In addition, an inference method and system using the neural network model including the reduced attention can be provided. In addition, a method and system for reducing the weight of a neural network by considering the connection relationship of attention can complement the limitations of existing neuron pruning, which does not affect speed and model size, while preserving the advantages of neuron pruning. In addition, the learning or inference costs of very large models, such as ChatGPT, can be reduced, and a more powerful model can be created than existing models by acting as a regularization term. In addition, by presenting neuron-level sparsity that takes into account the characteristics of the neural network, a lightweight method and system that can preserve existing performance to a certain extent without learning are provided, thereby eliminating the need for relearning or minimizing relearning.

[0142] The systems or devices described above may be implemented as hardware components, or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0143] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0144] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0145] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0146] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for reducing a neural network model including attention by a computer device including at least one processor, A step of receiving a neural network model including attention; and A step of generating a lightweight neural network model by removing neurons of the same index of the query layer and the key layer including the attention by at least one processor. A lightweight method including:

2. In paragraph 1, The step of creating the above lightweight neural network model is: A step of selecting neurons to be removed from the query layer or the key layer; A step of removing the selected neurons from the query layer or the key layer; and A step of removing a neuron of the key layer having the same index as the selected neuron from the key layer when the neuron to be removed is selected from the query layer, and removing a neuron of the query layer having the same index as the selected neuron from the query layer when the neuron to be removed is selected from the key layer. A lightweight method characterized by including:

3. In paragraph 1, Further comprising a step of removing at least one neuron of the value layer included in the above attention, A lightweight method, characterized in that the number and index of neurons removed from the above value layer are determined independently from the neurons removed from the above query layer and key layer.

4. In paragraph 1, The step of creating the above lightweight neural network model is: A lightweight method characterized in that, when the above attention is multi-head self-attention (MHSA), the lightweight neural network model is generated by removing the same amount of neurons for each head included in the multi-head self-attention.

5. In paragraph 1, The above attention is multi-head self-attention, and the output of each of the query layer, key layer, and value layer of the multi-head self-attention is reshaped into a matrix having the number of heads and the number of neurons of each head as rows and columns, The step of creating the above lightweight neural network model is: A step of determining the index by considering the connection relationship with the operator after the reshaping; and A step of removing a neuron corresponding to the above-determined index A lightweight method characterized by including:

6. In paragraph 5, The step of creating the above lightweight neural network model is: A step of adjusting the reshaping layer and the operator that performs the reshaping based on the determined index. A lightweight method characterized by further including:

7. In paragraph 5, The step of determining the above index is: a step of determining one of multiple optimization policies; and A step of determining the index according to the above-determined policy A lightweight method characterized by including:

8. In paragraph 7, The above multiple optimization policies include two or more of (1) a None policy that removes the same number of neurons for each head of the multi-head self-attention, (2) a Sum policy that determines an index for removing a neuron based on the sum of the importance of neurons of the same index of heads included in the multi-head self-attention, (3) an Average policy that determines an index for removing a neuron based on the average of the importance of neurons of the same index of the heads, and (4) a Count policy that counts the number of neurons to be removed from the heads based on importance by index and determines an index for removing a neuron based on the number of neurons counted by index. A lightweight method characterized by .

9. In paragraph 1, The step of creating the above lightweight neural network model is: A step of calculating the sum of matrix products of outputs from neurons of the same index of the query layer and the key layer; A step of generating a target matrix by decomposing the calculated sum; A step of selecting a neuron to be removed from the attention based on the similarity of the matrix products with the target matrix; and A step of generating a lightweight neural network model by removing the selected neurons from the above attention. A lightweight method characterized by including:

10. In paragraph 9, The step of generating the above target matrix is: A lightweight method characterized by generating the target matrix through singular value decomposition of the calculated sum.

11. In paragraph 9, A lightweight method characterized in that the above target matrix is ​​composed of singular vectors corresponding to the smallest singular value.

12. In paragraph 9, The step of selecting the above neurons is: A lightweight method characterized in that the method selects the neuron to be removed by determining whether to prune the neurons of the query layer and the neurons of the key layer used in the calculation of each of the matrix products based on the similarity of the matrix products.

13. In paragraph 12, The step of selecting the above neurons is: A lightweight method characterized in that the neurons of the query layer and the neurons of the key layer corresponding to the matrix product with the highest similarity to the target matrix are selected as neurons to be removed through pruning.

14. In paragraph 9, A lightweight method characterized in that neurons of the same index in the query layer and the key layer are removed.

15. In paragraph 9, The step of calculating the sum, the step of generating the target matrix, the step of selecting the neuron, and the step of generating the lightweight neural network model are: A lightweight method characterized in that it is repeatedly performed a number of times determined based on the performance of the lightweight neural network model or while the performance of the lightweight neural network model satisfies the performance range set by the user without retraining the lightweight neural network model.

16. A computer-readable recording medium having recorded thereon a program for executing the method of paragraph 1 on a computer device.

17. A method for reasoning in a computer device including at least one processor, A step of processing inference on input data using a neural network model that includes attention with neurons of the same index in the query layer and the key layer removed. An inference method including .

18. In paragraph 17, An inference method characterized in that, when the above attention is multi-head self-attention (MHSA), the same amount of filters are removed for each head included in the multi-head self-attention.

19. In paragraph 17, The above attention is an inference method characterized in that it is lightweight by removing neurons selected based on the similarity between the matrix products with the target matrix generated by decomposing the sum between the matrix products of the outputs from the neurons of the same index of the query layer and the key layer.

20. At least one processor implemented to execute instructions readable by a computer device. Including, By at least one processor, Receive a neural network model that includes attention, Generating a lightweight neural network model by removing neurons of the same index in the query layer and key layer that the above attention includes. A computer device characterized by:

Images