Attention-based networks comprising softmax for memory-efficient training and method thereof

Abstract

The present invention relates to an attention-based network and a networking method including a softmax for memory-efficient learning, which can reduce the memory storage space used for activation by efficiently utilizing memory used in artificial intelligence learning utilizing softmax and by reducing and partially storing the results of the softmax. The attention-based network including a softmax for memory-efficient learning according to the present invention comprises an attention-based model that receives input data and outputs an attention-based artificial intelligence solution, an optimization unit that optimizes the artificial intelligence solution based on the output of the attention-based model, and a memory that stores information of the input data, the attention-based model, and the optimization unit.

Description

Technology Field

[0001] The present invention relates to an attention-based network and a networking method including a softmax for memory-efficient learning, and more specifically, to efficiently utilize memory used in artificial intelligence learning.

[0002] In addition, the present invention relates to an attention-based network and a networking method including a softmax for memory-efficient learning that can reduce the memory storage space used for activation by reducing and storing the result of the softmax. Background Technology

[0003] Deep learning refers to Deep Neural Network (DNN) algorithms and methods for learning them. Initially, its utility was low due to the disadvantages of being time-consuming and overfitting with training data.

[0004] However, with the advent of GPUs (Graphic Processing Units) capable of parallel processing and new algorithms, image object recognition technology has been dramatically improved, and currently, the error rate has been significantly reduced to a level very close to the human cognitive error rate.

[0005] Meanwhile, deep learning has the advantage of being able to express the result of an existing operation as a probability value between 0 and 1 using the softmax function. However, since the operation of softmax is very complex and occupies a lot of memory, research on efficiently performing the operation of the softmax layer has been ongoing.

[0006] For example, Korean Patent Publication No. 10-2020-0184869 discloses a method of operation using a successive approximation method for the softmax layer of a Convolutional Neural Network (CNN) algorithm, which is configured to receive N input values ​​to derive the output value of the softmax layer, perform exponential operations on the input values ​​using a mathematical formula, and perform sequential exponential operations using a successive approximation method of Loop Up Table (LUT) data included in the Exponential (EXP) block within the operation circuit.

[0007] However, even in this case, there is a disadvantage in that memory is not used efficiently because a method to reduce and store the softmax results is not considered. Prior art literature

[0008] Republic of Korea Published Patent Application No. 10-2020-0184869 (December 28, 2020) The problem to be solved

[0009] The objective of the present invention is to provide an attention-based network and a networking method including a softmax for memory-efficient learning that can efficiently utilize memory used in artificial intelligence learning utilizing softmax.

[0010] Another objective of the present invention is to provide an attention-based network and a networking method including a softmax for memory-efficient learning that can reduce the memory storage space used for activation by reducing and partially storing the results of the softmax. means of solving the problem

[0011] An attention-based network including a softmax for memory-efficient learning according to the present invention may include an attention-based model that receives input data and outputs an attention-based artificial intelligence solution, an optimization unit that optimizes the artificial intelligence solution based on the output of the attention-based model, and a memory that stores information of the input data, the attention-based model, and the optimization unit.

[0012] Here, the artificial intelligence solution may include any one of text classification, sentiment analysis, and machine translation.

[0013] In addition, attention-based models may include a linear computation layer that performs linear transformations.

[0014] Here, the attention-based model may include an attention compute layer that extracts attention.

[0015] Additionally, the attention compute layer may include a softmax compute module that performs softmax compute.

[0016] Here, the attention-based model may include other layers that perform processing excluding the linear computation layer and the attention computation layer.

[0017] In addition, the memory may include an input data storage area that stores information about the input data.

[0018] Here, the memory may include a model parameter storage area that stores various parameters among the information of the attention-based model.

[0019] In addition, the memory may include an activation memory area that stores various activation information among the information of the attention-based model.

[0020] Here, the memory may include an optimization state storage area that stores state information among the information of the optimization unit.

[0021] Additionally, the activation memory area may include a softmax activation memory section that stores activation information of the softmax calculation module.

[0022] Here, the activation memory area may include other activation memory sections that store activation information of other layers.

[0023] In addition, the softmax calculation module may include a forward propagation submodule for processing the forward propagation process and a backpropagation submodule for processing the backpropagation process.

[0024] Here, the forward propagation process submodule may include a softmax calculation function that performs softmax calculation.

[0025] Additionally, the forward propagation process submodule may include a compression parameter calculation function that compresses the output of the softmax calculation function and stores the compressed information in the softmax activation memory section.

[0026] Here, the backpropagation process submodule may include an estimate calculation function that calculates an estimate of the output of a softmax calculation function based on compressed information stored in a softmax activation memory section.

[0027] Additionally, the backpropagation process submodule may include a gradient matrix calculation function that predicts the input of the softmax calculation function using the gradient matrix on the output of the estimate calculation function.

[0028] Meanwhile, the backpropagation process submodule may include a weight update calculation function that updates the weights of variables within the attention-based model based on the output of the gradient matrix calculation function.

[0029] An attention-based networking method including a softmax for memory-efficient learning according to another embodiment of the present invention may include an attention-based modeling step that receives input data from an attention-based model and outputs an attention-based artificial intelligence solution, and an optimization step that optimizes the artificial intelligence solution based on the output of the attention-based model in an optimization unit.

[0030] Here, the artificial intelligence solution may include any one of text classification, sentiment analysis, and machine translation.

[0031] In addition, the attention-based modeling step may include a step of performing a linear transformation in the linear computation layer within the attention-based model.

[0032] Here, the attention-based modeling step may include a step of extracting attention from the attention computation layer within the attention-based model.

[0033] In addition, the attention-based modeling step may include a step of performing softmax calculations in a softmax calculation module within the attention calculation layer.

[0034] Here, the attention-based modeling step may include a step of performing processing in other layers within the attention-based model, excluding the linear computation layer and the attention computation layer.

[0035] In addition, the attention-based modeling step may include a step of storing information about the input data in the attention-based model in the input data storage area in memory.

[0036] Here, the attention-based modeling step may include a step of storing various parameters among the information of the attention-based model in the attention-based model in a model parameter storage area in memory.

[0037] In addition, the attention-based modeling step may include a step of storing various activation information among the information of the attention-based model in the attention-based model in the activation memory area within the memory.

[0038] Here, the optimization step may include a step of storing state information among the information of the optimization unit in the optimization unit in the optimization state storage area in memory.

[0039] Additionally, the attention-based modeling step may include a step of storing the activation information of the softmax calculation module in the softmax activation memory section within the activation memory area.

[0040] Here, the attention-based modeling step may include a step of storing the activation information of the other layer in the other activation memory section within the activation memory area.

[0041] Additionally, the attention-based modeling step may include a forward propagation calculation step that processes the forward propagation process in a forward propagation submodule within the softmax calculation module, and a backpropagation calculation step that processes the backpropagation process in a backpropagation submodule within the softmax calculation module.

[0042] Here, the forward propagation process calculation step may include a step of performing a softmax calculation in a softmax calculation function within a forward propagation process submodule.

[0043] Additionally, the forward propagation process calculation step may include a step of compressing the output of the softmax calculation function in the compression parameter calculation function within the forward propagation process submodule and storing the compressed information in the softmax activation memory section.

[0044] Here, the backpropagation process calculation step may include a step of calculating an estimate of the output of a softmax calculation function based on compressed information stored in a softmax activation memory section within an estimate calculation function in a backpropagation process submodule.

[0045] Additionally, the backpropagation process calculation step may include a step of predicting the input of a softmax calculation function using a gradient matrix on the output of an estimate calculation function within a gradient matrix calculation function in the backpropagation process submodule.

[0046] Here, the backpropagation calculation step may include a step of updating the weights of variables within the attention-based model based on the output of the gradient matrix calculation function in the weight update calculation function within the backpropagation submodule. Effects of the invention

[0047] The attention-based network and networking method including a softmax for memory-efficient learning according to the present invention has the advantage of efficiently managing memory used in artificial intelligence learning utilizing softmax.

[0048] In addition, the attention-based network and networking method including a softmax for memory-efficient learning according to the present invention has the advantage of reducing the memory storage space used for activation by reducing and partially storing the result of the softmax. Brief explanation of the drawing

[0049] FIG. 1 is a schematic diagram showing an attention-based network including a softmax for memory-efficient learning according to one embodiment of the present invention. Figure 2 is a detailed diagram of the softmax calculation module of Figure 1. Figure 3 is a diagram illustrating the function of the softmax calculation module of Figure 1. FIG. 4 is a flowchart illustrating an attention-based networking method including a softmax for memory-efficient learning according to an embodiment of the present invention. Figure 5 is a flowchart showing the attention-based modeling steps of Figure 4 in detail. Specific details for implementing the invention

[0050] Hereinafter, specific embodiments for carrying out the present invention will be described with reference to the attached drawings.

[0051] In describing the present invention, terms such as first, second, etc. may be used to describe various components, but the components may not be limited by the terms. The terms are intended solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0052] When it is mentioned that a component is connected to or coupled with another component, it can be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0053] The terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions may include plural expressions unless the context clearly indicates otherwise.

[0054] In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not excluding in advance the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0055] In addition, the shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

[0057] Hereinafter, an attention-based network including a softmax for memory-efficient learning and a networking method according to the present invention will be described in detail with reference to the attached drawings.

[0058] During the training of a Deep Neural Network (DNN), the elements occupying memory generally include the data to be learned, parameters of the artificial intelligence model, state information stored for optimizer state weight updates, and elements stored in activation memory; the elements stored in activation memory are saved during the forward propagation process, used during the backpropagation process, and then erased.

[0059] An attention-based network including a softmax for memory-efficient learning according to the present invention stores only some values ​​of the softmax output (Forward pass), and when the output needs to be used again (Backward pass), replaces the unstored values ​​with values ​​approximated from the stored values.

[0060] Here, softmax is a part that must be included to obtain the attention score in the attention mechanism, and likewise, it is an activation function that must be included in deep learning models that use the attention mechanism.

[0061] In addition, when a deep learning model is trained, various values ​​are stored in hardware memory, most notably 'input data', 'model parameters', 'activation memory', and 'optimizer state'.

[0062] 'Input data' refers to the goal to be learned, 'model parameters' are the elements that constitute the model and serve as the medium for learning, and 'activation memory' and 'optimizer state' refer to values ​​that store the elements used before updating the 'model parameters'.

[0063] The largest proportion of these is 'activation memory,' and it becomes larger for models that need to learn a large amount of information. Due to these elements that must be stored, a large amount of memory resources is required to train deep learning models.

[0064] According to the attention-based network including softmax for memory-efficient learning of the present invention, this problem can be resolved by optimizing the activation memory, which accounts for the largest proportion.

[0065] In addition, the attention-based network including a softmax for memory-efficient learning according to the present invention is suitable for attention mechanisms that form the basis of large-scale language processing models using high-capacity 'activation memory'.

[0066] Large Language Models are becoming larger in scale to learn large amounts of information, and representative examples include GPT-3, MT-NLG, BERT, ALBERT, and XLNet.

[0067] These models are becoming too large for small-scale users to learn, and since they are based on attention mechanisms, the activation memory required for softmax outputs is also increasing, necessitating efficiency improvements.

[0068] In other words, depending on the artificial intelligence solution, the use of activation memory is required to be 1.5 to 3.5 times greater than the memory used for storing parameters and state; therefore, minimizing the capacity of activation memory is important for efficient memory usage.

[0069] According to the attention-based network including a softmax for memory-efficient learning of the present invention, the memory storage space used for such activation can be reduced.

[0070] Hereinafter, with reference to FIGS. 1 to 3, an attention-based network including a softmax for memory-efficient learning according to an embodiment of the present invention will be described.

[0071] FIG. 1 is a schematic diagram showing an attention-based network including a softmax for memory-efficient learning according to an embodiment of the present invention, and FIG. 2 and FIG. 3 are detailed drawings for explaining FIG. 1 in detail.

[0072] First, referring to FIG. 1, an attention-based network including a softmax for memory-efficient learning according to one embodiment of the present invention comprises an attention-based model (200) that receives input data (100) and outputs an attention-based artificial intelligence solution, an optimization unit (300) that optimizes artificial intelligence solutions such as text classification, sentiment analysis, and machine translation based on the output of the attention-based model (200), and a memory (400) that stores information of the input data (100), the attention-based model (200), and the optimization unit (300).

[0073] Here, the attention-based model (200) consists of a linear computation layer (210) that performs a linear transformation, an attention computation layer (220) that extracts attention, and other layers (230) that perform processing excluding the linear computation layer (210) and the attention computation layer (220).

[0074] At this time, the attention calculation layer (220) includes a softmax calculation module (221) that performs softmax calculation, and the forward propagation and backpropagation processes are performed in the softmax calculation module (221), and the forward propagation and backpropagation processes are explained in detail in FIGS. 2 and FIGS. 3.

[0075] Meanwhile, the memory (400) is composed of an input data storage area (410) that stores information of input data (100), a model parameter storage area (420) that stores various parameters among the information of an attention-based model (200), an activation memory area (430) that stores various activation information among the information of an attention-based model (200), and an optimization state storage area (440) that stores state information among the information of an optimization unit (300). At this time, the activation memory area (430) includes a softmax activation memory section (431) that stores activation information of a softmax calculation module (221) and an other activation memory section (432) that stores activation information of other layers (230).

[0076] In the present invention, the storage capacity of the softmax activation memory section (431) used in the forward and back-propagation processes in the softmax calculation module (221) can be reduced, and this is explained in detail in FIGS. 2 and FIGS. 3 below.

[0078] FIG. 2 is a detailed drawing of the softmax calculation module (221) of FIG. 1.

[0079] As can be seen in FIG. 2, the softmax calculation module (221) includes a forward pass submodule (500) that processes the forward pass and a backward pass submodule (600) that processes the backward pass.

[0080] Here, the forward propagation process submodule (500) includes a softmax calculation function (510) that performs a softmax calculation and a compression parameter calculation function (520) that compresses the output of the softmax calculation function (510) and stores the compressed information in a softmax activation memory section (431).

[0081] Meanwhile, the backpropagation process submodule (600) consists of an estimate calculation function (610) that calculates an estimate of the output of a softmax calculation function (510) based on compressed information stored in a softmax activation memory section (431), a gradient matrix calculation function (620) that predicts the input of a softmax calculation function (510) by applying a gradient matrix to the output of the estimate calculation function (610), and a weight update calculation function (630) that updates the weights.

[0082] Here, the softmax calculation function (510) used in the forward propagation process is defined by the following [Equation 1], and is suitable for application in classification problems as it is made more probabilistic and normalized.

[0083]

[0084] Here, is the number of input data, is input data, is output data.

[0086] Additionally, the compression parameter calculation function (520) is defined by the following [Equation 2], wherein a part of the output of the compression parameter calculation function (520) is stored as is, and the remaining part is a parameter capable of estimating data values. and It is to add.

[0087]

[0088] Here, is the smallest value among the stored top values, is the largest value among the unsaved values, The largest value among the stored sub-values, is the number of softmax outputs.

[0090] also, and In this case, the minimum value among the upper values ​​and the maximum value among the lower values ​​adjacent to the middle part were used to approximate the unstored middle part of the outputs sorted in descending order.

[0091] Meanwhile, since the input size of the above [Mathematical Equation 2] varies depending on the data, it has a feature that allows setting constraints and imposing limits (Range value), a feature that imposes a limit so that the sum of the approximated values ​​from the expression is 1 (Normalization Factor), and class There is a feature (Rate value) that considers the decreasing trend of approximated values ​​by taking into account some of the differences.

[0092] More specifically, since the input size varies depending on the data, quantitatively If we take the value of as an integer, a problem may arise because the approximation range increases differently depending on the input. To solve this, we set the range to only x with a slope orthogonal to the slope of 0 on the x-axis.

[0093] In addition, to reduce the error between the approximated value and the original value and to satisfy the basic properties of softmax, the total sum of the values ​​extracted from the approximation function and the stored values ​​is set to 1.

[0094] one side, class Because it is difficult to approximate all intermediate values ​​that are not stored in the alone In other words, it is used to predict the trend of values ​​to be approximated by considering the difference between the largest unstored value and the upper minimum value among the stored values.

[0095] Meanwhile, the estimate calculation function (610) used in the backpropagation process is defined by the following [Equation 3], in which the output of the softmax calculation function (510) is estimated in order based on the data stored as the output of the compression parameter calculation function (520), as explained in detail in FIG. 3.

[0096]

[0097] Here, Number of estimated data points = Number of input data points -

[0099] On the other hand, the exponential distribution has the advantage of being able to adequately represent the descending variation of the softmax output, having a slope that changes rapidly from the parameters, and being able to calculate efficiently using a small number of parameters.

[0100] Generally, exponential distributions have the characteristic of being difficult to fine-tune because they have only one parameter; however, the attention-based network including a softmax for memory-efficient learning according to the present invention determines the parameter determined from the stored values ​​using two values, i.e. and It has the advantage of being easy to fine-tune as it is subdivided.

[0101] The result of the estimate calculation function (610) is expressed by the following [Equation 4] and is provided to the gradient matrix calculation function (620) as basic data for weight updates.

[0102]

[0103] Here, is the highest level , is the lowest , silver The number of times to use as is, is the number of input data, is estimated data.

[0105] The above [Equation 4] sorts the values ​​of the softmax output in order of size and Top of the outputs The values ​​and sub-values It shows the process of selecting dog values. The values ​​selected in this way The values ​​are stored in the activation memory, that is, during the forward pass process, and The output of the dog is not stored.

[0106] Here, In this case, the value can be adjusted to a value desired by the user, and this can be selected by considering the characteristics of the model and data. When the value stored in this way is reused for the gradient update during the backpropagation process, an approximate value similar to the original value can be obtained, which is explained in detail in Fig. 3.

[0107] Additionally, the gradient matrix calculation function (620) is defined by [Equation 5] below, and the weights of the variables used in the attention-based model (200) are updated as a result of the gradient matrix calculation function (620).

[0108]

[0109] Here, is estimated , is input data, is the number of input data.

[0111] Here, the gradient matrix calculation function (620) is a gradient matrix constructed from the softmax output to learn the model's parameters. This is used during the backward pass, that is, when updating the model's parameters. During this process, the softmax output stored in the forward pass is constructed in the form of [Equation 5] above, and the output of the softmax calculation function (510) is stored to perform this process.

[0113] FIG. 3 is a diagram illustrating the function of the softmax calculation module (221) of FIG. 1. As can be seen in FIG. 3, Equation 4, which is the result of the estimate calculation function (610), can be expressed as a part of the result of the compression parameter calculation function (520) and a part of the predicted result of the compression parameter calculation function (520).

[0114] The basic property of softmax is that the sum of the outputs is 1, and the sum being 1 means that if a specific value increases, the remaining values ​​become relatively smaller.

[0115] In addition, the attention mechanism learns in a direction where specific input values ​​increase; that is, among various inputs, the input with the most appropriate association takes on a high value.

[0116] If the softmax outputs are sorted in order of size based on this, a sharp difference may occur because the next largest value is relatively small when a specific value is high, or a difference of moderate size may occur if no single value has a prominent size.

[0117] In this case, if the largest and smallest values ​​are known, the intermediate values ​​can be aligned with a somewhat constant slope, and can also be approximated from an exponential distribution, where the slope can change rapidly from a specific value.

[0118] That is, part of the result of the compression parameter calculation function (520) uses the top element (D100) and the bottom element (D300), and between the top element (D100) and the bottom element (D300), uses the value predicted by the predicted element (D200).

[0119] At this time, if we compare the softmax output (D400), which is the result of the compression parameter calculation function (520) within the interval of the predicted element (D200), with the predicted softmax output (D500), which is the predicted result, we can see that they are very similar to each other.

[0120] That is, the attention-based network including a softmax for memory-efficient learning according to the present invention has the advantage of reducing the memory storage space used for activation by reducing and partially storing the result of the softmax.

[0122] FIG. 4 is a flowchart illustrating an attention-based networking method including a softmax for memory-efficient learning according to an embodiment of the present invention, and FIG. 5 is a detailed flowchart for explaining FIG. 4 in detail.

[0123] Hereinafter, an attention-based networking method including a softmax for memory-efficient learning according to an embodiment of the present invention will be described with reference to FIGS. 4 and 5.

[0124] First, referring to FIG. 4, an attention-based networking method including a softmax for memory-efficient learning according to the present invention comprises an attention-based modeling step (S200) that receives input data (100) in an attention-based model (200) and outputs an attention-based artificial intelligence solution, and an optimization step (S300) that optimizes the artificial intelligence solution based on the output of the attention-based model (200) in an optimization unit (300).

[0125] Here, the artificial intelligence solution includes any one of text classification, sentiment analysis, and machine translation.

[0126] Additionally, in the attention-based modeling step (S200), a linear transformation is performed in the linear computation layer (210) within the attention-based model (200), and attention is extracted in the attention computation layer (220) within the attention-based model (200). At this time, a softmax calculation is performed in the softmax computation module (221) within the attention computation layer (220), and processing excluding the linear computation layer (210) and the attention computation layer (220) is performed in the other layer (230) within the attention-based model (200).

[0127] Here, the forward propagation and backpropagation processes are performed in the softmax calculation module (221), and the forward propagation and backpropagation processes are explained in detail in FIG. 5.

[0128] Meanwhile, information used in the input data (100), attention-based model (200), and optimization unit (300) is stored in memory (400). In the attention-based modeling step (S200), information of the input data (100) in the attention-based model (200) is stored in the input data storage area (410) in memory (400), and various parameters among the information of the attention-based model (200) in the attention-based model (200) are stored in the model parameter storage area (420) in memory (400), while various activation information among the information of the attention-based model (200) in the attention-based model (200) is stored in the activation memory area (430) in memory (400).

[0129] Meanwhile, in the optimization step (S300), the state information among the information of the optimization unit (300) in the optimization unit (300) is stored in the optimization state storage area (440) in the memory (400).

[0130] More specifically, in the attention-based modeling step (S200), the activation information of the softmax calculation module (221) is stored in the softmax activation memory section (431) within the activation memory area (430) in the softmax calculation module (221), and the activation information of the other layer (230) is stored in the other activation memory section (432) within the activation memory area (430) in the other layer (230).

[0131] Here, the storage capacity of the softmax activation memory section (431) used in the forward and back-propagation processes in the softmax calculation module (221) can be reduced, and is explained in detail below in FIG. 5.

[0133] FIG. 5 is a flowchart showing the attention-based modeling step (S200) of FIG. 4 in detail.

[0134] As can be seen in FIG. 5, the attention-based modeling step (S200) includes a forward propagation process calculation step (S500) that processes the forward propagation process in a forward propagation process submodule (500) within a softmax calculation module (221), and a backpropagation process calculation step (S600) that processes the backpropagation process in a backpropagation process submodule (600) within a softmax calculation module (221).

[0135] Here, in the forward propagation process calculation step (S500), a softmax calculation is performed in the softmax calculation function (510) within the forward propagation process submodule (500), and the output of the softmax calculation function (510) is compressed in the compression parameter calculation function (520) within the forward propagation process submodule (500), and the compressed information is stored in the softmax activation memory section (431).

[0136] That is, the softmax calculation function (510) used in the forward propagation process is defined by the [Mathematical Formula 1] described above, which makes it more probabilistic and normalized, making it suitable for application in classification problems.

[0137] Additionally, the compression parameter calculation function (520) is defined by the [Mathematical Formula 2] described above, wherein a part of the output of the compression parameter calculation function (520) is stored as is, and the remaining part is a parameter capable of estimating data values. and It involves adding [something], and since this has already been explained in Figures 2 and 3, a detailed explanation is omitted.

[0138] Meanwhile, in the backpropagation process calculation step (S600), the estimate of the output of the softmax calculation function (510) is calculated based on the compressed information stored in the softmax activation memory section (431) in the estimate calculation function (610) within the backpropagation process submodule (600), and the input of the softmax calculation function (510) is predicted using the gradient matrix with respect to the output of the estimate calculation function (610) in the gradient matrix calculation function (620) within the backpropagation process submodule (600), while the weights of the variables in the attention-based model (200) are updated based on the output of the gradient matrix calculation function (620) in the weight update calculation function (630) within the backpropagation process submodule (600).

[0139] That is, the estimate calculation function (610) used in the backpropagation process is defined by the [Mathematical Formula 3] described above, and at this time, the output of the softmax calculation function (510) is estimated in order based on the data stored as the output of the compression parameter calculation function (520).

[0140] Here, the result of the estimate calculation function (610) is expressed by the above-described [Equation 4] and is provided to the gradient matrix calculation function (620) as basic data for weight updates.

[0141] In addition, the gradient matrix calculation function (620) is defined by the above-described [Equation 5], and the weights of the variables used in the attention-based model (200) are updated as a result of the gradient matrix calculation function (620), and a detailed explanation is omitted as it has already been explained in FIG. 2 and FIG. 3.

[0142] The attention-based networking method including a softmax for memory-efficient learning according to the present invention has the advantage of reducing the memory storage space used for activation by reducing and partially storing the result of the softmax.

[0144] As described above, the attention-based network and networking method including a softmax for memory-efficient learning according to the present invention has the advantage of efficiently utilizing memory used in artificial intelligence learning utilizing softmax, and has the advantage of reducing the memory storage space used for activation by reducing and partially storing the results of the softmax.

[0146] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of the invention.

[0147] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0148] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that, based on design priorities, the specific order or hierarchy of steps in the processes may be rearranged within the scope of the invention. The appended method claims provide various step elements in a sample order, but do not imply limitation to the specific order or hierarchy presented.

[0149] The description of the presented embodiments is provided so that any person skilled in the art may use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present invention. Thus, the present invention is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

Claims

Claim 1 An attention-based model that receives input data and outputs an attention-based artificial intelligence solution; an optimization unit that optimizes the artificial intelligence solution based on the output of the attention-based model; and a memory that stores information of the input data, the attention-based model, and the optimization unit; wherein the attention-based model includes a linear computation layer that performs a linear transformation; the attention-based model includes an attention computation layer that extracts the attention; the attention computation layer includes a softmax computation module that performs softmax computation; and the attention-based model includes an other layer that performs processing excluding the linear computation layer and the attention computation layer; the memory includes an activation memory area that stores various activation information among the information of the attention-based model; the activation memory area includes a softmax activation memory section that stores activation information of the softmax computation module; and the softmax computation module includes a forward propagation process submodule that processes a forward propagation process; An attention-based network device including a softmax for memory-efficient learning, comprising: a backpropagation process submodule for processing a backpropagation process; wherein the forward propagation process submodule includes a softmax calculation function that performs a softmax calculation; the forward propagation process submodule includes a compression parameter calculation function that compresses the output of the softmax calculation function and stores the compressed information in the softmax activation memory section; and the backpropagation process submodule includes an estimate calculation function that calculates an estimate of the output of the softmax calculation function based on the compressed information stored in the softmax activation memory section. Claim 2 An attention-based network device comprising a softmax for memory-efficient learning, characterized in that, in claim 1, the artificial intelligence solution includes any one of text classification, sentiment analysis, and machine translation. Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 An attention-based network device including a softmax for memory-efficient learning, characterized in that, in claim 1, the memory includes an input data storage area for storing information of the input data. Claim 8 An attention-based network device including a softmax for memory-efficient learning, characterized in that, in claim 1, the memory includes a model parameter storage area that stores various parameters among the information of the attention-based model. Claim 9 delete Claim 10 An attention-based network device including a softmax for memory-efficient learning, characterized in that, in claim 1, the memory includes an optimization state storage area that stores state information among the information of the optimization unit. Claim 11 delete Claim 12 An attention-based network device including a softmax for memory-efficient learning, characterized in that, in claim 1, the activation memory region includes an other activation memory section that stores activation information of the other layer. Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 An attention-based network device including a softmax for memory-efficient learning, characterized in that, in claim 1, the backpropagation process submodule includes a gradient matrix calculation function that predicts the input of the softmax calculation function using a gradient matrix on the output of the estimate calculation function. Claim 18 An attention-based network device for memory-efficient learning comprising a softmax, characterized in that, in claim 17, the backpropagation process submodule includes a weight update calculation function that updates the weights of variables within the attention-based model based on the output of the gradient matrix calculation function. Claim 19 In an attention-based networking method performed by an attention-based network device, an attention-based modeling step that receives input data from an attention-based model and outputs an attention-based artificial intelligence solution; and an optimization step for optimizing the artificial intelligence solution based on the output of the attention-based model in the optimization unit; wherein the attention-based modeling step includes a step of performing a linear transformation in a linear computation layer within the attention-based model, wherein the attention-based modeling step includes a step of extracting the attention in an attention computation layer within the attention-based model, wherein the attention-based modeling step includes a step of performing a softmax calculation in a softmax computation module within the attention computation layer, wherein the attention-based modeling step includes a step of performing processing excluding the linear computation layer and the attention computation layer in other layers within the attention-based model, wherein the attention-based modeling step includes a step of storing various activation information among the information of the attention-based model in the attention-based model in an activation memory area within memory, wherein the attention-based modeling step includes a step of storing the activation information of the softmax computation module in the softmax computation module in a softmax activation memory section within the activation memory area, and wherein the attention-based The modeling step comprises: a forward propagation process calculation step for processing the forward propagation process in a forward propagation process submodule within the softmax calculation module; and a backpropagation process calculation step for processing the backpropagation process in a backpropagation process submodule within the softmax calculation module;An attention-based networking method including a softmax for memory-efficient learning, characterized in that the forward propagation calculation step includes a step of performing a softmax calculation in a softmax calculation function within the forward propagation submodule, the forward propagation calculation step includes a step of compressing the output of the softmax calculation function in a compression parameter calculation function within the forward propagation submodule and storing the compressed information in the softmax activation memory section, and the backpropagation calculation step includes a step of calculating an estimate of the output of the softmax calculation function based on the compressed information stored in the softmax activation memory section in an estimate calculation function within the backpropagation submodule. Claim 20 An attention-based networking method comprising a softmax for memory-efficient learning, characterized in that, in claim 19, the artificial intelligence solution comprises any one of text classification, sentiment analysis, and machine translation. Claim 21 delete Claim 22 delete Claim 23 delete Claim 24 delete Claim 25 An attention-based networking method including a softmax for memory-efficient learning, characterized in that, in the attention-based modeling step of claim 19, the attention-based modeling step includes a step of storing information of the input data in the attention-based model in an input data storage area in memory. Claim 26 An attention-based networking method including a softmax for memory-efficient learning, characterized in that, in the attention-based modeling step of claim 25, the attention-based modeling step includes a step of storing various parameters among the information of the attention-based model in the attention-based model in a model parameter storage area within the memory. Claim 27 delete Claim 28 An attention-based networking method including a softmax for memory-efficient learning, characterized in that, in the optimization step of claim 25, the optimization unit stores state information among the information of the optimization unit in an optimization state storage area within the memory. Claim 29 delete Claim 30 An attention-based networking method including a softmax for memory-efficient learning, characterized in that, in the attention-based modeling step of claim 19, the attention-based modeling step includes the step of storing activation information of the other layer in the other activation memory section within the activation memory region. Claim 31 delete Claim 32 delete Claim 33 delete Claim 34 delete Claim 35 An attention-based networking method including a softmax for memory-efficient learning, characterized in that, in the backpropagation process calculation step, the backpropagation process calculation step includes a step of predicting the input of the softmax calculation function using a gradient matrix for the output of the estimate calculation function in the gradient matrix calculation function within the backpropagation process submodule. Claim 36 An attention-based networking method including a softmax for memory-efficient learning, characterized in that, in the backpropagation process calculation step of claim 35, the weights of variables within the attention-based model are updated based on the output of the gradient matrix calculation function in the weight update calculation function within the backpropagation process submodule.

Images