Method for performing inference by condensing information of input data sequence in neural network, and apparatus therefor

Abstract

The present invention relates to a method for performing inference by condensing information of an input data sequence in a neural network, and an apparatus therefor. More particularly, disclosed in the present invention is a method performed in a computing apparatus for processing a token sequence by using a neural network, the method comprising the steps of: obtaining a first segment and a second segment on the basis of the token sequence; obtaining a first-1 segment on the basis of an output obtained by inputting the first segment into a first neural network; and obtaining a second-1 segment on the basis of an output calculated by inputting, into the first neural network, data obtained on the basis of a combination of the first-1 segment and the second segment.

Description

Method for performing inference by compressing information of a long input data sequence in a neural network and device therefor

[0001] The present invention relates to a neural network architecture, and more specifically, to a method and a device therefor that can effectively improve the operating characteristics of a neural network by compressing information of an input data sequence to perform inference.

[0002] Transformer, an attention-based sequence transduction neural network model proposed by Google in a 2017 paper titled "Attention Is All You Need," dramatically improved the problems of the previous RNN (Recurrent Neural Network) model, and since then, neural network technology using the attention mechanism has been widely used in various applications such as translation, text generation, chatbots, image generation, and music composition.

[0003] In particular, various language models have been announced, including OpenAI's ChatGPT, Google's Bard, Meta's Llama, Stanford University's Alpaca, and LMSYS.org's Vicuna, and they are showing potential as artificial general intelligence that can be utilized in various fields as well as text generation.

[0004] Figure 1 illustrates an example of a prior art transformer neural network model. Transformer models come in various types, including those with both an encoder and decoder, as presented in a 2017 Google paper; those with only an encoder, like BERT; and those with only a decoder, like GPT and Llama. Figure 1 illustrates a structure with only an encoder or decoder.

[0005] The transformer model (10) illustrated in Fig. 1 may include various layers such as an input transducer (11), n ​​stacked decoder (or encoder) blocks (20, 30, 40), a transposed embedding (50), and a softmax (60).

[0006] Here, each decoder / encoder block (20, 30, 40) may include layers such as a first layer norm (21), a masked multi-headed self-attention (22), a first add (23), a second layer norm (24), a pointwise feed forward network (25), and a second add (26). Depending on the implementation, the functions of the add and norm layers may be combined to represent an add & norm layer, a masking function may be selectively implemented in the multi-headed self-attention layer, and some layers or functions may be omitted or added as needed, and there may be variations in the detailed configuration.

[0007] In Fig. 1, the input sequence input X initmay be a sequence of input tokens, and the input transducer (11) generates an input token ID sequence by representing each token constituting the input sequence as a token ID, performs token embedding to convert it into an embedding vector corresponding to each token ID, and performs positional embedding to indicate the position of each token within the input sequence. Therefore, the input sequence input X init Through this process, it is converted into an embedding vector sequence, and the embedding vector sequence is input as input X in matrix form to the first encoder / decoder block (20).

[0008] Within each encoder / decoder block (20, 30, 40), self-attention operations are performed through a masked multi-headed self-attention (130) layer, and a transformation process is performed through a pointwise feed forward network. In this process, the first layer norm (21) and the second layer norm (24) layers can perform normalization on the output of the previous layer at appropriate locations, and the first add (23) and the second add (26) layers can be employed at appropriate locations for residual connection.

[0009] The output generated by the final encoder / decoder block (40) through the stacked (the number of stacks n is appropriately determined according to the model) encoder / decoder blocks (20, 30, 40) is converted into a logit through an operation with the transposed matrix of the matrix used in the embedding performed by the input transducer (11) in the transposed embedding (50) layer, and based on this, probability is calculated in the softmax (60) layer to obtain a probability vector (175) for the token, and an output token (during inference) or loss (during training) is obtained.

[0010] Typically, natural language processing technologies use large-scale corpora, typically consisting of tens of gigabytes and hundreds of millions of tokens, for learning. A vocabulary is a set of tokens obtained by tokenizing all sentences and documents in the corpus and removing duplicates. In large language models such as transformer models, a dictionary for the encoder / decoder is constructed and referenced in the token embedding process performed in the input transducer (105) and the logit generation process performed in the transposed embedding (165) layer.

[0011] However, transformer-based neural networks require memory proportional to the square of the number of input tokens to analyze the correlation between input tokens during learning and inference, and further consume computational time proportional to the cube of the number of input tokens. Therefore, increasing the number of input tokens could not only require enormous computing resources, but could also limit the performance of the neural network.

[0012] Accordingly, the length of input tokens may be limited for neural networks based on transformers, etc., and furthermore, when trying to increase the length of input tokens in a neural network, it may be difficult to secure the length for correlation analysis and context maintenance.

[0013] The present invention aims to improve the problem of rapidly increasing memory size and computational time required according to the number of tokens input into a neural network, thereby efficiently processing a large number of tokens with a small amount of memory to derive optimal prediction results.

[0014] In addition, the present invention is to efficiently perform association analysis while maintaining the length of context information for input tokens by compressing information of a sequence of preceding input tokens in a neural network.

[0015] More specifically, the present invention provides a method for efficiently performing inference by compressing a plurality of input tokens that occur prior to any input token into a small number of tokens, and further performing inference and learning while minimizing the resources required for this purpose.

[0016] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the contents described in this specification.

[0017] In a first aspect of the present invention, a method performed in a computing device for processing a token sequence using a neural network is provided, the method comprising: obtaining a first segment and a second segment based on the token sequence; obtaining a 1-1 segment based on an output obtained by inputting the first segment to a first neural network; and obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment to the first neural network.

[0018] Here, the step of obtaining a third segment based on the token sequence may further include; and the step of obtaining a third-1 segment based on an output produced by inputting data obtained based on the combination of the second-1 segment and the third segment into the first neural network.

[0019] In addition, the method may further include a step of obtaining a 1-2 segment based on an output obtained by inputting the 1-1 segment into a second neural network; and a step of obtaining a 2-2 segment based on an output obtained by inputting data obtained based on a combination of the 1-2 segment and the 2-1 segment into the second neural network.

[0020] Additionally, the first neural network may be a transformer layer and may include a memory that stores data regarding the first-first segment at least until the second segment is input.

[0021] In addition, the first neural network is a transformer layer, and the transformer layer performs an attention operation, and the attention operation can perform an operation based on a key and a value derived based on the first-first segment and the second segment, and a query derived based on the second segment.

[0022] At this time, the Key and the Value are derived from data including 2p vectors obtained by concatenating the 1-1 segment and the second segment, each including p vectors, and the Query can be derived from the second segment including p vectors.

[0023] In addition, the attention operation includes causal masking, and the causal masking can be applied to an attention score calculated to include a first sub-matrix region corresponding to the first segment and a second sub-matrix region corresponding to the second segment based on the query and the key.

[0024] Furthermore, the causal masking can be applied only to the second sub-matrix region corresponding to the second segment among the attention scores.

[0025] In addition, the first segment and the second segment may correspond to the sth and s+1th consecutive segments, respectively, among the plurality of segments obtained by dividing the token sequence.

[0026] At this time, in the neural network, the 1-1 segment may be calculated based on s segments from the first segment obtained by dividing the token sequence to the s-th segment, and the 2-1 segment may be calculated based on the 1-1 segment and the second segment.

[0027] Furthermore, the first-first segment can be produced to have the same size as the second segment.

[0028] In a second aspect of the present invention, a method for performing learning on a neural network that processes a token sequence is provided, the method comprising: obtaining a first segment and a second segment based on the token sequence; obtaining a 1-1 segment based on an output obtained by inputting the first segment to a first neural network; obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment to the first neural network; and performing backpropagation learning on the neural network using a loss function calculated based on the output for the first segment and the output for the second segment.

[0029] Here, the neural network is a transformer having a plurality of stacked layers, the first neural network is one layer of the plurality of layers, and in the performing step, the neural network can derive a loss function of the neural network based on a value obtained by summing each cross entropy obtained based on each of the output for the first segment and the output for the second segment.

[0030] In a third aspect of the present invention, a device is provided, comprising: a processor; and a memory, wherein the memory includes instructions configured to cause the device to implement specific operations for processing a token sequence using a neural network when executed by the processor, wherein the specific operations may include: obtaining a first segment and a second segment based on the token sequence; obtaining a 1-1 segment based on an output obtained by inputting the first segment to a first neural network; and obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment to the first neural network.

[0031] In a fourth aspect of the present invention, a device is provided, comprising: a processor; and a memory, wherein the memory includes instructions configured to cause the device to perform specific operations for performing learning on a neural network that processes a token sequence when executed by the processor, the specific operations including: obtaining a first segment and a second segment based on the token sequence; obtaining a 1-1 segment based on an output obtained by inputting the first segment to a first neural network; obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment to the first neural network; and performing backpropagation learning on the neural network using a loss function calculated based on the output of the first segment and the output of the second segment.

[0032] In a fifth aspect of the present invention, a computer-readable non-transitory storage medium is provided, storing instructions configured to cause a device including the processor to implement a specific operation for processing a token sequence using a neural network when executed by a processor, wherein the specific operation may include: obtaining a first segment and a second segment based on the token sequence; obtaining a 1-1 segment based on an output obtained by inputting the first segment to a first neural network; and obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment to the first neural network.

[0033] In a sixth aspect of the present invention, a computer-readable non-transitory storage medium is provided, storing instructions configured to cause a device including the processor to perform specific operations for performing learning on a neural network for processing a token sequence when executed by a processor, the specific operations including: obtaining a first segment and a second segment based on the token sequence; obtaining a 1-1 segment based on an output obtained by inputting the first segment to a first neural network; obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment to the first neural network; and performing backpropagation learning on the neural network using a loss function calculated based on the output for the first segment and the output for the second segment.

[0034] According to the present invention, the problem of the memory size and computation time required rapidly increasing according to the number of tokens input to a neural network is improved, thereby efficiently processing a large number of tokens with a small amount of memory to obtain optimal prediction results.

[0035] In addition, according to the present invention, it is possible to efficiently perform association analysis while maintaining the length of context information for input tokens by compressing information of a sequence of preceding input tokens in a neural network.

[0036] More specifically, according to the present invention, a method is provided for efficiently performing inference by compressing a plurality of input tokens that occurred prior to any input token into a small number of tokens, and further, performing inference and learning while minimizing the resources required for this purpose.

[0037] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the contents described in this specification.

[0038] The accompanying drawings are incorporated into and constitute a part of the detailed description to aid in the understanding of the present invention and together with the detailed description, illustrate embodiments and technical features of the present invention.

[0039] Figure 1 is a schematic diagram for exemplarily explaining the configuration and operation of a transformer neural network model of the prior art.

[0040] FIG. 2 is a conceptual diagram for explaining the configuration and operation of a neural network service providing system according to one embodiment of the present invention.

[0041] FIG. 3 is a flowchart illustrating a neural network-based inference method according to one embodiment of the present invention.

[0042] FIG. 4 is an example diagram of composing a segment from an input token sequence in a neural network-based inference method according to one embodiment of the present invention.

[0043] FIG. 5 is a block diagram for explaining the specific configuration and operation of a neural network-based inference method and device according to one embodiment of the present invention.

[0044] FIG. 6 is a conceptual diagram for explaining an inference process in a neural network having multiple layers in a neural network-based inference method according to one embodiment of the present invention.

[0045] FIG. 7 is a diagram illustrating a masking process in a neural network-based inference method according to one embodiment of the present invention.

[0046] FIG. 8 is a diagram illustrating the concept of path integration in a neural network-based inference method according to one embodiment of the present invention.

[0047] FIGS. 9 to 12 are diagrams illustrating performance improvements of a neural network-based inference method according to one embodiment of the present invention.

[0048] Figure 13 is a flowchart for explaining a neural network learning method according to one embodiment of the present invention.

[0049] Fig. 14 illustrates an apparatus capable of performing the proposed method of the present invention.

[0050] The present invention can be modified in various ways and has various embodiments. Hereinafter, specific embodiments will be described in detail based on the attached drawings.

[0051] The following examples are provided to provide a comprehensive understanding of the methods, devices, systems, and / or storage media described herein. However, these are merely examples and the scope of the present invention is not limited thereto.

[0052] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terms used in the detailed description are only for the purpose of describing embodiments of the present invention and should never be construed as limiting. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0053] Additionally, although terms such as first, second, etc. may be used to describe various components, the components are not limited by the terms, and the terms are used only for the purpose of distinguishing one component from another.

[0054] Next, exemplary embodiments of a method, device, system, storage device and computer program for performing inference by compressing information of an input data sequence in a neural network according to the present invention are described in detail with reference to the attached drawings.

[0055] First, FIG. 2 illustrates the configuration and operation of a neural network-based service providing system (100) according to one embodiment of the present invention. As can be seen in FIG. 2, the neural network-based service providing system (100) according to one embodiment of the present invention may be configured to include one or more terminals (110) and a neural network-based service providing device (120) that is interlocked with the one or more terminals (110) to provide a response to a user's request to the terminals (110) based on a neural network such as a transformer, or to perform tasks such as performing learning on the neural network using the terminals (110).

[0056] At this time, in the present invention, the terminal (110) can transmit a user's request to the neural network-based service providing device (120), receive and provide a response from the neural network-based service providing device (120) to the request, or provide a user interface environment for performing learning on the neural network.

[0057] Here, various terminals such as a personal computer (PC), a notebook PC, a tablet PC, a smartphone, a PDA, etc. may be used as the terminal (110), but the present invention is not necessarily limited thereto, and various devices capable of transmitting a user's request, receiving a response from the neural network-based service providing device (120) to the request, or providing a user interface environment for performing learning on a neural network may be used as the terminal (110).

[0058] In addition, the neural network-based service providing device (120) may be implemented using one or more physical server devices capable of providing a neural network-based service, but the present invention is not necessarily limited thereto, and may also be implemented in various forms, such as by using a cloud system or a dedicated device.

[0059] In addition, as the communication network (130) connecting the terminal (110) and the neural network-based service providing device (120) in FIG. 2, a wired network and a wireless network can be used, and specifically, various communication networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN) can be included. In addition, the communication network (130) may include the well-known World Wide Web (WWW). Furthermore, the communication network (130) may be implemented using a data bus configured to transmit and receive data, etc.

[0060] In addition, in FIG. 2, the terminal (110) and the neural network-based service providing device (120) are implemented separately, but the present invention is not necessarily limited thereto, and it is possible to implement it in various forms, such as in a combined form in which the terminal (110) and the neural network-based service providing device (120) are configured.

[0061] In addition, FIG. 3 illustrates a flowchart of a neural network-based inference method according to one embodiment of the present invention.

[0062] Here, the method illustrated in FIG. 3 can be performed by, for example, the neural network-based service providing device (120) of FIG. 2, and further, the neural network-based service providing device (120) can be implemented using the computing device (700) described below in relation to FIG. 14 and FIG. 14. For example, the neural network-based service providing device (120) includes a processor (710), and the processor (710) can execute instructions configured to implement operations for providing a service based on a neural network.

[0063] More specifically, as can be seen in FIG. 3, a neural network-based service providing method according to one embodiment of the present invention is a method performed in a computing device (700) for processing a token sequence using a neural network, the method including a step (S110) of obtaining a first segment and a second segment based on the token sequence, a step (S120) of obtaining a 1-1 segment based on an output obtained by inputting the first segment into a first neural network, and a step (S130) of obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment into the first neural network.

[0064] Here, the neural network-based service providing method may further include a step (not shown) of obtaining a third segment based on the token sequence and a step (not shown) of obtaining a third-1 segment based on an output obtained by inputting data obtained based on a combination of the second-1 segment and the third segment into the first neural network.

[0065] In addition, the neural network-based service providing method may further include a step (not shown) of obtaining a 1-2 segment based on an output obtained by inputting the 1-1 segment into a second neural network, and a step (not shown) of obtaining a 2-2 segment based on an output obtained by inputting data obtained based on a combination of the 1-2 segment and the 2-1 segment into the second neural network.

[0066] Additionally, the first neural network may be a transformer layer and may include a memory that stores data regarding the first-first segment at least until the second segment is input.

[0067] In addition, the first neural network may be a transformer layer, and the transformer layer may perform an attention operation, and the attention operation may perform an operation based on a key and a value derived based on the first-first segment and the second segment, and a query derived based on the second segment.

[0068] At this time, the Key and the Value are derived from data including 2p vectors obtained by concatenating the 1-1 segment and the second segment, each including p vectors, and the Query can be derived from the second segment including p vectors.

[0069] Additionally, the attention operation may include causal masking, wherein the causal masking may be applied to an attention score calculated to include a first sub-matrix region corresponding to the first segment and a second sub-matrix region corresponding to the second segment based on the query and the key.

[0070] Furthermore, the causal masking can be applied only to the second sub-matrix region corresponding to the second segment among the attention scores.

[0071] In addition, the first segment and the second segment may correspond to the sth and s+1th consecutive segments, respectively, among the plurality of segments obtained by dividing the token sequence.

[0072] At this time, in the neural network, the 1-1 segment may be calculated based on s segments from the first segment obtained by dividing the token sequence to the s-th segment, and the 2-1 segment may be calculated based on the 1-1 segment and the second segment.

[0073] Furthermore, the first-first segment can be produced to have the same size as the second segment.

[0074] Accordingly, in a method, device, system, storage device, and computer program for performing inference by compressing information of an input data sequence in a neural network according to one embodiment of the present invention, the problem of a rapid increase in the size of memory and computation time required depending on the number of tokens input to the neural network is improved, so that a large number of tokens can be efficiently processed to derive an optimal prediction result, and further, by compressing input tokens in the neural network, association analysis can be efficiently performed while maintaining the length of context retention for the input tokens, and further, a plurality of input tokens that occurred prior to any input token can be compressed into a small number of tokens to efficiently perform inference, and further, a method for performing inference and learning while minimizing the resources required for this can be provided.

[0075] Hereinafter, with reference to each drawing, the specific configuration and operation of a neural network-based service providing method, device, and system according to one embodiment of the present invention will be examined in more detail.

[0076] First, in step S110, a computing device (700) such as a neural network-based service providing device (120) obtains a first segment and a second segment based on a given token sequence.

[0077] More specifically, when a user inputs a request for a task or query to be performed using a terminal (110), the terminal (110) can transmit the user's request to a neural network-based service providing device (120) via a communication network (130), etc.

[0078] Accordingly, the neural network-based service providing device (120) can generate a token sequence including multiple tokens based on the user's request.

[0079] For a more concrete example, the above token sequence can be given as a sequence listing at most n d-dimensional vectors.

[0080] In this regard, in the neural network-based service providing device (120), as can be seen in FIG. 4, the token sequence can be divided into p vector units to form a plurality of segments.

[0081] For example, referring to Figure 4, the vector sequence is divided into 8 vector units and sequentially x 1 , x 2 , x 3 , ... , x m It can be composed of .

[0082] Accordingly, each X in Fig. 4 s may be a segment consisting of a vector sequence including eight vectors, and the neural network-based service providing device (120) may perform inference based on the segment.

[0083] Therefore, in the above step S110, the first segment and the second segment are obtained based on the token sequence.

[0084] At this time, the first segment is x of Fig. 4 1 This can be, and the second segment is x in Fig. 4 2 However, the present invention is not necessarily limited thereto, and more generally, the first segment and the second segment may correspond to the s-th and s+1-th consecutive segments among the plurality of segments obtained by dividing the token sequence (for example, the first segment may correspond to x 2 corresponds to , and the second segment is x 3 (Also possible are cases corresponding to .)

[0085] Next, in step S120, the first segment (x) is provided in a computing device (700) such as a neural network-based service providing device (120). s-1 ,t n s-1) is input to the first neural network and the output is based on the 1-1 segment (x s-1 ,t n s-1 +t) is obtained.

[0086] More specifically, in the step S120, the first segment (x s-1 ,t n s-1 ) is input to the first neural network (220) to produce an output, and here, the output (x) produced by the first neural network (220) s-1 ,t n s-1 +t) can be used as the first segment (230).

[0087] At this time, the first neural network (220) may be a layer of a transformer model, but the present invention is not necessarily limited thereto.

[0088] More specifically, referring to FIG. 5, the first neural network (220) may include a block that performs attention operation (221) and a feed forward network (FFN) (222), but it is also possible to configure the first neural network (220) in various forms, such as adding an ADD & NORM function block or modifying some of the function blocks.

[0089] Furthermore, as can be seen in FIG. 5, the first neural network (220) may be the kth layer of a transformer model including a plurality of layers, and in this case, the present invention can be applied to some or all of the plurality of layers.

[0090] Accordingly, referring to FIG. 5, the first segment (x s-1 ,t n s-1 ) is input into the first neural network (220) and the first segment (x) is input in a similar manner to the second segment (210) in FIG. 5. s-1 ,tn s-1 ) into the first neural network (220), and the output produced from the first neural network (220) is the 1-1 segment (x s-1 ,t n s-1 +t) can be used (in a similar manner to the 2-1 segment (230) in FIG. 5, the 1-1 segment (x s-1 ,t n s-1 +t) can be produced).

[0091] Next, in step S130, the computing device (700) such as the neural network-based service providing device (120) provides the first-1 segment (240) (x s-1 ,t n s-1 +t) and the second segment (210) (x s ,t n s ) is input into the first neural network (220) and the second-1 segment (230) is obtained based on the output produced by inputting the data obtained based on the concatenation of the first neural network (220).

[0092] More specifically, referring to Fig. 5, the 1-1 segment (x) produced through the above step S120 s ,t n s +t) can be stored in the memory (223) provided in the first neural network, and then the second segment (210) (x s ,t n s ) is entered, the 1-1 segment (240) (x) stored in the memory (223) s ,t n s +t) and the second segment (210)(x s ,t n s ) are combined to derive a key and value based on this, and also the second segment (210) (x s ,t ns ) is used to derive a query and input it into the first neural network (220), and the first neural network (220) performs an attention operation (221) and a feedforward network (222) based on the key, value, and query to generate the 2-1 segment (x s ,t n s It is possible to produce +t).

[0093] At this time, referring to FIG. 5, the key and the value can be derived from data including 2p vectors obtained by concatenating the first-1 segment (240) and the second segment (230) each including p vectors, and the query can also be derived from the second segment (230) including p vectors, but the present invention is not necessarily limited thereto.

[0094] Here, the first-first segment (240) can be calculated to have the same size as the second segment (210) (e.g., each including p vectors).

[0095] In addition, a neural network-based service providing method according to one embodiment of the present invention may further include a step (not shown) of obtaining a third segment based on a token sequence and a step (not shown) of obtaining a third-1 segment based on an output produced by inputting data obtained based on a combination of the second-1 segment and the third segment into the first neural network.

[0096] Furthermore, a neural network-based service providing method according to one embodiment of the present invention may further include a step (not shown) of obtaining a 1-2 segment based on an output obtained by inputting a 1-1 segment into a 2nd neural network, and a step (not shown) of obtaining a 2-2 segment based on an output obtained by inputting data obtained based on a combination of a 1-2 segment and a 2-1 segment into a 2nd neural network.

[0097] Here, the first neural network (220) may be the kth layer provided in a neural network such as a transformer model, as illustrated in FIG. 5, and the second neural network may be the k+1th layer subsequent to the kth layer, and in addition, a third neural network and an nth neural network may be additionally provided.

[0098] More specifically, Fig. 6 illustrates the inference process in a transformer model with three layers.

[0099] Referring to Fig. 6, first, in step S210, an inference process for the first segment is performed.

[0100] More specifically, first, the first segment (410) is input into the first layer (440) (i.e., the first neural network) to produce the 1-1 segment (411), the 1-1 segment (411) is input into the second layer (450) (i.e., the second neural network) to produce the 1-2 segment (412), and further, the 1-2 segment (412) is input into the third layer (460) (i.e., the third neural network) to produce the 1-3 segment (413).

[0101] Next, in step S220, the inference process for the second segment is performed.

[0102] More specifically, data combining the 1-1 segment (411) and the 2nd segment (420) produced in step S210 and the 2nd segment (420) are input into the 1st layer (440) to produce the 2-1 segment (421), and data combining the 1-2 segment (412) and the 2-1 segment (421) and the 2-1 segment (421) are input into the 2nd layer (450) to produce the 2-2 segment (422), and data combining the 1-3 segment (413) and the 2-2 segment (422) and the 2-2 segment (422) are input into the 3rd layer (460) to produce the 2-3 segment (423).

[0103] Next, in step S230, the inference process for the third segment is performed.

[0104] More specifically, data combining the 2-1 segment (421) and the 3-3 segment (430) produced in step S220 and the 3-1 segment (430) are input into the 1st layer (440) to produce the 3-1 segment (431), and data combining the 2-2 segment (422) and the 3-1 segment (431) and the 3-1 segment (431) are input into the 2nd layer (450) to produce the 3-2 segment (432), and data combining the 2-3 segment (423) and the 3-2 segment (432) and the 3-2 segment (432) are input into the 3rd layer (460) to produce the 3-3 segment (433).

[0105] At this time, in Fig. 6, a transformer model having three layers is described as the standard, but the present invention is not limited thereto, and it is also possible to configure the present invention in various forms by changing the number of layers, etc.

[0106] Furthermore, in the present invention, causal masking can be performed in the attention operation (221) of the first neural network (220).

[0107] Here, the causal masking can be applied to an attention score calculated to include a first sub-matrix region corresponding to the first segment and a second sub-matrix region corresponding to the second segment based on the query and the key.

[0108] In this regard, the causal masking can be applied only to the second sub-matrix region corresponding to the second segment among the attention scores.

[0109] More specifically, referring to FIG. 5, the Key and the Value can be derived from data including 2p vectors obtained by concatenating the 1-1 segment (240) including p vectors and the second segment (210), and the Query can also be derived from the second segment (210) including p vectors.

[0110] At this time, referring to FIG. 7, when calculating the attention score based on the query and the key, the attention score can be calculated by including a first sub-matrix area (510) corresponding to the first segment (or a 1-1 segment derived based thereon) and a second sub-matrix area (520) corresponding to the second segment.

[0111] More specifically, when the first segment corresponds to the s-th segment and the second segment corresponds to the s+1-th segment, the 1-1 segment is calculated based on s segments of the previous time point from the first segment obtained by dividing the token sequence to the s-th segment, so that information on input data of the past time point (i.e., from the first segment to the s-th segment) can be compressed and configured.

[0112] In this regard, as can be seen in FIG. 7, the causal masking can be applied only to the second sub-matrix region (520) corresponding to the second segment among the attention scores.

[0113] Accordingly, it is possible to exclude the application of masking to the first sub-matrix area (510) corresponding to the input data of the past time while applying masking to the second sub-matrix area (520) corresponding to the second segment corresponding to the input data of the current time.

[0114] At this time, as can be seen in FIG. 7, masking can be performed by replacing the matrix elements (522 in FIG. 7) above the diagonal in the second sub-matrix region (520) with a predetermined value (e.g., -∞), and accordingly, it is possible to replace the result value of applying the softmax function to the matrix elements (522 in FIG. 7) above the diagonal in the second sub-matrix region (520) with a negative infinity value so that they can be ignored.

[0115] Accordingly, for a more specific example, if the input token sequence includes 2048 vectors and there are 255 preceding segments corresponding to 2040 vectors prior to the segment corresponding to the current point in time (including 8 vectors), the present invention compresses the 255 preceding segments into the 1-1 segment (including 8 vectors), and together with this, it is possible to perform effective inference through correlation analysis on the 2048 vectors using only a total of 16 vectors by using the second segment (including 8 vectors) corresponding to the current point in time.

[0116] In this way, the process of performing inference while compressing and reflecting a plurality of preceding segments using a neural network such as a transformer based on attention operation in the present invention can be systematized based on a path integral format.

[0117] More specifically, in physics, including quantum mechanics, it is possible to calculate the particle state by weighting all paths that a particle can travel within a system based on the path integral.

[0118] In this regard, Fig. 8 visualizes and illustrates the change in token status in path integration. At this time, |x i , t n+1 〉 is the time t before reaching n Status in |x i , t n 〉 can change through multiple paths, where each path represents a possible transition path, and thus it is possible to analyze the state of the particle by performing a weighted summation for all possible paths through path integration.

[0119] Additionally, the principle of least action, which states that the motion of the system follows the path that minimizes the action, can be applied here, and this can be defined by the Lagrangian integral over time, etc.

[0120] Accordingly, in the present invention, a generalized neural network structure can be constructed by applying the path integral technique to a neural network such as a transformer, and further, as described above, it can be applied to perform effective inference by compressing a preceding segment in an input token sequence and considering the segment at the current point in time together.

[0121] For a more concrete example, in a typical transformer or other neural network, the attention operation first determines the relationships between tokens in a sequence, and then the information is processed by a feedforward network, a process similar to how a particle's state evolves over time by summing over all possible paths. This passage through the transformer layers can be interpreted as a token's state evolving over time, where each token can follow multiple paths before forming the final result.

[0122] More specifically, position x in the sequence i and time t n Token status in |x i , t n 〉 is time t n+1 Token status in |x i , t n+1 〉If we look at the case where it changes over time, it is possible to formulate it as in mathematical expression 1 below.

[0123] [Mathematical Formula 1]

[0124]

[0125] Here, L denotes the Lagrangian in the principle of least action, N is the overall normalization coefficient for the contributions of all possible paths, and T denotes the operator of the time-ordered product.

[0126] At this time, Under the condition, mathematical expression 1 can be rewritten as mathematical expression 2 below.

[0127] [Equation 2]

[0128]

[0129] At this time, t n t in n+1 In order to consider all possible paths to , an identity matrix such as the following mathematical expression 3 can be input.

[0130] [Equation 3]

[0131]

[0132] At this time, N j is at each time t n,j = t n + normalization factor at (j-1)t, which is t n,j Assume a transition to all possible token states in x. Also, k can mean the kth token position.

[0133] Accordingly, it is finally possible to derive the mathematical expression 4 below.

[0134] [Equation 4]

[0135]

[0136] Furthermore, for the above mathematical expression 4, t n + (j-1)t to t n By reflecting the state change over time with + jt, the following mathematical expression 5 can be derived.

[0137] [Equation 5]

[0138]

[0139] Next, based on the above mathematical expression 5, the token embedded state and the context vector of the transformer for position i By converting according to the correspondence relationship of mathematical expression 6 below, it is possible to derive the attention and neural network structure of the j-th transformer layer as in mathematical expression 7 below.

[0140] [Equation 6]

[0141]

[0142] [Equation 7]

[0143]

[0144] Here, FF stands for feedforward network, represent the query, key, and value matrices of the transformer, respectively.

[0145] In this way, in the present invention, it is possible to apply the concept of time to the transformer by reflecting the correspondence between the path integral and the transformer, and at this time, the transformer can be interpreted as a temporal change of the input token, and attention can be interpreted as the input token state being converted to all possible token states on the same time line.

[0146] At this time, it can be interpreted that a temporal change occurs when the input token passes through each transformer layer, and through this, it is possible to interpret that path integration is performed by considering all possible location and time paths.

[0147] Furthermore, by reducing the cross entropy loss between the output of the transformer and the target, the transformer can be trained to find the minimum operation path of the path integral, which can be expressed as in Equation 8 below.

[0148] [Equation 8]

[0149]

[0150] If the cross-entropy loss of the transformer is calculated as the transition amplitude between two states through path integration, the cross-entropy loss on the left side of the above equation will decrease as the model learns, and similarly, the high-frequency paths on the right side will be eliminated to obtain a larger prediction probability, so that only the low-value motion paths will survive to achieve the minimum action principle, which can explain why the attention-based transformer exhibits excellent characteristics in estimating the next token state.

[0151] In this regard, referring to Fig. 5, the token status at position i in the s-th segment is It can be defined as , and at this time, the token state can be changed in the transformer layer as shown in mathematical expression 9 below.

[0152] [Equation 9]

[0153]

[0154] Here, And, This is it.

[0155] At this time, temporal changes in token states can induce phase shifts in token states, and these phase shifts can be different in each layer. Consequently, if the phase differences between different segments are not properly handled, tokens in different segments can cancel each other out and interfere, resulting in the loss of important historical information from previous segments. To address this issue, a solution that creates constructive interference between segments can be to periodically apply time intervals between segments. More specifically, this can be implemented by applying a recursive transformer layer to each segment.

[0156] Accordingly, it is possible to derive the mathematical expression 10 below by inputting the s-1th and sth segments in parallel to the transformer layer.

[0157] [Equation 10]

[0158]

[0159] Here, as illustrated in Fig. 5, x s-1 Output for and x s are combined (concatenation), and the boundary condition of mathematical expression 11 below is applied.

[0160] [Equation 11]

[0161]

[0162] In this way, all segments are aligned at integer multiple time intervals so that all segments contribute to constructive interference, and this periodic temporal variation and the transition between the s-1th and sth segments can be expressed as in the following mathematical expression 12, as illustrated in the transformer layer of FIG. 5.

[0163] [Equation 12]

[0164]

[0165] Additionally, masking for the attention matrix can be applied in the same manner as the example described above in Fig. 7.

[0166] Accordingly, the present invention improves the problem of the memory size and computation time required rapidly increasing according to the number of tokens input to the neural network, thereby efficiently processing a large number of tokens to derive optimal prediction results, and furthermore, by compressing input tokens in the neural network, it is possible to efficiently perform association analysis while maintaining the context retention length for the input tokens, and furthermore, it is configured to efficiently perform inference by compressing multiple input tokens that occurred prior to any input token into a small number of tokens, and furthermore, it is possible to perform inference and learning while minimizing the resources required for this.

[0167] In this regard, Fig. 9 shows an experimental example of memory usage.

[0168] At this time, in the present invention, a token sequence of length L is divided into segments of Ns tokens, so that the memory required for attention calculation can be reduced, and since the segment length is fixed, the memory requirement according to the length of the token sequence can be reduced from an exponential increase to a linear increase.

[0169] In this regard, Fig. 9 illustrates a comparison between the present invention (condense batch of Fig. 9) and the Llama model, with the present invention being indicated by a solid line and the Llama model being indicated by a dotted line.

[0170] Here, when the total sequence length is 8K, the Llama model used 4.33 times more memory than the case of the present invention using batch 1, and when the total sequence length is 4K, the Llama model used 2.78 times more memory than the case of the present invention using batch 4.

[0171] Additionally, Fig. 10 illustrates a case where memory usage is compared according to the entire sequence length with FlashAttention enabled.

[0172] Here, when the total sequence length is 16K, the Llama model used about 1.77 times more memory than the case of the present invention with a batch size of 1, and when the total sequence length is 4K, the Llama model used about 1.71 times more memory than the case of the present invention with a batch size of 4.

[0173] Additionally, Fig. 11 compares the learning speed between the present invention and the L.ama model at various sequence lengths for each case with and without FlashAttention.

[0174] Here, for short sequences less than 2K tokens, the Llama model showed a somewhat faster learning speed than the present invention regardless of whether FlashAttention was applied. However, as the sequence length increased beyond 2K tokens, the present invention showed better characteristics than Llama in terms of efficiency, and in particular, a clear performance difference was shown in sequences of 4K and 8K tokens, where the present invention showed a shorter iteration time.

[0175] More specifically, when the total sequence length is 4K, when FlashAttention is applied to the present invention, learning iterations are processed approximately 1.68 times faster than Llama with FlashAttention applied.

[0176] The above memory and speed experiments were performed using the settings shown in Fig. 12, and both the present invention and the Llama model were tested on a single NVIDIA H100 GPU, and all models were trained and evaluated using the Lightning trainer framework without applying any additional optimization strategies to isolate the influence of the base architecture and FlashAttention.

[0177] According to these experimental results, in terms of memory efficiency, the architecture according to the present invention can effectively reduce memory usage by smoothing the memory increase curve, thereby reducing memory overhead and processing considerably long sequences. In addition, in terms of training speed, it showed higher efficiency in terms of speed as the sequence length increased beyond 2K tokens, demonstrating faster learning ability than Llama at 4K and 8K tokens.

[0178] In addition, FIG. 13 illustrates a flowchart for a neural network learning method according to one embodiment of the present invention.

[0179] Here, the neural network learning method illustrated in FIG. 13 can be performed by, for example, the neural network-based service providing device (120) of FIG. 2, and further, the neural network-based service providing device (120) can be implemented using the computing device (700) described below in relation to FIG. 14 and FIG. 14. For example, the neural network-based service providing device (120) has a processor (710), and the processor (710) can execute instructions configured to implement an operation for processing a token sequence using a neural network.

[0180] More specifically, as can be seen in FIG. 13, a neural network learning method according to one embodiment of the present invention is a method performed in a computing device (700) for performing learning on a neural network that processes a token sequence, the method including the steps of: obtaining a first segment and a second segment based on the token sequence (S310); obtaining a 1-1 segment based on an output obtained by inputting the first segment to a first neural network (S320); obtaining a 2-1 segment based on an output obtained by inputting data obtained based on a combination of the 1-1 segment and the second segment to the first neural network (S330); and performing backpropagation learning on the neural network using a loss function calculated based on the output for the first segment and the output for the second segment (S340).

[0181] As one embodiment of the present invention, a method for performing learning on a neural network includes a configuration that is substantially overlapping with the method for performing inference based on a neural network described above, the following description will omit overlapping descriptions and focus on the characteristic configurations applied in the learning process.

[0182] First, in order to perform learning on a neural network that processes a token sequence in the present invention, a first segment and a second segment can be produced based on the token sequence in step S310.

[0183] Next, in step S320, the first segment can be input into the first neural network, and the first-1 segment can be produced based on the output obtained.

[0184] Additionally, in step S330, the data obtained based on the combination of the first-1 segment and the second segment can be input into the first neural network, and the second-1 segment can be produced based on the output produced.

[0185] Accordingly, in step S340, backpropagation learning is performed on the neural network using a loss function calculated based on the output for the first segment and the output for the second segment.

[0186] Here, the neural network may be a transformer having a plurality of stacked layers, and the first neural network may be one layer of the plurality of layers.

[0187] At this time, in the step (S340) performed above, the neural network can derive the loss function of the neural network based on the value obtained by adding the cross entropies obtained based on each of the output for the first segment and the output for the second segment.

[0188] More specifically, referring to FIG. 6, the neural network calculates the first cross entropy based on the 1-3 segment (413) and the first target (414), which are the outputs for the first segment, and calculates the second cross entropy based on the 2-3 segment (423) and the second target (424), which are the outputs for the second segment, and further calculates the third cross entropy based on the 3-3 segment (433) and the third target (434), which are the outputs for the third segment, and then calculates the cross entropy (L) for each segment as in the following mathematical expression 13. s ) and sum them to form the overall loss function (L total ) is possible to produce.

[0189] [Equation 13]

[0190]

[0191] In addition, FIG. 14 illustrates a device (700) capable of performing the proposed method of the present invention.

[0192] Referring to FIG. 14, the device (700) may be configured to implement a process for compressing information of an input data sequence and performing inference or learning in the proposed method of the present invention and a neural network according to the method. For example, the device (700) may be a computing device, a server device, a terminal device, a network device, or the like for performing the process of the present invention.

[0193] For example, the device (700) to which the proposed method of the present invention can be applied may include network devices such as repeaters, hubs, bridges, switches, routers, gateways, etc., computer devices such as desktop computers, workstations, etc., mobile terminals such as smartphones, portable devices such as laptop computers, etc., home appliances such as digital TVs, etc., and transportation means such as automobiles, etc. As another example, the device (700) to which the present invention can be applied may be included as a part of an ASIC (Application Specific Integrated Circuit) implemented in the form of a SoC (System On Chip).

[0194] The memory (720) can be connected to the processor (710) when it operates, and can store programs and / or commands for processing and controlling the processor (710), and can store data and information used in the present invention, control information required for data and information processing according to the present invention, temporary data generated during data and information processing, etc. The memory (720) can be implemented as a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, an SRAM (Static RAM), an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0195] The processor (710) may be operatively connected to the memory (720) and the network interface (730), and controls the operation of each module within the device (700). In particular, the processor (700) may perform various control functions for performing the proposed method of the present invention. The processor (700) may also be referred to as a controller, a microcontroller, a microprocessor, a microcomputer, etc. The proposed method of the present invention may be implemented by hardware, firmware, software, or a combination thereof. When the present invention is implemented using hardware, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. configured to perform the present invention may be provided in the processor (710). Meanwhile, when implementing the proposed method of the present invention using firmware or software, the firmware or software may include instructions related to modules, procedures, or functions that perform functions or operations necessary to implement the proposed method of the present invention, and the instructions may be stored in the memory (720) or stored in a computer-readable recording medium (not shown) separate from the memory (720) and, when executed by the processor (710), the device (120) may be configured to implement the proposed method of the present invention.

[0196] In addition, the device (700) may include a network interface device (730). The network interface device (730) is connected to the processor (710) when in operation, and the processor (10) may control the network interface device (730) to transmit or receive wireless / wired signals carrying information and / or data, signals, messages, etc. through a wireless / wired network. The network interface device (730) supports various communication standards, such as, for example, IEEE 802 series, 3GPP LTE(-A), 3GPP 5G, etc., and may transmit and receive control information and / or data signals according to the communication standards. The network interface device (730) may also be implemented outside the device (700) as needed.

[0197] The embodiments described above are combinations of components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented without being combined with other components or features. Furthermore, it is also possible to form an embodiment of the present invention by combining some components and / or features. The order of operations described in the embodiments of the present invention may be changed. Some components or features of one embodiment may be included in another embodiment or may be replaced with corresponding components or features of another embodiment. It is self-evident that claims that do not have an explicit citation relationship in the patent claims may be combined to form an embodiment or incorporated as a new claim through a post-application amendment.

Claims

1. A method performed on a computing device for processing a token sequence using a neural network, A step of obtaining a first segment and a second segment based on the above token sequence; A step of obtaining a first-first segment based on an output obtained by inputting the first segment into a first neural network; and A method comprising the step of inputting data obtained based on the combination of the first-1 segment and the second segment into the first neural network and obtaining the second-1 segment based on the output produced.

2. In paragraph 1, A step of obtaining a third segment based on the above token sequence; and A method further comprising a step of obtaining a 3-1 segment based on an output obtained by inputting data obtained based on a combination of the 2-1 segment and the 3rd segment into the 1st neural network.

3. In paragraph 1, A step of obtaining a 1-2 segment based on an output obtained by inputting the 1-1 segment to a 2nd neural network; and A method further comprising the step of obtaining a 2-2 segment based on an output obtained by inputting data obtained based on a combination of the 1-2 segment and the 2-1 segment into the 2nd neural network.

4. In paragraph 1, The above first neural network is a transformer layer, A method comprising a memory for storing data regarding the first-first segment at least until the second segment is input.

5. In paragraph 1, The above first neural network is a transformer layer, In the above transformer layer, attention operations are performed, A method in which, in the above attention operation, an operation is performed based on a key and a value derived based on the first-first segment and the second segment, and a query derived based on the second segment.

6. In paragraph 5, The above key and the above value are derived from data including 2p vectors obtained by concatenating the first-1 segment and the second segment, each including p vectors, A method wherein the above query is derived from the second segment containing p vectors.

7. In paragraph 5, The above attention operation includes causal masking, A method in which the above causal masking is applied to an attention score calculated to include a first sub-matrix region corresponding to the first segment and a second sub-matrix region corresponding to the second segment based on the query and the key.

8. In paragraph 7, A method in which the above causal masking is applied only to the second sub-matrix region corresponding to the second segment among the attention scores.

9. In paragraph 1, The first segment and the second segment, A method in which, among a plurality of segments obtained by dividing the above token sequence, each segment corresponds to the sth and s+1th consecutive segments.

10. In paragraph 9, In the above neural network, The above 1-1 segment is calculated based on s segments from the first segment obtained by dividing the token sequence to the s-th segment, A method in which the above 2-1 segment is produced based on the above 1-1 segment and the above 2nd segment.

11. In paragraph 10, A method in which the above 1-1 segment is produced to have the same size as the above 2nd segment.

12. A method performed on a computing device for performing training on a neural network that processes a token sequence, A step of obtaining a first segment and a second segment based on the above token sequence; A step of obtaining a 1-1 segment based on an output obtained by inputting the first segment into a first neural network; A step of obtaining a 2-1 segment based on an output produced by inputting data obtained based on the combination of the 1-1 segment and the 2nd segment into the 1st neural network; and A method comprising the step of performing backpropagation learning for the neural network using a loss function calculated based on the output for the first segment and the output for the second segment.

13. In paragraph 12, The above neural network is a transformer having multiple stacked layers, The above first neural network is one layer among the plurality of layers, In the above-mentioned performing step, A method for deriving a loss function of the neural network based on a value obtained by adding up each cross entropy obtained based on the output for the first segment and the output for the second segment.

14. In a device including a processor and memory, The memory comprises instructions configured to cause the device to implement specific operations for token sequence processing using a neural network when executed by the processor, the specific operations comprising: Obtaining a first segment and a second segment based on the above token sequence; Obtaining the 1-1 segment based on the output obtained by inputting the first segment to the first neural network; and A device comprising: inputting data obtained based on the combination of the first-1 segment and the second segment into the first neural network and obtaining the second-1 segment based on the output produced.

15. A device including a processor and a memory, The memory comprises instructions configured to cause the device to perform specific operations for training a neural network to process a sequence of tokens when executed by the processor, the specific operations comprising: Obtaining a first segment and a second segment based on the above token sequence; Obtaining the 1-1 segment based on the output obtained by inputting the above 1st segment into the 1st neural network; Obtaining a 2-1 segment based on the output produced by inputting data obtained based on the combination of the 1-1 segment and the 2nd segment into the 1st neural network; and A device comprising performing backpropagation learning on the neural network using a loss function calculated based on the output for the first segment and the output for the second segment.

16. A computer-readable non-transitory storage medium storing instructions configured to cause a device including the processor to implement a specific operation for token sequence processing using a neural network when executed by the processor, wherein the specific operation is: Obtaining a first segment and a second segment based on the above token sequence; Obtaining the 1-1 segment based on the output obtained by inputting the first segment to the first neural network; and A computer-readable non-transitory storage medium comprising inputting data obtained based on the combination of the first segment and the second segment into the first neural network and obtaining the second segment based on the output produced.

17. A computer-readable non-transitory storage medium storing instructions configured to cause a device including the processor to perform a specific operation for training a neural network that processes a token sequence when executed by the processor, the specific operation comprising: Obtaining a first segment and a second segment based on the above token sequence; Obtaining the 1-1 segment based on the output obtained by inputting the above 1st segment into the 1st neural network; Obtaining a 2-1 segment based on the output produced by inputting data obtained based on the combination of the 1-1 segment and the 2nd segment into the 1st neural network; and A computer-readable non-transitory storage medium comprising performing backpropagation learning for the neural network using a loss function calculated based on the output for the first segment and the output for the second segment.

Images