Information processing systems and methods

Abstract

To provide an information processing system and method that enables high-speed processing of transformer-type neural networks. [Solution] The storage device stores the first key and the first value. The self-attention layer performs the following processing each time a self-attention input is received. The self-attention layer generates a query, a second key, and a second value based on the self-attention input. The self-attention layer stores the generated key and value in the storage device. The self-attention layer obtains an attention score by the dot product of the query and the key matrix containing the first key and the second key stored in the storage device. The self-attention layer performs a second calculation to calculate the dot product of the attention score and the value matrix containing the first value and the second value stored in the storage device.

Description

[Technical Field]

[0001] This embodiment relates to an information processing system and method. [Background technology]

[0002] One widely known machine learning model is the transformer neural network. Transformer neural networks incorporate an attention layer. The attention layer enhances important information and weakens irrelevant information based on the relationships between elements in the input sequence. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Patent No. 10452978 [Patent Document 2] Japanese Patent Publication No. 2024-93565 [Overview of the project] [Problems that the invention aims to solve]

[0004] One embodiment aims to provide an information processing system and method that enables high-speed processing of transformer-type neural networks. [Means for solving the problem]

[0005] According to one embodiment, the information processing system comprises one or more computers that execute instructions and one or more storage devices that store a first set of a first key and a first value. In executing instructions, one or more computers execute processing by a transformer neural network. The transformer neural network comprises a self-attention layer. The transformer neural network inputs self-attention inputs to the self-attention layer for each of the multiple input positions in the order of input. The transformer neural network then outputs an output sequence corresponding to the first input sequence based on the output from the self-attention layer corresponding to the self-attention input. The self-attention layer performs the following processing each time a self-attention input is received: The self-attention layer generates a query based on the self-attention input and a first input position which is the input position corresponding to the self-attention input among the multiple input positions. The self-attention layer generates a key based on the self-attention input and the first input position and stores the generated key, the second key, in one or more storage devices. The self-attention layer generates a value based on the self-attention input and stores the generated value, the second value, in one or more storage devices. The self-attention layer performs a first calculation to obtain an attention score by the inner product of the query and a key matrix containing the first and second keys stored in one or more storage devices. The self-attention layer performs a second calculation to calculate the inner product of the attention score and a value matrix containing the first and second values ​​stored in one or more storage devices. The self-attention layer outputs the result of the second calculation. [Brief explanation of the drawing]

[0006] [Figure 1] A diagram showing an example of the configuration of an information processing system according to the first embodiment. [Figure 2] A diagram showing an example of information stored in the SSD according to the first embodiment. [Figure 3] A diagram showing an example of the configuration of the transformer according to the first embodiment. [Figure 4] A diagram showing an example of the configuration of the self-attention layer according to the first embodiment. [Figure 5] A diagram for explaining the data structure of the index information according to the first embodiment. [Figure 6] A flowchart showing an example of the operation according to the second input sequence of the information processing system according to the first embodiment. [Figure 7] A flowchart showing an example of the operation according to the first input sequence of the information processing system according to the first embodiment. [Figure 8] A diagram showing an example of information stored in the SSD according to the second embodiment. [Figure 9] A diagram showing an example of the configuration of the self-attention layer according to the second embodiment. [Figure 10] A diagram showing an example of information stored in the SSD according to the third embodiment. [Figure 11] A flowchart showing an example of the operation according to the first input sequence of the information processing system according to the third embodiment.

Embodiments for Carrying Out the Invention

[0007] The information processing system and method according to the embodiments will be described in detail below with reference to the accompanying drawings. Note that the present invention is not limited by these embodiments.

[0008] (First Embodiment) FIG. 1 is a diagram showing an example of the configuration of the information processing system according to the first embodiment.

[0009] The information processing device 1 is an information processing system according to the first embodiment. In the example shown in Figure 1, the information processing device 1 comprises a processor 11, an interface 12, an SSD (Solid State Drive) 13, a RAM (Random Access Memory) 14, and a bus 15. The processor 11, interface 12, SSD 13, and RAM 14 are electrically connected to the bus 15.

[0010] Interface 12 is a device for inputting and outputting information to the information processing device 1. Interface 12 includes interfaces for communication via a network, interfaces to which a storage device may be connected, interfaces to which input devices such as a keyboard may be connected, and so on.

[0011] SSD13 is a large-capacity non-volatile memory that functions as a storage device in the information processing device 1. Note that the storage device applicable to the information processing device 1 is not limited to SSDs. The information processing device 1 may also include a magnetic disk drive (HDD in one example) as a storage device.

[0012] RAM14 is memory that offers faster access than storage devices. RAM14 functions as a cache area, buffer area, or work area, among other things. DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), or a combination of these can be used as RAM14. However, the types of memory that can be used as RAM14 are not limited to these. RAM14 may be volatile or non-volatile memory.

[0013] The processor 11 is an arithmetic unit capable of executing computer programs and realizing functions defined by the computer programs. The processor 11 is, for example, a CPU (Central Processing Unit). In the information processing device 1, the processor 11 executes the processing of a transformer-type neural network according to the information processing program (information processing program PRG, described later).

[0014] Note that processor 11 is an example of a computer. SSD 13 and RAM 14 are examples of one or more storage devices.

[0015] In the example shown in Figure 1, the information processing system according to the embodiment is composed of one information processing device 1. The information processing system according to the embodiment may be composed of two or more information processing devices. Furthermore, the information processing device 1 may include two or more processors, each having a configuration equivalent to that of processor 11.

[0016] Figure 2 shows an example of information stored in the SSD 13 according to the first embodiment. In the example shown in this figure, the SSD 13 stores the information processing program PRG and index information IDX.

[0017] As described above, the processor 11 executes the processing of the transformer-type neural network by executing the information processing program PRG. The transformer-type neural network that the processor 11 implements according to the information processing program PRG is referred to as the transformer 100.

[0018] Note that the information processing program PRG is an example of an instruction.

[0019] Before explaining the IDX index information, let's briefly explain Transformer 100.

[0020] Transformer 100 converts the input sequence SQ into an output sequence corresponding to the input sequence SQ. The input sequence SQ is an array in which multiple elements (hereinafter referred to as tokens) are arranged in input order. Each of the multiple elements (i.e., tokens) is an example of a network input. That is, the input sequence SQ has a network input at each of the multiple input positions in input order.

[0021] The transformer 100 can, for example, receive text data as input sequence SQ. When input sequence SQ is text data, for example, each word that makes up the text is a token. However, the input sequence SQ that can be input to the transformer 100 is not limited to text data. Image data, audio data, or video data may also be input to the transformer 100 as input sequence SQ.

[0022] The transformer 100 generates an output sequence by sequentially processing each token. In this specification, tokens in the process of processing by the transformer 100 are also referred to as tokens.

[0023] The transformer 100 includes an attention layer. The attention layer enhances noteworthy information and diminishes non-noteworthy information based on the relationships between tokens in the input sequence SQ. To take into account the relationships between tokens in the input sequence SQ, the attention layer generates key-value vector pairs as intermediate data for each token input to the attention layer.

[0024] More specifically, each time a new token is input, the attention layer generates a query vector, a key vector, and a value vector from the newly input token. The attention layer stores the generated key vector and value vector in pairs in a predetermined memory area. A pair of a key vector and a value vector is referred to as a key-value pair. The attention layer identifies the key vector that is closest to the query vector from among the stored group of key-value pairs. The attention layer then considers the value vector that forms a pair with the identified key vector from the stored group of value vectors as noteworthy information and strengthens it, considers the other value vectors as unnoteworthy information and weakens them, and takes the sum of the stored group of value vectors as output.

[0025] Hereafter, the value vector that forms a pair with a key vector will be referred to as the value vector corresponding to the key vector.

[0026] This section describes technologies that are comparable to the embodiments. Technologies that are comparable to the embodiments are referred to as comparative examples. According to the comparative examples, when the transformer neural network completes the generation or output of an output sequence corresponding to an input sequence, it discards the group of key-value pairs generated by the attention layer processing for that input sequence and stored in a predetermined memory area. In other words, the attention layer processing for a single input sequence is performed based only on the group of key-value pairs generated from that single input sequence.

[0027] Therefore, according to the comparative example, the longer the length of the input sequence, the greater the amount of key-value pairs generated and stored as intermediate data, and the longer the time required to obtain the output corresponding to the input sequence.

[0028] In the first embodiment, processing of a second input sequence SQ by a transformer neural network is performed before a certain input sequence SQ (referred to as the first input sequence SQ) is input. The group of key-value pairs obtained by processing the second input sequence SQ by the transformer neural network is stored in SSD13. When generating an output sequence corresponding to the first input sequence SQ, the attention layer performs processing using a group of key-value pairs that includes key-value pairs generated for each token contained in the first input sequence SQ and key-value pairs pre-generated based on the second input sequence SQ. In other words, by completing the processing of some of the input sequence SQs (the second input sequence SQ) in advance, the time required for processing the transformer neural network on later input sequence SQs (the first input sequence SQ) is suppressed.

[0029] Furthermore, in the first embodiment, the group of key-value pairs that have been generated in advance and stored in SSD13, that is, the group of key-value pairs generated based on the second input sequence SQ, is graphed so that it can be searched using the Approximate Nearest Neighbor search (ANN) method.

[0030] As mentioned above, in the attention layer, value vectors corresponding to key vectors other than the one closest to the query vector are weakened. Therefore, even if we ignore the value vectors corresponding to key vectors that are not close to the query vector among the group of key-value pairs generated based on the second input sequence SQ, the effect on the processing result of the transformer neural network is negligible or nonexistent.

[0031] Therefore, in the first embodiment, from the group of key-value pairs generated based on the second input sequence SQ, only a predetermined number of key-value pairs containing one of a predetermined number of key vectors that are closest in distance to the query vector are read from SSD13 and used for processing by the attention layer. To identify the predetermined number of key-value pairs containing one of a predetermined number of key vectors that are closest in distance to the query vector, the attention layer performs a search using the ANN method. Note that "determined number" is a small number. More specifically, "determined number" is one or more. Hereafter, for the sake of simplicity, we will assume that "determined number" is one.

[0032] The index information IDX is obtained by graphing the group of key-value pairs generated based on the second input sequence SQ. Therefore, in addition to the information of the group of key-value pairs generated based on the second input sequence SQ, the index information IDX includes information that defines the graph structure. The attention layer searches for the key-value pair containing the key vector closest to the query vector from the group of key-value pairs generated based on the second input sequence SQ, along the graph structure defined by the index information IDX. The detailed data structure of the index information IDX will be described later.

[0033] In this specification, distance is a measure of similarity between data. In the attention layer, the distance between the query vector and the key vector is obtained by calculating the dot product of the query vector and the key vector. In the ANN, the distance may be obtained by calculating the dot product, or by a calculation other than the dot product. Note that the value calculated by the dot product is larger the closer the distance between the query vector and the key vector.

[0034] Figure 3 shows an example of the configuration of the transformer 100 according to the first embodiment.

[0035] The transformer 100 comprises an embedding layer 110, N layers of transformer blocks 120, a normalization layer 130, and a linear layer 140, where N is a natural number greater than or equal to 1.

[0036] The transformer 100 receives the input sequence SQ. The input sequence SQ is an array consisting of multiple tokens. The multiple tokens are input to the transformer 100 in the order they are entered. Hereafter, "input order" refers to the order in which the tokens are entered in the input sequence SQ.

[0037] The embedding layer 110 vectorizes the tokens and maps the vectorized tokens to fixed-length embedding vectors using a linear transformation. In other words, the embedding layer 110 converts tokens into embedding representations. Multiple tokens are input to the embedding layer 110 in the order they are input. Therefore, the embedding layer 110 converts each of the multiple tokens into an embedding representation in the order they are input.

[0038] Multiple tokens, converted into an embedded representation by the embedding layer 110, are input to the N-layer transformer block 120 in the order they were entered.

[0039] If N is 1, the transformer block 120 processes each token that is sequentially input from the embedding layer 110, and inputs the processed tokens into the normalization layer 130 in the order they were input.

[0040] If N is 2 or greater, the N-layer transformer blocks 120 are connected serially. The head transformer block 120 of the N-layer transformer blocks 120 processes each token that is sequentially input from the embedding layer 110, and outputs the processed tokens to the subsequent transformer blocks 120 in the order they were input. Each transformer block 120 connected after the head of the N-layer transformer blocks 120 processes each token that is sequentially input from the preceding transformer block 120, and outputs the processed tokens to the subsequent transformer block 120 or normalization layer 130 in the order they were input.

[0041] The normalization layer 130 normalizes each individual token for multiple tokens that are input sequentially. The multiple tokens normalized by the normalization layer 130 are then input into the linear layer 140 in the order they were input.

[0042] The linear layer 140 performs a linear transformation on each token of a sequence of input tokens. The linear layer 140 may have learning parameters. The multiple tokens linearly transformed by the linear layer 140 are output from the transformer 100 as an output sequence.

[0043] Each transformer block 120 comprises a normalization layer 121, a self-attention layer 122, a coupling section 123, a normalization layer 124, a feedforward layer 125, and a coupling section 126. The self-attention layer 122 is a type of attention layer.

[0044] Multiple tokens are sequentially input to the normalization layer 121 and the junction 123 in the order they were received from the embedding layer 110 or the preceding transformer block 120.

[0045] The normalization layer 121 normalizes each individual token for multiple tokens that are input sequentially. The multiple tokens normalized by the normalization layer 121 are then input to the self-attention layer 122 in the order they were input.

[0046] The self-attention layer 122 processes each token individually for multiple tokens that are input sequentially. Details of the processing in the self-attention layer 122 will be described later. The multiple tokens processed by the self-attention layer 122 are input to the coupling unit 123 in the order they were input.

[0047] Hereafter, each token input to the self-attention layer 122 may be referred to as a self-attention input. Each token output from the self-attention layer 122 may be referred to as a self-attention output.

[0048] The joiner 123 joins multiple tokens sequentially input from the embedded layer 110 or the preceding transformer block 120 with multiple tokens sequentially input from the self-attention layer 122, joining tokens at the same input position. The multiple tokens joined by the joiner 123 are input to the normalization layer 124 and the joiner 126 in the order they were input.

[0049] The normalization layer 124 normalizes each individual token for multiple tokens that are input sequentially. The multiple tokens normalized by the normalization layer 124 are then input to the feedforward layer 125 in the order they were input.

[0050] The feedforward layer 125 is a neural network. The feedforward layer 125 processes each token of a sequentially input sequence of tokens using the neural network.

[0051] The coupling unit 126 combines multiple tokens sequentially input from the coupling unit 123 and multiple tokens sequentially input from the feedforward layer 125, combining tokens at the same input position. The multiple tokens combined by the coupling unit 126 are input in the order of input to the subsequent transformer block 120 or normalization layer 130.

[0052] Note that the configuration of the transformer 100 shown in Figure 3 is merely an example. The components included in the transformer 100 and the connections between them can be changed in various ways. For example, the order of connection between the normalization layer 124 and the feedforward layer 125 may be reversed from the connection relationship shown in Figure 3.

[0053] In the first embodiment, the transformer 100 processes the first input sequence SQ and the second input sequence SQ. The transformer 100 that processes the first input sequence SQ and the transformer 100 that processes the second input sequence SQ may have different configurations. In such cases, the processing of the attention layer may be referred to as cross-attention rather than self-attention.

[0054] Figure 4 shows an example of the configuration of the self-attention layer 122 according to the first embodiment.

[0055] The self-attention layer 122 receives multiple tokens that have undergone processing prior to the self-attention layer 122 in the order they were received. That is, the self-attention layer 122 receives a token for each of the multiple input positions. The self-attention layer 122 processes each of the received tokens sequentially and outputs the processed token as an attention output.

[0056] For this purpose, the self-attention layer 122 comprises fully connected neural networks 201, 202, and 203, positional encoding layers 211 and 212, a K-cache storage unit 222, a V-cache storage unit 223, an ANN search unit 231, concatenation units 232 and 233, a first calculation unit 241, and a second calculation unit 242.

[0057] Frequent vector write / read operations are performed on the K cache storage 222 and the V cache storage 223. Therefore, the K cache storage 222 and the V cache storage 223 can be allocated to the RAM 14. One or both of the K cache storage 222 and the V cache storage 223 may be allocated to the SSD 13.

[0058] Tokens input to the self-attention layer 122 as self-attention input are shared input to the fully connected neural networks 201, 202, and 203.

[0059] Tokens processed by the fully connected neural network 201 are input to the position encoding layer 211. The position encoding layer 211 embeds the input position of each token. Tokens with their input positions embedded by the position encoding layer 211 are input as query vectors Q to the ANN search unit 231 and the first calculation unit 241.

[0060] Note that the above token input position is just one example of the first input position.

[0061] Tokens processed by the fully connected neural network 202 are input to the position encoding layer 212. The position encoding layer 212 embeds the input position of each token. Tokens with their input positions embedded by the position encoding layer 212 are stored as key vectors K in the K cache memory unit 222. Thus, the K cache memory unit 222 accumulates key vectors K for all tokens processed by the self-attention layer 122 since the input of the input sequence SQ began.

[0062] Tokens processed by the fully connected neural network 203 are stored in the V cache storage unit 223 as value vectors V. Therefore, the V cache storage unit 223 accumulates value vectors V for all tokens processed by the self-attention layer 122 from the start of input of the input sequence SQ.

[0063] The ANN search unit 231 searches for a key-value pair containing the key vector K that is closest to the query vector Q from a pre-generated group of key-value pairs by performing a search using the ANN method along the graph structure defined by the index information IDX. The key vector K that constitutes the key-value pair obtained by the search (key vector K str (This is written as) and value vector V (Value vector V str (To be denoted as) Among these, key vector K str The value vector V is input to the coupling section 232. str This is input to the connecting section 233.

[0064] The group of key vectors K stored in the K cache storage unit 222 is input to the concatenation unit 232. The concatenation unit 232 processes the key vectors K str And, by concatenating the group of key vectors K input from the K cache memory unit 222, the key vector K strA matrix (denoted as the K matrix) is generated that includes all the key vectors K stored in the K cache storage unit 222. The generated K matrix is ​​input to the first calculation unit 241.

[0065] All the value vectors V stored in the V cache storage unit 223 are input to the concatenation unit 233. The concatenation unit 233 inputs the value vectors V str And, by concatenating the group of value vectors V input from the V cache storage unit 223, the value vector V str A matrix (denoted as the V matrix) is generated that includes all the value vectors V stored in the K cache storage unit 222. The generated V matrix is ​​input to the second calculation unit 242.

[0066] The first calculation unit 241 performs a first calculation that includes calculating the inner product of the query vector Q and the K matrix. By calculating the inner product of the query vector Q and the K matrix, the distance to the query vector Q is obtained for each key vector K contained in the K matrix.

[0067] In the first calculation, the first calculation unit 241 further applies the softmax function to the result of the inner product calculation. As a result, the distances for each key vector K contained in the K matrix are converted into calculated values ​​having the following properties.

[0068] In other words, the calculated value is close to 1 when the distance to the query vector Q is shortest, and is almost 0 when the distance to the query vector Q is not shortest. All calculated values ​​of key vectors K fall within the interval between 0 and 1, and the sum of all calculated values ​​of key vectors K is 1.

[0069] Value vectors V corresponding to key vectors K whose calculated values ​​from the first calculation are close to 1 are considered to be of greater importance, while value vectors V corresponding to key vectors K whose calculated values ​​from the first calculation are close to 0 are considered to be of less importance. Therefore, the vector obtained by collecting the calculated values ​​for each key vector K in matrix K is called the attention score, in the sense that it represents the level of attention each value vector V in matrix V is receiving.

[0070] The attention score generated by the first calculation unit 241 is input to the second calculation unit 242. The second calculation unit 242 performs a second calculation, which includes calculating the inner product of the attention score and the V matrix. The second calculation calculates the sum of all value vectors V included in the V matrix, using the attention score as a weight. In other words, the sum of all value vectors V included in the V matrix is ​​calculated while strengthening the value vectors V that deserve attention and weakening the value vectors V that do not deserve attention. The vector obtained by the second calculation is output as a self-attention output.

[0071] Figure 5 is a diagram illustrating the data structure of the index information IDX according to the first embodiment.

[0072] The pair set 300 is a group of key-value pairs generated in the self-attention layer 122 based on the second input sequence SQ. Each key vector K in the pair set 300 is considered a node and given a node ID. Then, a directed graph GF is generated in which each node is connected by an edge.

[0073] In the example shown in Figure 5, a set of 300 pairs is pre-generated, including pairs of key vector K1 and value vector V1, key vector K2 and value vector V2, key vector K3 and value vector V3, and so on. The key vector Ki (where i is a unique number) is assigned the node ID i. In Figure 5, the node with node ID i is denoted as node NDi. The node ID of node NDi is denoted as NIDi.

[0074] Furthermore, in the example of the directed graph GF shown in Figure 5, node ND1 is connected to edges headed by node ND20, node ND7, node ND13, and node ND12. Node ND20 is connected to edges headed by node ND3, node ND6, node ND15, and node ND11. Node ND7 is connected to edges headed by node ND10, node ND19, node ND16, and node ND5. Node ND13 is connected to edges headed by node ND8, node ND21, node ND4, and node ND2. Node ND12 is connected to edges headed by node ND14, node ND17, node ND9, and node ND18.

[0075] Note that the structure of the directed graph GF described above is merely an example. Furthermore, the method for generating the directed graph GF from the pair set 300 is arbitrary, as long as the generated directed graph GF can be explored using an ANN method.

[0076] In this specification, if node NDA and node NDb are connected by an edge with node NDb as the head, node NDb will be referred to as the adjacent node of node NDA.

[0077] Based on the pair set 300 and the structure of the directed graph GF, index information IDX is generated. In the example shown in Figure 5, index information IDX has a data structure in which multiple node information is arranged in the order of node IDs. Each node information includes a key vector K of one node (referred to as the target node), a value vector V corresponding to the key vector K of the target node, and a list of node IDs of the neighboring nodes of the target node. In other words, each node information includes a key-value pair and information about neighboring nodes as information that defines the structure of the directed graph GF.

[0078] The ANN search unit 231 identifies key-value pairs containing the key vector closest to the query vector Q, according to the directed graph GF defined by the index information IDX. The ANN search unit 231 can employ any algorithm for the search, including greedy search and beam search. To briefly explain one example, the ANN search unit 231 sequentially switches the target node to be searched along the graph GF among multiple nodes. Each time the target node to be searched is switched, the ANN search unit 231 calculates the distance between each adjacent node of the target node to the query vector Q. Then, the ANN search unit 231 sets the node closest to the query vector Q among one or more adjacent nodes of the target node that is currently close to the query vector Q as the next new target node to be searched. The ANN search unit 231 sequentially switches the target nodes to be searched along the directed graph GF until it reaches the node that is presumed to be closest to the query vector Q. This process of switching the target nodes to be searched along the graph GF is also called "hopping".

[0079] Next, the operation of the information processing device 1 according to the first embodiment will be described.

[0080] Figure 6 is a flowchart showing an example of the operation of the information processing device 1 according to the first embodiment in response to the second input sequence SQ.

[0081] First, the processor 11 acquires the second input sequence SQ (S101). The method for acquiring the second input sequence SQ is not limited to a specific method.

[0082] For example, the second input sequence SQ may be input to the information processing apparatus 1 by an operator of the information processing apparatus 1. Alternatively, the processor 11 may acquire some content as the second input sequence SQ via a network such as the Internet according to an information processing program PRG or another program. TM

[0083] Subsequently, the processor 11 executes the processing of the transformer 100 for the second input sequence SQ according to the information processing program PRG (S102). The processor 11 outputs the acquired second input sequence SQ to the transformer 100 and executes the processing of the transformer 100 for the second input sequence SQ.

[0084] At the time of S102, the index information IDX has not yet been generated. Therefore, in the self-attention layer 122, the search by the ANN search unit 231 is not performed. That is, the K matrix is configured based only on the key vectors K stored in the K cache storage unit 222, and the V matrix is configured based only on the value vectors V stored in the V cache storage unit 223.

[0085] Note that this is not limited to the case where another input sequence SQ (denoted as input sequence SQ') is input to the transformer 100 before the second input sequence SQ before S102. For example, in the transformer 100 to which the input sequence SQ' is input, another pair set (denoted as pair set 300') is generated by the processing of the self-attention layer 122, and index information (denoted as index information IDX') is acquired based on the pair set 300'. Then, in S102, the ANN search unit 231 may perform a search using the index information IDX'.

[0086] ​ Once the transformer 100 has finished processing the second input sequence SQ, the accumulation of the key vector K in the K cache storage unit 222 is completed, and the accumulation of the value vector V in the V cache storage unit 223 is completed. The processor 11 obtains the pair set 300 from the K cache storage unit 222 and the V cache storage unit 223 according to the information processing program PRG or other program (S103).

[0087] The processor 11 generates index information IDX according to the information processing program PRG or another program (S104), and stores the generated index information IDX in the SSD 13 (S105). Then, the operation corresponding to the second input sequence SQ is completed.

[0088] In the example shown in Figure 6, the information processing device 1 performed the processes S101 to S105. Some or all of the processes S101 to S105 may be performed by an information processing device other than the information processing device 1. For example, the processes S101 to S105 may be performed by a certain information processing device, and the generated index information IDX may be transferred from that certain information processing device to the SSD 13 provided by the information processing device 1.

[0089] Figure 7 is a flowchart showing an example of the operation of the information processing device 1 according to the first embodiment in response to the first input sequence SQ.

[0090] First, the processor 11 obtains the first input sequence SQ (S201). The processor 11 then executes the processing of the transformer 100 for the first input sequence SQ according to the information processing program PRG (S202). In S202, the index information IDX generated by the series of operations shown in Figure 6 is used.

[0091] The processor 11 outputs the output sequence generated by the processing of the transformer 100 for the first input sequence SQ (S203), and the operation corresponding to the first input sequence SQ is completed.

[0092] As described above, according to the first embodiment, the SSD 13 stores index information IDX which defines the graph structure of a directed graph GF in which each of the multiple key vectors K is considered a node. A value vector V is associated with each of the multiple key vectors K included in the index information IDX. The self-attention layer 122 performs the following operations each time a token is input as a self-attention input. That is, the self-attention layer 122 generates a query vector Q based on the input token (denoted as the target token) and the input position of the target token, generates a key vector K based on the target token and the input position of the target token, and generates a value vector V based on the target token. The self-attention layer 122 stores the generated key vector K in the K cache storage unit 222 and stores the generated value vector V in the V cache storage unit 223. The self-attention layer 122 performs a search using the ANN method with the index information IDX and the query vector Q to find the key vector K str and value vector V str The self-attention layer 122 obtains the query vector Q and the key vector K. str The attention score is obtained by a first calculation which includes calculating the inner product of the attention score and the K matrix containing all the key vectors K stored in the K cache storage unit 222. The self-attention layer 122 obtains the attention score and the value vector V str Then, a second calculation is performed to calculate the inner product with the V matrix containing all the value vectors V stored in the V cache storage unit 223. The self-attention layer 122 outputs the result of the second calculation.

[0093] Therefore, since the transformation 100 is processed using pre-generated key-value pairs, the time required for the transformation 100 to process is reduced. In other words, high-speed processing of the transformer-type neural network is achieved.

[0094] Furthermore, according to the first embodiment, the index information IDX is stored in the SSD13. The self-attention layer 122 then retrieves the key vector K from the SSD13 through a search. str and value vector V str To obtain and

[0095] Non-volatile storage devices like SSD13 generally have larger capacities and are less expensive than high-speed memory like RAM14. Therefore, it is possible to implement the processing of transformer100 using large-scale index information IDX at a low cost.

[0096] Furthermore, according to the first embodiment, the information processing device 1 inputs another input sequence SQ (i.e., a second input sequence SQ) to the transformer 100 before the first input sequence SQ. Key vector K str This is the key vector K generated in the self-attention layer 122 in response to the input of the second input sequence SQ to the transformer 100. The value vector V str This is the value vector V generated in the self-attention layer 122 in response to the input of the second input sequence SQ to the transformer 100.

[0097] Furthermore, the information processing device 1 obtains a pair set 300 from a plurality of key vectors K generated in the self-attention layer 122 in response to the input of the second input sequence SQ to the transformer 100, and a plurality of value vectors V generated in the self-attention layer 122 in response to the input of the second input sequence SQ to the transformer 100. Then, the information processing device 1 generates index information IDX based on the pair set 300.

[0098] Furthermore, some or all of the operations of acquiring the pair set 300 based on the second input sequence SQ, and generating the index information IDX based on the pair set 300, may be performed by a device other than the information processing device 1.

[0099] (Second embodiment) In the first embodiment, a predetermined number of key-value pairs (for example, one) from the pair set 300 are obtained by searching using the ANN method, and the obtained predetermined number of key-value pairs are used for processing in the self-attention layer 122. All key-value pairs included in the pair set 300 may be used for processing in the self-attention layer 122. In the second embodiment, a configuration in which all key-value pairs included in the pair set 300 are used for processing in the self-attention layer 122 will be described. In the second embodiment, the same matters as in the first embodiment will be omitted or described in a simplified manner.

[0100] Figure 8 shows an example of information stored in the SSD 13 according to the second embodiment.

[0101] SSD13 stores a pair set 300 generated based on a second input sequence SQ, instead of the index information IDX.

[0102] Figure 9 shows an example of the configuration of the self-attention layer 122a according to the second embodiment.

[0103] The self-attention layer 122a differs from the self-attention layer 122 in that it does not include the ANN search unit 231.

[0104] The self-attention layer 122a receives all key vectors K(key vector K) from SSD13 that are included in the pair set 300. str (referred to as a group) and all the value vectors V (value vector V) included in the pair set 300 str The self-attention layer 122a leads the group (referred to as the group). The key vector K str The group is input to the coupling unit 232. The self-attention layer 122a receives the value vector V str The group is input to the connecting unit 233.

[0105] The connecting portion 232 is key vector K str By concatenating the group with the group of key vectors K input from the K cache memory unit 222, the key vector K str A K matrix is ​​generated that includes the group and all the key vectors K stored in the K cache memory unit 222. The generated K matrix is ​​input to the first calculation unit 241.

[0106] The connecting portion 233 is the value vector V str By concatenating the group with the group of value vectors V input from the V cache storage unit 223, the value vector V str A V matrix is ​​generated that includes the group and all the value vectors V stored in the K cache storage unit 222. The generated V matrix is ​​input to the second calculation unit 242.

[0107] Thus, the SSD 13 stores the pair set 300 generated based on the second input sequence SQ, and the self-attention layer 122a may be configured to use all key-value sets contained in the pair set 300 for attention layer processing.

[0108] (Third embodiment) In the third embodiment, the processor 11 is configured to select the index information IDX to be used in the self-attention layer 122 from among a plurality of index information IDX. In the third embodiment, the same matters as in the first embodiment are omitted or described in a simplified manner.

[0109] Figure 10 shows an example of information stored in the SSD 13 according to the third embodiment.

[0110] SSD13 stores multiple index information IDX, each generated based on a different second input sequence SQ, instead of the index information IDX.

[0111] In the example shown in Figure 10, index information IDX1, index information IDX2, and index information IDX3 are stored. Index information IDX1 is index information obtained by graphing the pair sets generated based on the second input sequence SQ1. Index information IDX2 is index information obtained by graphing the pair sets generated based on the second input sequence SQ2, which is different from the second input sequence SQ1. Index information IDX3 is index information obtained by graphing the pair sets generated based on the second input sequence SQ3, which is different from both the second input sequence SQ1 and the second input sequence SQ2.

[0112] Figure 11 is a flowchart showing an example of the operation of the information processing device 1 according to the third embodiment in response to the first input sequence SQ.

[0113] First, the processor 11 obtains the first input sequence SQ (S201). Then, according to the information processing program PRG or another program, the processor 11 selects the index information IDX to be used from among the multiple index information IDX stored in the SSD 13 (S301).

[0114] For example, the processor 11 selects an index information IDX generated based on a second input sequence SQ that is most similar to the first input sequence SQ among multiple index information IDXs.

[0115] The method for selecting index information IDX is not limited to this. For example, metadata such as summaries may be associated with multiple index information IDX. The processor 11 may be configured to select index information IDX based on a comparison between the first input sequence SQ and the metadata associated with each index information IDX.

[0116] Alternatively, the processor 11 may be configured to select index information IDX based on arbitrary information in addition to, or instead of, the first input sequence SQ.

[0117] Furthermore, the number of IDX index information entries selected in S301 is not limited to one.

[0118] Note that S301 is an example of a selection process.

[0119] Following S301, the processor 11 performs the processing of the transformer 100 for the first input sequence SQ according to the information processing program PRG (S202). In S202, the index information IDX selected by the processing in S301 is used.

[0120] The processor 11 outputs the output sequence generated by the processing of the transformer 100 for the first input sequence SQ (S203), and the operation corresponding to the first input sequence SQ is completed.

[0121] Furthermore, the technology of the third embodiment can be used in conjunction with the technology of the second embodiment. That is, the explanation given in the third embodiment is valid even if the index information IDX is read as pair set 300.

[0122] As described above, according to the first, second, and third embodiments, the SSD13 has at least a key vector K str and value vector V strThe pair is stored. The self-attention layer 122 performs the following operations each time a token is input as a self-attention input. That is, the self-attention layer 122 generates a query vector Q based on the input token (denoted as the target token) and the input position of the target token, generates a key vector K based on the target token and the input position of the target token, and generates a value vector V based on the target token. The self-attention layer 122 stores the generated key vector K in the K cache storage unit 222 and stores the generated value vector V in the V cache storage unit 223. The self-attention layer 122 stores the query vector Q and the key vector K str The attention score is obtained by a first calculation which includes calculating the inner product of the attention score and the K matrix containing all the key vectors K stored in the K cache storage unit 222. The self-attention layer 122 obtains the attention score and the value vector V str Then, a second calculation is performed to calculate the inner product with the V matrix containing all the value vectors V stored in the V cache storage unit 223. The self-attention layer 122 outputs the result of the second calculation.

[0123] Therefore, high-speed processing by transformer-type neural networks is achieved.

[0124] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0125] 1 Information processing unit, 11 Processor, 12 Interface, 13 SSD, 14 RAM, 15 Bus, 100 Transformer, 110 Embedding layer, 120 Transformer block, 121,124,130 Normalization layer, 122,122a Self-attention layer, 123,126 Connecting layer, 125 Feedforward layer, 140 Linear layer, 201,202,203 Connected layer neural network, 211,212 Position encoding layer, 222 K cache memory unit, 223 V cache memory unit, 231 ANN search unit, 232,233 Concatenation unit, 241 First calculation unit, 242 Second calculation unit, 300,300' Pair set, IDX, IDX1, IDX2, IDX3, IDX' Index information, Q Query vector, K,K str Key vector, V,V str Value vector.

Claims

1. One or more computers to execute the instructions, One or more storage devices that store a first set of a first key and a first value, Equipped with, The one or more computers mentioned above execute the processing of a transformer-type neural network in the execution of the instruction. The transformer-type neural network is configured to include a self-attention layer, and in response to the input to the transformer-type neural network of a first input sequence having a network input at each of a plurality of input positions in input order, the self-attention layer receives a self-attention input for each of the plurality of input positions, and outputs an output sequence corresponding to the first input sequence based on the output from the self-attention layer corresponding to the self-attention input. The self-attention layer, each time a self-attention input is received, The process involves generating a query based on the self-attention input and a first input position which corresponds to the self-attention input among the plurality of input positions; generating a key based on the self-attention input and the first input position and storing the generated second key in one or more storage devices; generating a value based on the self-attention input and storing the generated second value in one or more storage devices; performing a first calculation to obtain an attention score by the inner product of the query and a key matrix containing the first key and the second key stored in one or more storage devices; performing a second calculation to calculate the inner product of the attention score and a value matrix containing the first value and the second value stored in one or more storage devices; and outputting the result of the second calculation. Configured to perform, Information processing system.

2. The one or more storage devices mentioned above include non-volatile storage devices. The first key and the first value are stored in the storage device. The information processing system according to claim 1.

3. The non-volatile storage device stores index information that defines the graph structure of a directed graph in which each of the multiple fourth keys is considered a node, and each of the multiple fourth keys is associated with a fourth value. The self-attention layer is configured to perform a search using the approximate nearest neighbor search method with the index information and the query. The first key is the fourth key obtained by the search among the plurality of fourth keys, The first value is the fourth value corresponding to the first key. The information processing system according to claim 2.

4. In the transformer-type neural network, a second input sequence is input before the first input sequence is input. The plurality of fourth keys are a plurality of keys generated by the self-attention layer in the processing of the transformer-type neural network for the second input sequence. Each of the aforementioned plurality of fourth keys corresponds to a fourth value which is a value generated by the self-attention layer in the processing of the second input sequence by the transformer-type neural network. The information processing system according to claim 3.

5. In the transformer-type neural network, a second input sequence is input before the first input sequence is input. The first key is a key generated by the self-attention layer in the processing of the transformer-type neural network for the second input sequence, The first value is a value generated by the self-attention layer in the processing of the transformer-type neural network for the second input sequence. The information processing system according to claim 1.

6. The aforementioned first set includes a plurality of second sets of third keys and third values, The one or more computers mentioned above perform a selection process in the execution of the instruction. The selection process is a process of selecting at least one second set from the plurality of second sets, The self-attention layer is configured to perform the first calculation using a third key included in the at least one second set selected by the selection process, and to perform the second calculation using a third value included in the at least one second set selected by the selection process. The information processing system according to claim 1.

7. A method for an information processing system comprising one or more computers, The above-mentioned one or more computers perform processing on a transformer-type neural network, and the transformer-type neural network includes a self-attention layer. The transformer-type neural network is input a first input sequence having a network input at each of the multiple input positions in the order of input, In response to the input of the first input sequence, the self-attention layer is provided with a self-attention input for each of the plurality of input positions, Each time the self-attention input is received by the self-attention layer, The process involves generating a query based on the self-attention input and a first input position which corresponds to the self-attention input among the plurality of input positions; reading the first key and the first value from one or more storage areas where the first key and the first value are stored; generating a key based on the self-attention input and the first input position; obtaining a value based on the self-attention input; performing a first calculation to obtain an attention score by the inner product of the query and a key matrix including the first key and the generated second key; performing a second calculation to calculate the inner product of the attention score and a value matrix including the first value and the generated second value; and outputting the result of the second calculation. To execute, The transformer-type neural network outputs an output sequence corresponding to the first input sequence based on the output from the self-attention layer, A method that includes this.

Images