Attention processing method and device in reasoning process, equipment and medium

Abstract

The invention provides an attention processing method and device in a reasoning process, equipment and a medium. The method is applied to a large language model and comprises the steps that an input token input into the large language model is obtained through an input sequence; querying an embedded vector corresponding to the input token from the embedded table, and respectively querying a query vector, a key vector and a value vector corresponding to the input token from the query table, the key table and the value table; and utilizing the embedded vector, the query vector, the key vector and the value vector corresponding to the input token to obtain an implicit vector corresponding to the input token, and using the implicit vector for reasoning to obtain a reasoning result. The reasoning cost can be reduced, and the reasoning efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a method, device, equipment and medium for processing attention in a reasoning process. Background Art

[0002] The currently popular large language models are usually based on the Transformer architecture, in which attention processing is a very important part, and its performance directly affects the overall reasoning performance based on the large language model. The process of attention processing mainly includes: the input sequence is converted into multiple input tokens (elements) after being parsed by the tokenizer, where the input sequence can be an input text. Then, when performing attention processing, it is necessary to query the embedding table based on each input token to obtain the embedding vector corresponding to each input token. Then, the embedding vector corresponding to each input token is used for real-time calculation to obtain the query vector, key vector and value vector corresponding to each input token. The query vector, key vector and value vector are further used to obtain the attention vector of attention processing, and the attention vector and input vector are further used to obtain the hidden vector corresponding to the input token. In this way, the query vector, key vector and value vector corresponding to all input tokens need to be calculated and stored according to the input sequence during the entire reasoning process, which makes the calculation amount and the access to parameters very large, resulting in high reasoning cost and slow reasoning speed. Summary of the invention

[0003] In view of this, the embodiments of the present application are committed to providing a method, device, equipment and medium for attention processing in the reasoning process to reduce reasoning delay.

[0004] In a first aspect, a method for processing attention in a reasoning process is provided, which is applied to a large language model, and comprises: obtaining an input token input to the large language model using an input sequence, wherein the input sequence comprises a word sequence corresponding to an input text; obtaining an embedding vector corresponding to the input token by querying from an embedding table, and obtaining a query vector, a key vector and a value vector corresponding to the input token by querying from a query table, a key table and a value table respectively; wherein the embedding table comprises a correspondence between an embedding vector and each token, the query table comprises a correspondence between one or more tokens and the query vector, the key table comprises a correspondence between one or more tokens and the key vector, the value table comprises a correspondence between one or more tokens and the value vector, and the one or more tokens comprise the input token; obtaining a latent vector corresponding to the input token by using the embedding vector, the query vector, the key vector and the value vector corresponding to the input token, and the latent vector is used for inference to obtain an inference result corresponding to the input text.

[0005] In a second aspect, a device for processing attention in an inference process is provided, which is applied to a large language model. The device for processing attention in the inference process includes: an acquisition module, which is used to obtain an input token input into the large language model using an input sequence, and the input sequence includes a word sequence corresponding to the input text; a query module, which is used to query from an embedding table to obtain an embedding vector corresponding to the input token, and query from a query table, a key table and a value table to obtain a query vector, a key vector and a value vector corresponding to the input token respectively; wherein the embedding table includes a correspondence between the embedding vector and each token, the query table includes a correspondence between more than one token and the query vector, the key table includes a correspondence between more than one token and the key vector, the value table includes a correspondence between more than one token and the value vector, and the more than one token includes the input token; a processing module, which is used to obtain a latent vector corresponding to the input token using the embedding vector, the query vector, the key vector and the value vector corresponding to the input token, and the latent vector is used to infer and obtain an inference result corresponding to the input text.

[0006] According to a third aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the method described in the first aspect above.

[0007] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.

[0008] According to a fifth aspect, a program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the method described in the first aspect.

[0009] In the reasoning process involved in this embodiment, there is no need to calculate the corresponding query vector, key vector and value vector based on the embedded vector for the input token. Instead, a table lookup method is adopted, that is, the query vector, key vector and value vector corresponding to the input token are respectively queried from the query table, key table and value table. Compared with the real-time calculation method in the related technology, this table lookup method can reduce the reasoning cost, speed up the reasoning process, and improve the reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flowchart of the attention processing method in the reasoning process provided by an embodiment of the present application;

[0011] Figure 2 It is a schematic diagram for determining the query table, key table and value table;

[0012] Figure 3 is a schematic diagram of the output latent vector;

[0013] Figure 4 is a schematic diagram of an attention processing device in a reasoning process provided by an embodiment of the present application;

[0014] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] In this embodiment, the attention processing method in the reasoning process may include: first using the input sequence to obtain an input token input into the large language model; then querying from the embedding table to obtain the embedding vector corresponding to the input token, and querying from the query table, key table and value table to obtain the query vector, key vector and value vector corresponding to the input token; the embedding table contains the correspondence between the embedding vector and each token, the query table contains the correspondence between one or more tokens and the query vector, the key table contains the correspondence between one or more tokens and the key vector, the value table contains the correspondence between one or more tokens and the value vector, and the one or more tokens include the input token; using the embedding vector, query vector, key vector and value vector corresponding to the input token, the latent vector corresponding to the input token is obtained, and the latent vector is used to infer the inference result corresponding to the input text, which can reduce the inference cost compared with the related technology, speed up the inference process, and improve the inference speed.

[0017] For ease of understanding, this article first introduces the application scenarios applicable to the attention processing method in the reasoning process provided by this application, which is applied to a large language model (Large Language Model, LLM). A large language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Optionally, the large language model generally adopts a Transformer architecture.

[0018] In the embodiment of the present application, the large language model can be used in various application scenarios, such as question-and-answer application scenarios, text continuation application scenarios, text classification application scenarios, etc. Specifically, the question-and-answer application scenario can be a customer service scenario, such as an after-sales service robot.

[0019] Figure 1 This is a flowchart of the attention processing method in the reasoning process provided by the embodiment of the present application, and the application of the solution to an electronic device is used as an example for explanation. Figure 1 As shown in , the method may include the following steps:

[0020] Step S101: using an input sequence to obtain an input token to be input into a large language model, the input sequence includes a word sequence corresponding to an input text.

[0021] Step S102: Obtain an embedding vector corresponding to the input token by querying from the embedding table, and respectively obtain a query vector, a key vector and a value vector corresponding to the input token by querying from the query table, the key table and the value table; wherein the embedding table contains the correspondence between the embedding vector and each token, the query table contains the correspondence between one or more tokens and the query vector, the key table contains the correspondence between one or more tokens and the key vector, and the value table contains the correspondence between one or more tokens and the value vector.

[0022] Step S103: Using the embedding vector, query vector, key vector and value vector corresponding to the input token, the latent vector corresponding to the input token is obtained. The latent vector is used to infer the inference result corresponding to the input text.

[0023] It can be seen from the above process that when performing reasoning, the embodiment of the present application searches for the query vector, key vector and value vector corresponding to the input token from the constructed query table, key table and value table, which can reduce the reasoning cost, speed up the reasoning process and improve the reasoning speed compared to the related technology.

[0024] The above step S101, i.e., "obtaining an input token input into a large language model using an input sequence, wherein the input sequence includes a word sequence corresponding to an input text", is described in detail below in conjunction with an embodiment.

[0025] In this embodiment, after acquiring the input sequence, the electronic device uses the input sequence to determine the input token input into the large language model.

[0026] In one example, using an input sequence to determine an input token input to a large language model may include: passing the input sequence through a tokenizer to obtain an input token input to the large language model. The electronic device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. Among them, the tokenizer may include, but is not limited to, the following methods: Byte-level Byte-Pair Encoding (BBPE), WordPiece, SentencePiece, etc.

[0027] Here, the input sequence may include a word sequence corresponding to the input text, wherein the word sequence may be a series of words arranged in a specific order, wherein the word may be a phrase in English, or a character or phrase in Chinese.

[0028] The following is a detailed description of “obtaining an input token input into a large language model using a word sequence corresponding to an input text” in conjunction with an embodiment.

[0029] Here, token is the basic data unit processed by the large language model. A token can be a word, a part of a word (subword), or even a character, depending on the process of splitting the input text into word sequences.

[0030] After obtaining multiple tokens of the input sequence, an input token input to the large language model can be determined from the multiple tokens based on different cache mechanisms (ie, for caching keys and values).

[0031] In one example, the input token input to the large language model in the current round may include a new token obtained by reasoning based on the large language model in the previous round and an input token input to the large language model in the previous round.

[0032] For example, for an input sequence, the Transformer architecture will output an answer (length N), and perform N inference processes in the process of outputting an answer. That is, the Transformer architecture only outputs one token in one inference, and the output token (that is, the new token obtained by inference based on the large language model in the previous round) will be concatenated with the input token input to the large language model in the previous round, and then used as the input token input to the large language model in the current round.

[0033] The above step S102 is described in detail below in conjunction with an embodiment.

[0034] In this embodiment, for each input token, the corresponding embedding vector, query vector, key vector and value vector can be determined. In this embodiment of the present application, by looking up a table, the embedding vector corresponding to the input token is queried from the embedding table, and the query vector, key vector and value vector corresponding to the input token are queried from the query table, key table and value table respectively.

[0035] In deep learning and natural language processing (NLP), an embedding table is a lookup table that maps discrete input indices (i.e. token numbers) to continuous embedding vectors. The following is the detailed process of inputting a token to look up the embedding table:

[0036] Token to index mapping: Before passing the input tokens to the large language model, they need to be converted to corresponding indices (i.e. token numbers). This mapping process is usually done through a predefined vocabulary, where each token in the vocabulary corresponds to a unique index.

[0037] Mapping of index to embedding vector: Once the input token is converted into an index, these indices can be used to query the corresponding embedding vector in the embedding table. Specifically, the index is used as the row number to find the corresponding row in the embedding table, which contains the embedding vector of the token.

[0038] The embedding table can be constructed based on the following steps:

[0039] (1) Extract a preset vocabulary containing all the words to be embedded from the training data;

[0040] (2) The embedding dimension is the length of the embedding vector, which is usually a hyperparameter. The choice of this value will affect the performance and computational cost of large language models. Common embedding dimension sizes are 50, 100, 200, 300, etc.

[0041] (3) Initialize an embedding table whose size is [the number of words in the preset vocabulary, the embedding dimension]. The initial value of this embedding table can be generated randomly or using a pre-trained embedding (such as Word2Vec, etc.);

[0042] (4) Build a mapping from words to indices so that the word’s embedding vector can be looked up in the embedding table.

[0043] In the embodiment of the present application, by defining a preset vocabulary, embedding dimensions, initializing an embedding matrix, constructing a vocabulary-to-index mapping, and using an embedding layer in a large language model, an embedding representation of a word can be achieved. If pre-trained embeddings are used, these embeddings can be loaded and filled into the embedding table, and the use of pre-trained embeddings can significantly improve the performance of the large language model.

[0044] Here, the embedding table maps each word in the preset vocabulary into a fixed-size vector space, and these vectors can capture the semantic relationship between words, that is, the embedding table maps the token (or token number) to a high-dimensional dense vector. These vectors are usually called embedding vectors, which can capture the semantic and syntactic relationship between tokens.

[0045] In one example, an embedding vector corresponding to the input token is obtained by querying the embedding table according to the input token.

[0046] The embedding table is used to store the embedding vectors of each token in the preset vocabulary. In a large language model, the embedding table is usually a large matrix, in which each row corresponds to the embedding vector of a token. The size of this matrix is ​​determined by the size of the preset vocabulary and the dimension of the embedding vector. When the large language model processes the input token, it converts the input token into the corresponding embedding vector.

[0047] Embedding vectors are high-dimensional (e.g., hundreds or thousands of dimensions) vectors that capture the semantic meaning of words, sentences, or even entire documents. In large language models, embedding vectors are the basis for large language models to understand the semantics of input sequences.

[0048] The embedding vector is obtained by looking up the corresponding row in an embedding table with a dimension of [vocab_size, hidden_size] for the input token; where vocab_size is the number of words contained in the preset vocabulary. That is, the embedding vector corresponds to one of the rows in the embedding table, and there are vocab_size different possibilities. Since the query vector, key vector, and value vector are obtained based on the embedding vector, the embedding vector is normalized and then multiplied by the query parameter matrix, key parameter matrix, and value parameter matrix respectively. Then the query vector, key vector, and value vector are theoretically the same as the embedding vector, and there are also vocab_size different possibilities.

[0049] Based on the above theory, the query table, key table and value table can be obtained by processing the embedded vectors (corresponding rows) in the embedded table. Specifically, the following three methods can be used:

[0050] The first method is to pre-process each embedding vector in the embedding table to obtain a query table, a key table, and a value table.

[0051] Determine the query vector, key vector, and value vector of different values ​​of vocab_size in advance based on the embedding table, and cache the query vector, key vector, and value vector according to the row to obtain the query table (Query table), key table (Key table), and value table (Value table), and establish the corresponding relationship between each row and the token in the query table, key table, and value table. Next, obtain the embedding vector and query vector, key vector, and value vector corresponding to the input token by querying the input token, eliminating the process of online calculation of vocab_size query vectors, key vectors, and value vectors. The embodiment of the present application pre-caches the query vector, key vector, and value vector partially or completely, so that the amount of calculation and memory access are reduced during reasoning, thereby speeding up the reasoning process.

[0052] exist Figure 2 In the process, the embedding table is processed by a method such as RMS normalization to obtain the normalized embedding vectors corresponding to each token (i.e., the normalized table); next, the normalized embedding vectors corresponding to each token are multiplied by the query parameter matrix to obtain the query vectors corresponding to each token; the normalized embedding vectors corresponding to each token are multiplied by the key parameter matrix to obtain the key vectors corresponding to each token; the normalized embedding vectors corresponding to each token are multiplied by the value parameter matrix to obtain the value vectors corresponding to each token, so as to construct the query table, key table, and value table respectively. Among them, the embedding table, query table, key table, and value table are all matrices with vocab_size rows, each row of which corresponds to a token.

[0053] Optionally, the construction of the above query table, key table and value table can be executed in parallel or in sequence in any order.

[0054] It should be noted that the embedded table can be calculated as a whole or row by row to determine the query table, key table and value table, and cache them. Calculating the embedded table as a whole means inputting the entire embedded table and calculating the query table, key table and value table at one time; calculating the embedded table row by row means inputting the value of a row of the embedded table each time to calculate the value of a row of the output query table, key table and value table.

[0055] It should be further noted that whether to calculate the embedded table as a whole or to calculate the embedded table row by row can be determined according to the operating performance of the electronic device.

[0056] Here, the operating performance of an electronic device refers to the quality and efficiency of the electronic device in completing a specific task under specific conditions, including the capabilities, reliability, stability, response time, throughput, number of concurrent users, resource utilization, and other aspects of the electronic device.

[0057] The second method: If there is no corresponding query vector, key vector, and value vector in the query table, key table, and value table during the reasoning process, the results are calculated in real time and stored in the query table, key table, and value table so that other reasoning processes can directly query the query table, key table, and value table for the same input token.

[0058] Specifically, if the query table does not yet contain the query vector corresponding to the input token (for example, the row corresponding to the input token in the query table is marked as "uncached"), then the embedding vector corresponding to the input token in the embedding table is obtained, and the query vector corresponding to the input token is obtained using the query parameter matrix and the obtained embedding vector, and the query table is updated using the query vector corresponding to the input token (at this time, the row corresponding to the input token in the query table is marked as "cached").

[0059] If the key table does not yet contain the key vector corresponding to the input token (for example, the row corresponding to the input token in the key table is marked as "uncached"), then obtain the embedding vector corresponding to the input token in the embedding table, use the key parameter matrix and the obtained embedding vector to obtain the key vector corresponding to the input token, and use the key vector corresponding to the input token to update the query table (at this time, the row corresponding to the input token in the key table is marked as "cached").

[0060] If the value table does not yet contain the value vector corresponding to the input token (for example, the row corresponding to the input token in the value table is marked as "not cached"), then obtain the embedding vector corresponding to the input token in the embedding table, use the value parameter matrix and the obtained embedding vector to obtain the value vector corresponding to the input token, and use the value vector corresponding to the input token to update the query table (at this time, the row corresponding to the input token in the value table is marked as "cached").

[0061] If the query table, key table and value table contain the query vector, key vector and value vector corresponding to the input token, the query vector, key vector and value vector corresponding to the input token are directly queried.

[0062] It should be noted that the methods for determining the query vector, key vector, and value vector involved in the second and third methods are similar to those in the first method. For example, in the second method, after the embedding vector corresponding to the input token is normalized, the embedding vector obtained after normalization is multiplied by the query parameter matrix to obtain the query vector corresponding to the input token. The rest is similar and will not be elaborated in the following.

[0063] The third method: Pick out hot word tokens from the embedding table, and pre-build the values ​​of the corresponding rows in the query table, key table, and value table based on these hot word tokens, and cache them.

[0064] Specifically, obtain the embedding vector corresponding to the hot word token in the embedding table; use the query parameter matrix and the embedding vector corresponding to the hot word token to obtain the query vector corresponding to the hot word token, and use the query vector corresponding to the hot word token to build a query table; use the key parameter matrix and the embedding vector corresponding to the hot word token to obtain the key vector corresponding to the hot word token, and use the key vector corresponding to the hot word token to build a key table; and use the value parameter matrix and the embedding vector corresponding to the hot word token to obtain the value vector corresponding to the hot word token, and use the value vector corresponding to the hot word token to build a value table.

[0065] It should be noted that when performing reasoning, the query vector, key vector and value vector corresponding to the input token are obtained from the query table, key table and value table pre-calculated based on the embedding table. This method has the fastest reasoning speed, but because the query table, key table and value table pre-calculated based on the embedding table need to be cached in advance, the cache capacity required is relatively large.

[0066] When performing reasoning, the second method and the third method mentioned above can be combined. For example, after using the third method to establish a query table, a key table, and a value table based on hot words, if the query vector, key vector, and value vector corresponding to the input token cannot be queried from the query table, the key table, and the value table, then the second method mentioned above can be used to calculate the query vector, key vector, and value vector according to the embedding vector corresponding to the input token and update the query table, key table, and value table. In this way, the cache occupancy is not large at the beginning. As the reasoning proceeds, the cache occupancy gradually increases, and the reasoning speed gradually becomes faster.

[0067] It should be further noted that the second or third method mentioned above is more cache-saving than the first method.

[0068] The above step 103, i.e., "obtaining a latent vector corresponding to the input token by using the embedding vector, query vector, key vector and value vector corresponding to the input token", is described in detail below in conjunction with an embodiment.

[0069] In the embodiment of the present application, a latent vector corresponding to the input token is obtained based on the query vector, key vector and value vector corresponding to the input token, and the embedding vector corresponding to the input token. The latent vector is used to infer the inference result corresponding to the input text.

[0070] Among them, the latent vector refers to the internal representation obtained by the Transformer architecture after a series of transformations (such as self-attention mechanism, feedforward neural network, etc.) when processing the input token. These latent vectors can capture the characteristics of the input token and play a key role in the subsequent Transformer architecture calculations. In the Transformer encoder, the latent vector is usually used to represent the potential features of the input token, and these features are used to generate the output sequence (ie, the inference result) in the subsequent decoding process. Among them, the scenarios in which the Transformer architecture is applied may include but are not limited to text classification scenarios, image segmentation, and image recognition scenarios. The latent vector is also used to obtain corresponding text classification results, image segmentation results, image recognition results, etc. It should be noted that other scenarios in which the first-level self-attention of the Transformer architecture can be applied can also be included in the protection scope of the embodiments of the present application, and will not be repeated here.

[0071] In an embodiment of the present application, the query vector, key vector and value vector corresponding to the input token can be used to obtain an attention vector; the attention vector and the embedding vector corresponding to the input token are added element by element to obtain a latent vector corresponding to the input token.

[0072] like Figure 3 As shown in , the embedding table has been converted into a query table, a key table, and a value table in advance and cached. The corresponding row is searched from the cached embedding table, query table, key table, and value table according to the input token to obtain the corresponding embedding vector, query vector, key vector, and value vector.

[0073] The query vector, key vector and value vector corresponding to each input token constitute the query matrix (denoted as Q), key matrix (denoted as K) and value matrix (denoted as V) respectively. After rotating the position of Q, K and V respectively, the transpose of Q and K is multiplied. Figure 3 The "QK matrix multiplication" in the above formula is used to reduce the vector size to a certain extent and then perform softmax processing. Figure 3 The "scaled softmax" in the matrix is ​​used to obtain the attention score. Then the attention score is multiplied by the value matrix to obtain a matrix. The calculation process of the matrix is ​​shown in the following formula (1). Then the matrix is ​​linearly transformed (corresponding to Figure 3The attention matrix attn_out is obtained by multiplying the attn_out matrix in , which is essentially composed of the attention vectors of each input token. The latent vector of the input token can be obtained by adding the attention vector of the input token to the embedding vector element by element.

[0074]

[0075] The above-mentioned attention processing method provided in the embodiment of the present application is applicable to the first-layer self-attention processing in a large language model. The first-layer self-attention processing is usually an attention processing performed on the input token, and the subsequent layers of self-attention processing are all based on the latent vector output by the first-layer self-attention processing.

[0076] It should be noted that the first layer of self-attention is not the only component of the Transformer architecture. The Transformer architecture also includes other key components, such as Feed-Forward Neural Network and Positional Encoding. These components work together to enable the Transformer architecture to achieve excellent performance during reasoning.

[0077] Combination of the above Figure 1 , describes the method embodiment of the present application in detail, and the following is combined with Figure 4 , describes the device embodiment of the present application in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0078] Figure 4 FIG. 1 is a schematic diagram of the structure of an attention processing device in the reasoning process provided by an embodiment of the present application. Figure 4As shown, the attention processing device in the reasoning process provided by the embodiment of the present application is applied to a large language model, and the attention processing device in the reasoning process includes an acquisition module 401, a query module 402 and a processing module 403. Among them, the acquisition module 401 is used to obtain an input token input to the large language model using an input sequence, and the input sequence includes a word sequence corresponding to the input text; the query module 402 is used to query the embedding vector corresponding to the input token from the embedding table, and query the query vector, key vector and value vector corresponding to the input token from the query table, key table and value table respectively; wherein the embedding table contains the corresponding relationship between the embedding vector and each token, the query table contains the corresponding relationship between more than one token and the query vector, the key table contains the corresponding relationship between more than one token and the key vector, and the value table contains the corresponding relationship between more than one token and the value vector; the processing module 403 is used to obtain the hidden vector corresponding to the input token using the embedding vector, query vector, key vector and value vector corresponding to the input token, and the hidden vector is used to infer the reasoning result corresponding to the input text.

[0079] In one example, the lookup table, the key table, and the value table are obtained using the embedding vectors corresponding to more than one token in the embedding table.

[0080] In one embodiment, the attention processing device further includes: a construction module (not shown in the figure), which is used to obtain a query vector corresponding to each token using the query parameter matrix and the embedding vector corresponding to each token in the embedding table, and construct a query table using the query vector corresponding to each token;

[0081] The construction module is further used to obtain the key vector corresponding to each token using the key parameter matrix and the embedding vector corresponding to each token in the embedding table, and construct the key table using the key vector corresponding to each token; and

[0082] The construction module is also used to obtain the value vector corresponding to each token by using the embedding vector corresponding to each token after normalization in the value parameter matrix and the embedding table, and construct the value table by using the value vector corresponding to each token.

[0083] In one embodiment, the query module 402 is further used to obtain the embedding vector corresponding to the hot word token in the embedding table.

[0084] A construction module is used to obtain a query vector corresponding to the hot word token using the query parameter matrix and the embedding vector corresponding to the hot word token, and to construct a query table using the query vector corresponding to the hot word token;

[0085] The construction module is further used to obtain the key vector corresponding to the hot word token using the key parameter matrix and the embedding vector corresponding to the hot word token, and to construct a key table using the key vector corresponding to the hot word token; and

[0086] The construction module is also used to obtain the value vector corresponding to the hot word token using the value parameter matrix and the embedding vector corresponding to the hot word token, and to construct a value table using the value vector corresponding to the hot word token.

[0087] In one embodiment, the query module 402 is also used to obtain the embedding vector corresponding to the input token in the embedding table if it is determined that the query table does not yet contain the query vector corresponding to the input token, and trigger the construction module to use the query parameter matrix and the obtained embedding vector to obtain the query vector corresponding to the input token, and update the query table using the query vector corresponding to the input token.

[0088] The query module 402 is further used to obtain the embedding vector corresponding to the input token in the embedding table if it is determined that the key table does not yet contain the key vector corresponding to the input token, and trigger the construction module to obtain the key vector corresponding to the input token using the key parameter matrix and the obtained embedding vector, and update the query table using the key vector corresponding to the input token;

[0089] The query module 402 is also used to obtain the embedding vector corresponding to the input token in the embedding table if it is determined that the value table does not yet contain the value vector corresponding to the input token, and trigger the construction module to use the value parameter matrix and the obtained embedding vector to obtain the value vector corresponding to the input token, and use the value vector corresponding to the input token to update the query table.

[0090] In one embodiment, the processing module 403 is specifically used to: obtain an attention vector using a query vector, a key vector, and a value vector corresponding to an input token; and add the attention vector and the embedding vector corresponding to the input token element by element to obtain a latent vector corresponding to the input token.

[0091] Below, reference Figure 5 To describe an electronic device according to an embodiment of the present application. Figure 5 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application.

[0092] like Figure 5 As shown, the electronic device includes one or more processors 501 and a memory 502 .

[0093] The processor 501 may be one or more central processing units (CPUs) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and at least one of the units may control other components in the electronic device to perform desired functions.

[0094] The memory 502 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may run the program instructions to implement functions such as the attention processing method in the reasoning process of each embodiment of the present application mentioned above. Various contents such as embedded tables, query tables, key tables, and value tables may also be stored in the computer-readable storage medium.

[0095] In one example, the electronic device may further include: an input device 503 and an output device 504, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0096] The input device 503 may include, for example, a keyboard, a mouse, etc.

[0097] The output device 504 can output various information to the outside, including inference results, etc. The output device 504 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0098] Of course, to simplify, Figure 5 Only some of the components in the electronic device related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0099] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the attention processing method in the reasoning process according to various embodiments of the present application described above in this specification.

[0100] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server.

[0101] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the attention processing method in the reasoning process according to various embodiments of the present application described above in this specification.

[0102] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0103] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0104] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0105] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0106] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0107] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

[0108] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for attention processing during reasoning, applied to large language models, where The attention processing method comprises: Obtaining an input token input into the large language model using an input sequence, wherein the input sequence includes a word sequence corresponding to an input text; Obtain an embedding vector corresponding to the input token by querying from the embedding table, and respectively obtain a query vector, a key vector, and a value vector corresponding to the input token by querying from the query table, the key table, and the value table; wherein the embedding table includes a correspondence between the embedding vector and each token, the query table includes a correspondence between one or more tokens and the query vector, the key table includes a correspondence between the one or more tokens and the key vector, and the value table includes a correspondence between the one or more tokens and the value vector, and the one or more tokens include the input token; The embedded vector, query vector, key vector and value vector corresponding to the input token are used to obtain a latent vector corresponding to the input token, and the latent vector is used to infer an inference result corresponding to the input text.

2. The method according to claim 1, wherein: The query table, key table and value table are obtained by using the embedding vectors corresponding to the one or more tokens in the embedding table.

3. The method according to claim 2, wherein: Before obtaining the input token input into the large language model by using the input sequence, the method further includes: Using the query parameter matrix and the embedding vector corresponding to each token in the embedding table, a query vector corresponding to each token is obtained, and the query table is constructed using the query vector corresponding to each token; Using the key parameter matrix and the embedding vector corresponding to each token in the embedding table, a key vector corresponding to each token is obtained, and the key table is constructed using the key vector corresponding to each token; and The value parameter matrix and the embedding vector corresponding to each token in the embedding table after normalization are used to obtain the value vector corresponding to each token, and the value table is constructed using the value vector corresponding to each token.

4. The method according to claim 2, wherein: Before obtaining an input token input into the large language model using the input sequence, the method further includes: Get the embedding vector corresponding to the hot word token in the embedding table; Using the query parameter matrix and the embedding vector corresponding to the hot word token, obtain the query vector corresponding to the hot word token, and construct the query table using the query vector corresponding to the hot word token; Using the key parameter matrix and the embedding vector corresponding to the hot word token, obtain the key vector corresponding to the hot word token, and construct the key table using the key vector corresponding to the hot word token; and The value parameter matrix and the embedding vector corresponding to the hot word token are used to obtain the value vector corresponding to the hot word token, and the value table is constructed using the value vector corresponding to the hot word token.

5. The method according to claim 2 or 4, wherein: The method further comprises at least one of the following: If the query table does not yet contain the query vector corresponding to the input token, obtain the embedding vector corresponding to the input token in the embedding table, use the query parameter matrix and the obtained embedding vector to obtain the query vector corresponding to the input token, and update the query table using the query vector corresponding to the input token; If the key table does not yet contain the key vector corresponding to the input token, obtain the embedding vector corresponding to the input token in the embedding table, use the key parameter matrix and the obtained embedding vector to obtain the key vector corresponding to the input token, and update the query table using the key vector corresponding to the input token; If the value table does not yet contain the value vector corresponding to the input token, obtain the embedding vector corresponding to the input token in the embedding table, use the value parameter matrix and the obtained embedding vector to obtain the value vector corresponding to the input token, and use the value vector corresponding to the input token to update the query table.

6. The method according to claim 1, characterized in that The obtaining the latent vector corresponding to the input token by using the embedding vector, the query vector, the key vector and the value vector corresponding to the input token comprises: Using the query vector, key vector and value vector corresponding to the input token, an attention vector is obtained; The attention vector and the embedding vector corresponding to the input token are added element by element to obtain the latent vector corresponding to the input token.

7. An attention processing device in the reasoning process, applied to a large language model, wherein: The attention processing device comprises: An acquisition module, configured to obtain an input token input into the large language model using an input sequence, wherein the input sequence includes a word sequence corresponding to an input text; A query module, used to query the embedding vector corresponding to the input token from the embedding table, and query the query vector, key vector and value vector corresponding to the input token from the query table, key table and value table respectively; wherein the embedding table contains the corresponding relationship between the embedding vector and each token, the query table contains the corresponding relationship between one or more tokens and the query vector, the key table contains the corresponding relationship between the one or more tokens and the key vector, the value table contains the corresponding relationship between the one or more tokens and the value vector, and the one or more tokens include the input token; A processing module is used to obtain a latent vector corresponding to the input token using an embedding vector, a query vector, a key vector and a value vector corresponding to the input token, wherein the latent vector is used to infer an inference result corresponding to the input text.

8. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 6 by running the program in the memory.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 6.

Images