Sparse attention method, apparatus, device, medium and product for retrieval enhancement generation

Abstract

The invention discloses a sparse attention method and device for retrieval enhancement generation, equipment, a medium and a product, and relates to the field of retrieval enhancement generation, the method comprises the following steps: pruning keys and values of all residual tokens to obtain keys and values of all selected tokens; obtaining a cache key matrix and a cache value matrix under the current dialogue round according to keys and values of tokens for user query, special tokens and cue word templates under the historical dialogue round and the current dialogue round as well as keys and values of starting tokens and selection tokens under the current dialogue round according to the keys and values of the tokens under the current dialogue round as well as keys and values of the tokens under the special tokens and the cue word templates under the current dialogue round; and processing the cache value matrix under the current dialogue round, the problem vector processed by the position embedding operation and the cache key matrix to obtain the context under the current dialogue round, and the application can solve the problems that the context length of a large language model is limited and the existing long context attention mechanism is not suitable for retrieving an enhanced generation scene.

Description

Technical Field

[0001] This application relates to the field of retrieval enhancement generation, and in particular to a sparse attention method, apparatus, device, medium, and product for retrieval enhancement generation. Background Technology

[0002] In recent years, Large Language Models (LLMs) have achieved remarkable success in the field of natural language processing. However, they still face significant limitations in specific domains or knowledge-intensive tasks. In particular, when the generated content involves knowledge beyond the scope of their training data, large language models are prone to "illusions," i.e., generating content that does not conform to reality. This phenomenon is particularly prominent in knowledge-intensive domains (such as medicine, law, and other fields requiring highly specialized knowledge).

[0003] As one method to address the "illusion" problem, Retrieval-Augmented Generation (RAG) has emerged. The main idea of ​​RAG is to retrieve information related to user input from external knowledge bases, providing additional reference knowledge for large language model generation and improving the reliability of the generated content. Its typical process includes: text slicing, vectorization, retrieval, and generation. In text slicing, the text in the original document is usually divided into several blocks according to certain rules. These blocks are then vectorized to obtain corresponding embedding representations, which are stored in a vector database. Before using the large language model for generation, the system first vectorizes the user input and retrieves the most similar vectors to the user input from the vector database based on vector similarity, thus obtaining documents related to the user input. Finally, these documents are used to assist the large language model in generating answers. The use of RAG has several advantages: firstly, it significantly reduces the occurrence of illusion phenomena in large language model generation; secondly, it improves the performance of large language models on tasks in specific domains; and thirdly, it obtains the latest information during large language model generation, thereby compensating for insufficient training data.

[0004] Despite the advantages mentioned above, retrieval-enhanced generation techniques also have significant drawbacks. Large language models typically have a limited context length, restricting the number of available retrieval results. In the generation phase, the more retrieval results a large language model can utilize, the higher its generation quality. However, limited by the inherent context length, large language models can only use a few top-ranked retrieval results as supplementary information to aid generation. This context length limitation primarily stems from the model's self-attention mechanism. While some research on long-context attention can extend the model's context length, these studies are mostly geared towards general dialogue scenarios, and their application in retrieval-enhanced generation reduces generation quality. This constitutes a significant obstacle to improving the effectiveness of retrieval-enhanced generation in long text scenarios.

[0005] Existing research on extending the context length of large language models can be categorized into four types: sparse attention, linear attention, hierarchical attention, and I / O-aware attention. While sparse attention, linear attention, and hierarchical attention effectively improve the context length of large language models, they all have significant limitations. Although I / O-aware attention in existing research can achieve near-infinite context length, it is not entirely suitable for retrieval enhancement generation of large models with long text. The input sequence in retrieval enhancement scenarios is a combination of user input and reference text (retrieval results), the latter being much longer than the former. Current techniques, while increasing the context length of large models, do not consider the characteristics of this type of input sequence, easily losing information from the reference text in the middle, resulting in unsatisfactory generation results. Therefore, researching long-context attention mechanisms suitable for retrieval enhancement generation is crucial. Summary of the Invention

[0006] The purpose of this application is to provide a sparse attention method, apparatus, device, medium, and product for retrieval enhancement generation, which can solve the problems of limited context length of large language models and the inadequacy of existing long context attention mechanisms for retrieval enhancement generation scenarios.

[0007] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a sparse attention method for retrieval enhancement generation, including: in the current dialogue turn, obtaining the input text sequence for retrieval enhancement generation in the current dialogue turn.

[0008] The input text sequence generated by the retrieval enhancement under the current dialogue round is segmented to obtain multiple tokens under the current dialogue round and the token type corresponding to each token; the token type is user query, special token, prompt word template or reference text.

[0009] Prune the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round; the remaining tokens in the current dialogue round are the tokens of the reference text type in the current dialogue round excluding the starting token in the current dialogue round; the starting token in the current dialogue round is the first n tokens of the reference text type in the current dialogue round; n is a positive integer.

[0010] Based on the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type under the historical dialogue rounds, as well as the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type, the keys of each starting token, and the keys of each selection token under the context and the current dialogue round, the cache key matrix for the current dialogue round is obtained; the historical dialogue rounds are all dialogue rounds before the current dialogue round.

[0011] Based on the values ​​of each token of type 'user query', 'special token', and 'prompt template' from the historical dialogue rounds, as well as the values ​​of each token of type 'user query', 'special token', 'prompt template', 'start token', and 'select token' from the context and the current dialogue round, a cache value matrix for the current dialogue round is obtained.

[0012] Perform position embedding operations on the question vector input by the user in the current dialogue round and the cache key matrix in the current dialogue round to obtain the processed question vector and the processed cache key matrix in the current dialogue round.

[0013] A self-attention mechanism is used to process the question vector, cache key matrix, and cache value matrix processed in the current dialogue round to obtain the context of the current dialogue round.

[0014] Secondly, this application provides a sparse attention apparatus for retrieval enhancement generation, comprising: an acquisition module for acquiring, in the current dialogue turn, the input text sequence for retrieval enhancement generation.

[0015] The word segmentation module is used to segment the input text sequence generated by the retrieval enhancement in the current dialogue round to obtain multiple tokens in the current dialogue round and the token type corresponding to each token; the token type is user query, special token, prompt word template or reference text.

[0016] The pruning module is used to prune the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round. The remaining tokens in the current dialogue round are the tokens of the reference text type in the current dialogue round, excluding the starting token in the current dialogue round. The starting token in the current dialogue round is the first n tokens of the reference text type in the current dialogue round, where n is a positive integer.

[0017] The cache key matrix determination module is used to obtain the cache key matrix for the current dialogue round based on the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type in the historical dialogue rounds, as well as the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type, the keys of each starting token, and the keys of each selection token in the context and the current dialogue round. The historical dialogue rounds refer to all dialogue rounds before the current dialogue round.

[0018] The cache value matrix determination module is used to obtain the cache value matrix for the current dialogue round based on the values ​​of each token of the user query type, the value of each token of the special token type, the value of each token of the prompt word template type in the historical dialogue rounds, as well as the values ​​of each token of the user query type, the value of each token of the special token type, the value of each token of the prompt word template type, the value of each starting token, and the value of each selection token in the context and the current dialogue round.

[0019] The position embedding operation module is used to perform position embedding operations on the user input question vector and the cache key matrix in the current dialogue round, so as to obtain the processed question vector and the processed cache key matrix in the current dialogue round.

[0020] The self-attention mechanism module is used to process the question vector, cache key matrix, and cache value matrix processed in the current dialogue round using a self-attention mechanism to obtain the context of the current dialogue round.

[0021] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sparse attention method for retrieving enhanced generation described above.

[0022] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sparse attention method for retrieval enhancement generation described above.

[0023] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the sparse attention method for retrieval enhancement generation described above.

[0024] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a sparse attention method, apparatus, device, medium and product for retrieving enhanced generation. In the historical rounds, the relevant information in the reference text has been extracted by the large model and integrated into the generated content, providing context information for the generation of the current round. This application only caches the key values ​​of each token of user query, special token and prompt word template in the historical rounds, and removes redundant historical reference text. It will not damage the generation quality of the current round, but will instead free up a large amount of occupied context length.

[0025] This application proposes a sparse attention mechanism that prunes the keys and values ​​of the remaining tokens in the current dialogue round while preserving the key values ​​of other types of tokens in the current dialogue round. This greatly reduces the loss of important information and, while expanding the context length of large models, can obtain useful information for long context reasoning in retrieval-enhanced generation scenarios, thereby improving the generation quality of large language models in retrieval-enhanced generation scenarios.

[0026] In summary, this application addresses the issues of limited context length in large language models and the inadequacy of existing long context attention mechanisms for retrieval enhancement generation scenarios. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a sparse attention method for retrieval enhancement generation, provided as an embodiment of this application.

[0029] Figure 2 This is a roadmap for sparse attention techniques used in retrieval enhancement generation, provided as an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of an attention pruning strategy provided in an embodiment of this application.

[0031] Figure 4 This is a schematic diagram of the functional modules of a sparse attention device for retrieval enhancement generation, provided in an embodiment of this application.

[0032] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] This application provides a method for retrieving enhanced sparse attention, as illustrated in an exemplary embodiment, such as... Figure 1 and Figure 2 As shown, the sparse attention method for retrieval enhancement generation includes: Step 201: In the current dialogue round, obtain the input text sequence for retrieval enhancement generation in the current dialogue round.

[0036] Step 202: Perform word segmentation on the input text sequence generated by the retrieval enhancement under the current dialogue round to obtain multiple tokens under the current dialogue round and the token type corresponding to each token; the token type is user query T_query, special token T_special, prompt word template T_prompt or reference text T_reference.

[0037] Step 203: Prune the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round; the remaining tokens in the current dialogue round are the tokens of the reference text type in the current dialogue round excluding the starting token in the current dialogue round; the starting token in the current dialogue round is the first n tokens of the reference text type in the current dialogue round; n is a positive integer.

[0038] Step 204: Based on the keys of each token of type queried by the user in the historical dialogue rounds, the keys of each token of type special token, the keys of each token of type prompt word template in the historical dialogue rounds, as well as the keys of each token of type queried by the user in the current dialogue round, the keys of each token of type special token, the keys of each token of type prompt word template in the current dialogue rounds, the keys of each starting token, and the keys of each selection token, the historical dialogue rounds are all dialogue rounds before the current dialogue round.

[0039] Step 205: Based on the values ​​of each token of type user query, each token of type special token, each token of type prompt word template in the historical dialogue rounds, as well as the values ​​of each token of type user query, each token of type special token, each token of type prompt word template, each starting token, and each selection token in the context and current dialogue round, obtain the cache value matrix for the current dialogue round.

[0040] Step 206: Perform position embedding operation on the question vector input by the user in the current dialogue round and the cache key matrix in the current dialogue round to obtain the processed question vector and the processed cache key matrix in the current dialogue round.

[0041] Step 207: Use a self-attention mechanism to process the question vector, cache key matrix, and cache value matrix processed in the current dialogue round to obtain the context of the current dialogue round.

[0042] In practical applications, the input text sequence generated by retrieval enhancement includes the user's original input, special tokens, prompt word templates, and reference text. When constructing the input text sequence generated by retrieval enhancement, several special markers are used to distinguish the user's original input, special tokens, prompt word templates, and reference text.

[0043] In another exemplary embodiment of this application, the input text sequence generated by retrieval enhancement in the current dialogue round is segmented to obtain multiple tokens and the token type corresponding to each token in the current dialogue round. Specifically, this includes: inputting the input text sequence generated by retrieval enhancement in the current dialogue round into a token segmenter to obtain multiple tokens and the token type corresponding to each token. When the token segmenter splits the input text sequence generated by retrieval enhancement into a token list, it simultaneously outputs a token type list, including the token type corresponding to each token. Specifically, the input text sequence generated by retrieval enhancement is divided into four parts: special tokens used during instruction fine-tuning training, system prompt word templates, user queries (i.e., original user input), and formatted reference text, and a unique type identifier (token type) is assigned to each part. For example, when the token list = [" <bos>", please, according to, the following, reference, content, answer, question, : what is the principle of quantum computing?, reference 1,... ], there is a token type list = [T_special, T_prompt,..., T_query,..., T_reference,... ].

[0044] In another example embodiment of the present application, before pruning the keys and values of each remaining token in the current dialogue turn to obtain the keys and values of each selected token in the current dialogue turn, the keys and values of each token in the current dialogue turn are calculated.

[0045] In another example embodiment of the present application, the cache key matrix in the current dialogue turn is obtained according to the keys of each token of the token type of the user query in the historical dialogue turn, the keys of each token of the token type of the special token, the keys of each token of the token type of the prompt word template, the context, and the keys of each token of the token type of the user query in the current dialogue turn, the keys of each token of the token type of the special token, the keys of each token of the token type of the prompt word template, the keys of each starting token, and the keys of each selected token, specifically including: obtaining the key matrix of the panoramic information in the current dialogue turn according to the keys of each token of the token type of the user query in the historical dialogue turn, the keys of each token of the token type of the special token, the keys of each token of the token type of the prompt word template, the context, and the keys of each token of the token type of the user query in the current dialogue turn, the keys of each token of the token type of the special token, and the keys of each token of the token type of the prompt word template.

[0046] The segment information key matrix in the current dialogue turn is obtained according to the keys of each starting token in the current dialogue turn.

[0047] The selection information key matrix in the current dialogue turn is obtained according to the keys of each selected token in the current dialogue turn.

[0048] The cache key matrix in the current dialogue turn is obtained according to the key matrix of the panoramic information in the current dialogue turn, the segment information key matrix, and the information key matrix.

[0049] In another exemplary embodiment of this application, a cached value matrix for the current dialogue round is obtained based on the values ​​of each token of type user query, each token of type special token, each token of type prompt word template under the historical dialogue rounds, as well as the values ​​of each token of type user query, each token of type special token, each token of type prompt word template, each starting token, and each selection token under the context and the current dialogue round. Specifically, this includes obtaining a value matrix of panoramic information under the current dialogue round based on the values ​​of each token of type user query, each token of type special token, each token of type prompt word template under the historical dialogue rounds, as well as the values ​​of each token of type user query, each token of type special token, and each token of type prompt word template under the context and the current dialogue round.

[0050] The fragment information value matrix for the current dialogue round is obtained based on the values ​​of each starting token in the current dialogue round.

[0051] The selection information value matrix for the current dialogue round is obtained based on the values ​​of each selection token.

[0052] The cached value matrix for the current dialogue round is obtained by using the value matrix of the panoramic information, the value matrix of the fragment information, and the value matrix of the information in the current dialogue round.

[0053] In another exemplary embodiment of this application, the keys and values ​​of the remaining tokens in the current dialogue round are pruned to obtain the keys and values ​​of the selected tokens in the current dialogue round. Specifically, this includes selecting the first n tokens among all tokens of type reference text in the current dialogue round to obtain the starting tokens in the current dialogue round.

[0054] Based on the starting tokens in the current dialogue round, obtain the remaining tokens in the current dialogue round.

[0055] Perform top-k pruning on the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round.

[0056] In practical applications, if the dialogue round is only one, the special token, prompt word template, and the key and value corresponding to the user query will be cached. , , The entire panoramic information is preserved. Preserving their original relative order, the representation is as follows: The key and value of the beginning portion of the reference text (consisting of the key and value of the starting token) are preserved as fragment information. The remaining keys and values ​​(consisting of the keys and values ​​of the remaining tokens). pass Select to perform key value The key matrix is ​​represented as panoramic information. Represented as a value matrix of panoramic information, It consists of the keys of all tokens of a special token type. It consists of the keys of all tokens of the prompt word template type. It consists of the keys of all tokens of the user query type. It consists of the values ​​of all tokens of a special token type. It consists of the values ​​of all tokens of the prompt word template type. It consists of the values ​​of all tokens of the user query type. The key matrix representing fragment information consists of the keys of each starting token; The value matrix representing fragment information consists of the values ​​of each starting token; It is composed of the keys of each remaining token. It consists of the values ​​of all remaining tokens. If the dialogue round is i-th, where i is greater than 1, then the keys of each token of type queried by the user, the keys of each token of type special token, the keys of each token of type prompt word template from the previous dialogue rounds, and the context are concatenated in order, and then appended to... The preceding section concatenates the values ​​of tokens of type user query, tokens of type special token, tokens of type prompt word template, and the context from the historical dialogue rounds in that order, and then appends them to the previous section. Front.

[0057] In practical applications, top-k pruning is performed on the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round. Specifically, top-k pruning is handled by the large language model. In top-k pruning, each attention layer of the large language model independently performs top-k selection. Each layer will obtain k position indices. Then, through a multi-head attention mechanism, k' position indices are selected from all position indices as the final pruning result.

[0058] The execution process of each attention layer is as follows: calculate the current query vector. That is, the user-input question vector and The similarity score matrix; select the k highest-scoring position indices, expressed by the formula: .

[0059] Finally, based on the k' position indices, from and Selected from and The selected position and its adjacent preset length range of context tokens are retained. Specifically: .

[0060] in, () represents the index of the first k largest values. express The transpose of . express The Middle The key for the location token (for selecting the token). express The Middle The value of the location token, The key matrix represents the selection tokens, consisting of the keys of each selection token. This represents the value matrix of the selection tokens, which consists of the values ​​of each selection token.

[0061] In practical applications, the cache key matrix is ​​specifically represented as: .

[0062] In practical applications, the cached value matrix is ​​specifically represented as follows: .

[0063] In practical applications, steps 203 to 205 are the dynamic key-value cache selection method, and steps 203 to 207 are processed by the large language model.

[0064] In practical applications, the question vector input by the user in the current dialogue round... and the cache key matrix in the current dialogue round Perform the position embedding operation PE() to obtain the processed question vector for the current dialogue round. And the cache key matrix processed in the current dialogue round Specifically, this includes: if the current dialogue round is the first, the self-attention calculation adopts a delayed position embedding strategy, applying position embedding to the key-value matrix only after the dynamic key-value caching is complete, and using the original key matrix without position information for top-k selection calculation, specifically represented as follows: .

[0065] In practical applications, a self-attention mechanism is adopted. ( ) the question vector processed in the current dialogue round, the cache key matrix processed in the current dialogue round, and the cache value matrix in the current dialogue round. Process the data to obtain the context of the current dialogue turn. Specifically: .

[0066] If the current dialogue round is the i-th round, where i is greater than 1, then the token from the previous dialogue round is obtained based on the key matrix of the panoramic information, the key matrix of the fragment information, the key matrix of the selection information, the value matrix of the panoramic information, the value matrix of the fragment information, and the value matrix of the selection information. This token is then concatenated to the cached key matrix and cached value matrix of the current dialogue round. Figure 3 As shown, a subsequent delayed position embedding strategy and self-attention mechanism are implemented. Figure 3 The yellow area represents the overall information, the blue area represents the fragment information, the gray area represents the top-k selected parts to be retained, the dashed area represents the top-k selected parts to be discarded, and the red area represents the token from the previous dialogue round, which does not participate in pruning. It represents the result generated by the large model after implementing the pruning strategy.

[0067] This application implements a dialogue history cache cleanup strategy for multi-turn dialogues. Specifically, before the current dialogue turn is generated, the key-value cache of reference text from previous turns is removed; only special tokens, user queries, prompt word templates, and model-generated content from previous turns are retained as global information; dynamic caching selection is still performed for reference text in the current turn. In previous turns, relevant information from the reference text has already been extracted by the large model and integrated into the generated content, providing contextual information for the current turn's generation. Therefore, the dialogue history cache cleanup strategy, by removing redundant historical reference text, does not impair the generation quality of the current turn. On the contrary, it frees up a significant amount of occupied context length, allowing the current turn's generation to use more context length to retain key information from the current turn.

[0068] This application treats input components such as special tokens, user input, and prompt word templates as panoramic information and retains corresponding key-value caches for panoramic and fragment information. At the same time, it implements a dynamic key-value cache selection strategy for other components. In addition, it uses a dialogue history cache cleanup strategy for input sequences of multi-turn dialogues to further improve the available context length.

[0069] This application specifically relates to a sparse attention method for retrieval-enhanced generation based on token types. The large language model in this application prunes the key-value cache (KV Cache) using a dynamic key-value cache selection method and a dialogue history cache cleanup method, based on the input token list and token types. The retained portion of the KV Cache is concatenated in relative order and self-attention is calculated. This addresses the problems of limited context length in large language models, the inadequacy of existing long-context attention mechanisms for retrieval-enhanced generation scenarios, reliance on fixed pruning rules leading to a lack of flexibility, failure to consider the characteristics of the model input sequence in RAG scenarios, the tendency to discard important contextual information during pruning leading to a decrease in model generation performance, and the significant drop in generation performance due to excessive context length occupied by reference text in the dialogue history.

[0070] In summary, this application proposes a more scientific and precise sparse attention mechanism. By performing targeted pruning on key-value caches of different types of tokens, it optimizes the KV Cache pruning strategy. While extending the context length of large models, it can obtain useful information for long context reasoning in retrieval-enhanced generation scenarios, significantly improving the generation quality in retrieval-enhanced generation scenarios.

[0071] Based on the same inventive concept, this application also provides a sparse attention apparatus for retrieving and enhancing sparse attention generation, which implements the aforementioned sparse attention method for retrieval and enhancement generation. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the sparse attention apparatus for retrieval and enhancement generation provided below can be found in the limitations of the sparse attention method for retrieval and enhancement generation described above, and will not be repeated here.

[0072] In one exemplary embodiment, such as Figure 4 As shown, a sparse attention device for retrieval enhancement generation is provided, comprising: an acquisition module, used to acquire the input text sequence for retrieval enhancement generation in the current dialogue turn.

[0073] The word segmentation module is used to segment the input text sequence generated by the retrieval enhancement in the current dialogue round to obtain multiple tokens in the current dialogue round and the token type corresponding to each token; the token type is user query, special token, prompt word template or reference text.

[0074] The pruning module is used to prune the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round. The remaining tokens in the current dialogue round are the tokens of the reference text type in the current dialogue round, excluding the starting token in the current dialogue round. The starting token in the current dialogue round is the first n tokens of the reference text type in the current dialogue round, where n is a positive integer.

[0075] The cache key matrix determination module is used to obtain the cache key matrix for the current dialogue round based on the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type in the historical dialogue rounds, as well as the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type, the keys of each starting token, and the keys of each selection token in the context and the current dialogue round. The historical dialogue rounds refer to all dialogue rounds before the current dialogue round.

[0076] The cache value matrix determination module is used to obtain the cache value matrix for the current dialogue round based on the values ​​of each token of the user query type, the value of each token of the special token type, the value of each token of the prompt word template type in the historical dialogue rounds, as well as the values ​​of each token of the user query type, the value of each token of the special token type, the value of each token of the prompt word template type, the value of each starting token, and the value of each selection token in the context and the current dialogue round.

[0077] The position embedding operation module is used to perform position embedding operations on the user input question vector and the cache key matrix in the current dialogue round, so as to obtain the processed question vector and the processed cache key matrix in the current dialogue round.

[0078] The self-attention mechanism module is used to process the question vector, cache key matrix, and cache value matrix processed in the current dialogue round using a self-attention mechanism to obtain the context of the current dialogue round.

[0079] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores sparse attention data for retrieving augmented generation. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for retrieving augmented generation sparse attention.

[0080] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0081] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0082] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0083] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0084] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0085] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0086] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0087] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.< / bos>

Claims

1. A sparse attention method for retrieval enhancement generation, characterized in that, The sparse attention method used for retrieving enhanced generation includes: In the current dialogue round, obtain the input text sequence generated by the retrieval enhancement in the current dialogue round; The input text sequence generated by the retrieval enhancement under the current dialogue round is segmented to obtain multiple tokens under the current dialogue round and the token type corresponding to each token; the token type is user query, special token, prompt word template or reference text; Prune the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round; the remaining tokens in the current dialogue round are the tokens of the reference text type in the current dialogue round excluding the starting token in the current dialogue round; the starting token in the current dialogue round is the first n tokens of the reference text type in the current dialogue round; n is a positive integer; Based on the keys of each token of type queried by the user in the historical dialogue rounds, the keys of each token of type special token, the keys of each token of type prompt word template in the historical dialogue rounds, as well as the keys of each token of type queried by the user in the historical dialogue rounds, the keys of each token of type special token, the keys of each token of type prompt word template in the historical dialogue rounds, the keys of each starting token, and the keys of each selection token in the historical dialogue rounds, we obtain the cache key matrix for the current dialogue round; the historical dialogue rounds are all dialogue rounds before the current dialogue round; Based on the values ​​of each token of type 'user query', 'special token', and 'prompt template' in the historical dialogue rounds, as well as the values ​​of each token of type 'user query', 'special token', 'prompt template', 'start token', and 'select token' in the context and current dialogue round, a cache value matrix for the current dialogue round is obtained. Perform position embedding operations on the question vector input by the user in the current dialogue round and the cache key matrix in the current dialogue round to obtain the processed question vector and the processed cache key matrix in the current dialogue round. A self-attention mechanism is used to process the question vector, cache key matrix, and cache value matrix processed in the current dialogue round to obtain the context of the current dialogue round.

2. The sparse attention method for retrieval enhancement generation according to claim 1, characterized in that, The input text sequence generated by the retrieval enhancement in the current dialogue round is segmented to obtain multiple tokens for the current dialogue round and the token type corresponding to each token, specifically including: The input text sequence generated by the retrieval enhancement in the current dialogue round is input into the tokenizer to obtain multiple tokens and the token type corresponding to each token.

3. The sparse attention method for retrieval enhancement generation according to claim 1, characterized in that, Prune the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round, specifically including: Select the first n tokens of the reference text type from all tokens in the current dialogue round to obtain the starting tokens for the current dialogue round; Based on the starting tokens in the current dialogue round, obtain the remaining tokens in the current dialogue round; Perform top-k pruning on the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round.

4. The sparse attention method for retrieval enhancement generation according to claim 1, characterized in that, Based on the keys of tokens of type 'user query', 'special token', and 'prompt template' from the historical dialogue rounds, as well as the keys of tokens of type 'user query', 'special token', 'prompt template', 'start token', and 'select token' from the context and the current dialogue round, a cache key matrix for the current dialogue round is obtained, specifically including: Based on the keys of each token of the user query type, the keys of each token of the special token type, and the keys of each token of the prompt word template type in the historical dialogue rounds, as well as the keys of each token of the user query type, the keys of each token of the special token type, and the keys of each token of the prompt word template type in the context and the current dialogue round, the key matrix of the panoramic information in the current dialogue round is obtained. The fragment information key matrix for the current dialogue round is obtained based on the keys of each starting token in the current dialogue round. The selection information key matrix for the current dialogue round is obtained based on the keys of each selection token in the current dialogue round. The cache key matrix for the current dialogue round is obtained by using the key matrix of the panoramic information, the key matrix of the fragment information, and the key matrix of the information in the current dialogue round.

5. The sparse attention method for retrieval enhancement generation according to claim 1, characterized in that, Based on the values ​​of tokens of type user query, tokens of type special token, tokens of type prompt word template from previous dialogue rounds, as well as the values ​​of tokens of type user query, tokens of type special token, tokens of type prompt word template, start token, and selection token from the context and current dialogue round, a cached value matrix for the current dialogue round is obtained, specifically including: Based on the values ​​of each token of type user query, each token of type special token, and each token of type prompt word template in the historical dialogue rounds, as well as the values ​​of each token of type user query, each token of type special token, and each token of type prompt word template in the context and the current dialogue round, the value matrix of panoramic information in the current dialogue round is obtained. The fragment information value matrix for the current dialogue round is obtained based on the values ​​of each starting token in the current dialogue round. The selection information value matrix for the current dialogue round is obtained based on the values ​​of each selection token in the current dialogue round. The cached value matrix for the current dialogue round is obtained by using the value matrix of the panoramic information, the value matrix of the fragment information, and the value matrix of the information in the current dialogue round.

6. The sparse attention method for retrieval enhancement generation according to claim 1, characterized in that, Before pruning the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round, the following steps are also included: Calculate the key and value of each token in the current dialogue round.

7. A sparse attention apparatus for retrieving enhanced generation, characterized in that, The sparse attention device for retrieving enhanced generation includes: The acquisition module is used to acquire the input text sequence generated by the retrieval enhancement in the current dialogue round. The word segmentation module is used to segment the input text sequence generated by the retrieval enhancement in the current dialogue round to obtain multiple tokens in the current dialogue round and the token type corresponding to each token; the token type is user query, special token, prompt word template or reference text; The pruning module is used to prune the keys and values ​​of the remaining tokens in the current dialogue round to obtain the keys and values ​​of the selected tokens in the current dialogue round. The remaining tokens in the current dialogue round are the tokens of the reference text type in the current dialogue round, excluding the starting token in the current dialogue round. The starting token in the current dialogue round is the first n tokens of the reference text type in the current dialogue round, where n is a positive integer. The cache key matrix determination module is used to obtain the cache key matrix for the current dialogue round based on the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type in the historical dialogue rounds, as well as the keys of each token of the user query type, the keys of each token of the special token type, the keys of each token of the prompt word template type, the keys of each starting token, and the keys of each selection token in the context and the current dialogue round. The historical dialogue rounds are all dialogue rounds before the current dialogue round. The cache value matrix determination module is used to obtain the cache value matrix for the current dialogue round based on the values ​​of each token of the user query type, the value of each token of the special token type, the value of each token of the prompt word template type in the historical dialogue rounds, as well as the values ​​of each token of the user query type, the value of each token of the special token type, the value of each token of the prompt word template type, the value of each starting token, and the value of each selection token in the context and the current dialogue round. The position embedding operation module is used to perform position embedding operations on the user input question vector and the cache key matrix in the current dialogue round to obtain the processed question vector and the processed cache key matrix in the current dialogue round. The self-attention mechanism module is used to process the question vector, cache key matrix, and cache value matrix processed in the current dialogue round using a self-attention mechanism to obtain the context of the current dialogue round.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the sparse attention method for retrieving augmented generation as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the sparse attention method for retrieval enhancement generation as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the sparse attention method for retrieval enhancement generation as described in any one of claims 1-6.

Images