Retrieval enhancement generation method and system based on dynamic pipeline

Abstract

The invention discloses a retrieval enhancement generation method and system based on a dynamic assembly line, and belongs to the crossing field of solid-state storage and large language models.The method comprises the steps that a pre-established text corpus is retrieved through a preset retrieval method, so that top-k texts most similar to query content input by a user are obtained from the text corpus to serve as retrieval results, putting the retrieval result into a result set; the retrieval method comprises multiple iterations, and a part of text most similar to query content can be retrieved in each iteration; in the process of retrieving the text corpus through the retrieval method, pre-fetching part of the text from the result set for multiple times and putting the part of the text into an assembly line pool pre-established in a memory; and extracting the text from the pipeline pool for multiple times and providing the text to the large language model, calculating corresponding keys and values by the large language model, and filling a KV cache of the large language model with a calculation result for use in a decoding stage of the large language model. According to the method, the delay of knowledge retrieval can be masked, so that the response time of a retrieval enhancement generation system is shortened.

Description

Technical Field

[0001] The present invention belongs to the intersection field of solid-state storage and large language models, and more specifically, relates to a dynamic pipeline-based retrieval enhancement generation method and system. Background Art

[0002] Large language models have demonstrated remarkable capabilities in understanding and generating human language. However, their performance is limited by the limitations of training datasets, which are manifested in challenges such as hallucinations, outdated knowledge, and difficulty in tracing the reasoning process. To address these issues, retrieval-augmented generation (RAG) has emerged as a promising solution that enriches user input by retrieving relevant information from external databases, thereby improving the generation quality of large language models. In this process, knowledge retrieval represented by approximate nearest neighbor search (ANNS) has received widespread attention as a key technology that balances retrieval accuracy and speed. However, as the size of databases expands to tens of billions, it becomes impractical to store knowledge retrieval indexes represented by approximate nearest neighbor search completely in DRAM, prompting knowledge retrieval algorithms represented by approximate nearest neighbor search based on other storage devices to become a research hotspot.

[0003] However, knowledge retrieval faces efficiency bottlenecks when integrated into a retrieval enhancement generation system. Taking a specific knowledge retrieval system, DiskANN (Disk-based Approximate Nearest Neighbor), as an example, by comparing the performance of the DRAM-based knowledge retrieval system FAISS and the knowledge retrieval system DiskANN on the OpenWebText and Wikipedia datasets, it can be found that the retrieval delay of DiskANN significantly affects the overall response time of the retrieval enhancement generation system, that is, the entire process time (TTFT) from user input to knowledge retrieval completion, and then to the generation of the first word by the large language model. Although DiskANN optimizes the performance of graph-based knowledge retrieval, the growth of database size, the high demand for external knowledge of large language models, and the expansion of the search scope of graph indexes have all exacerbated retrieval delays.

[0004] In general, when implementing a retrieval enhancement generation system based on storage devices, how to reduce the overall response time is an urgent problem to be solved. Summary of the invention

[0005] In view of the defects and improvement needs of the prior art, the present invention provides a retrieval enhancement generation method and system based on dynamic pipeline, which aims to cover up the delay of knowledge retrieval and reduce the response time of the retrieval enhancement generation system.

[0006] To achieve the above object, according to one aspect of the present invention, a dynamic pipeline-based search enhancement generation method is provided, comprising:

[0007] Knowledge retrieval: A pre-established text corpus is searched through a preset retrieval method to obtain the top-k texts that are most similar to the query content entered by the user as retrieval results, and the retrieval results are placed in a result set; the retrieval method includes multiple iterations, and each iteration retrieves the part of the text that is most similar to the query content from the text corpus;

[0008] Pre-fetching of retrieval results: In the process of retrieving a text corpus through a retrieval method, some text is pre-fetched from the result set multiple times and put into a pipeline pool pre-established in memory;

[0009] Pre-filling: Take text from the pipeline pool multiple times and provide it to the large language model so that the large language model can calculate the corresponding keys and values, and fill the calculation results into the KV cache of the large language model for use in the decoding stage of the large language model;

[0010] Among them, the text corpus is stored on the hard disk; top-k is a preset positive integer.

[0011] Furthermore, the frequency of pre-fetching text from the result set decreases as the number of iterations in the knowledge retrieval process increases.

[0012] Furthermore, the nth operation of pre-fetching text from the result set occurs at the Opt n iterations, then,

[0013]

[0014] Among them, Opt n+1 Indicates the number of iterations when the n+1th prefetch occurs; PP.size indicates the number of texts in the pipeline pool; δ indicates the preset increment, δ>0; Balancer n Indicates the balance factor after the nth prefetch is completed; Indicates rounding down.

[0015] Furthermore, the amount of pre-fetched text from the result set is positively correlated with the stability of the result set.

[0016] Furthermore, the number of texts to be pre-fetched from the result set for the nth time is MaxNum n for:

[0017]

[0018] Stability indicates the number of texts in the result set that have not changed position compared with the previous prefetch. Balancern-1 Indicates the balance factor after the n-1th prefetch is completed; D is a preset positive integer.

[0019] Further,

[0020] Balancer n =(Stability-p*Unstability+Balancer n-1 ) / s

[0021] Among them, Unstability indicates the number of texts that have changed their positions in the result set and are prefetched into the pipeline pool compared with the previous prefetch; p indicates the penalty factor, and p∈[1,10]; s indicates the smoothing factor, and s>1.

[0022] Furthermore, before taking text from the pipeline pool and providing it to the large language model each time, it also includes: deleting unstable text in the pipeline pool from the pipeline pool;

[0023] The unstable text refers to the text whose position in the result set changes during the retrieval iteration.

[0024] Furthermore, the dynamic pipeline-based retrieval enhancement generation method provided by the present invention also includes: in the process of the large language model calculating the corresponding keys and values ​​and filling the calculation results into the KV cache, if it is found that the text taken out from the pipeline pool contains erroneous text, it is determined whether the amount of data filled into the KV cache exceeds a preset threshold. If so, the large language model continues to calculate the keys and values ​​of the currently taken out text, and stores the calculation results in the KV cache. After the calculation of the currently taken out text is completed, the key and value calculation results corresponding to the erroneous text in the KV cache are deleted; otherwise, the data in the KV cache is deleted, and after deleting the erroneous text from the currently taken out text, the large language model recalculates the keys and values ​​of each remaining text and fills them into the KV cache.

[0025] Further, taking text from the pipeline pool and providing it to the large language model includes: predicting corresponding pre-filling delays when taking different amounts of text from the pipeline pool and taking text from the pipeline pool according to the amount with the minimum pre-filling delay, and providing it to the large language model;

[0026] The pre-fill delay is the time from when the first text to be retrieved enters the pipeline pool to when the large language model completes the calculation of keys and values ​​for all retrieved texts and fills the corresponding calculation results into the KV cache.

[0027] According to another aspect of the present invention, there is provided a dynamic pipeline-based retrieval enhancement generation system, comprising: a knowledge retrieval module, a pre-fetching module and a pre-filling module;

[0028] The knowledge retrieval module is used to retrieve a pre-established text corpus through a preset retrieval method to obtain the top-k texts most similar to the query content input by the user as retrieval results, and put the retrieval results into a result set; the retrieval method includes multiple iterations, and each iteration retrieves the part of the text most similar to the query content from the text corpus;

[0029] A pre-fetch module is used to pre-fetch part of the text from the result set and put it into a pipeline pool pre-established in memory during the process of retrieving the text corpus through the retrieval method;

[0030] The pre-filling module is used to take text from the pipeline pool and provide it to the large language model so that the large language model can calculate the corresponding keys and values, and fill the calculation results into the KV cache of the large language model for use in the decoding stage of the large language model;

[0031] Among them, the text corpus is stored on the hard disk; top-k is a preset positive integer.

[0032] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0033] (1) The present invention introduces a pipeline pool to pre-fetch retrieval results during the iterative process of knowledge retrieval, and then returns part of the retrieval results in advance to start the pre-filling stage of the large language model. Compared with the traditional retrieval enhancement generation system that starts the pre-filling stage of the large language model after the knowledge retrieval is completed, the present invention effectively conceals the delay of knowledge retrieval and reduces the overall response time of retrieval enhancement generation.

[0034] (2) The recall rate of knowledge retrieval increases with the number of iterations, which satisfies the requirement that it increases at a faster rate in the initial iteration stage and then increases at a slower rate. In the process of knowledge retrieval, the present invention controls the frequency of pre-fetching text from the result set to decrease as the number of knowledge retrieval iterations increases, thereby realizing a recall-aware pre-fetching. When the recall rate increases rapidly, pre-fetching is performed at a higher frequency, and when the recall rate increases slowly, pre-fetching is performed at a lower frequency. This reduces the overall response delay while maintaining a low additional recall loss, thereby ensuring the accuracy of the final retrieval enhancement generation. In a further preferred embodiment, a nonlinear fitting function is constructed for the pre-fetching timing. The fitting function comprehensively considers the real-time pre-fetching progress and the recall rate. The pre-fetching timing is determined based on the fitting function, and the pre-fetching speed can be accurately and dynamically adjusted according to the knowledge retrieval pre-fetching recall rate.

[0035] (3) The more stable the text in the result set is, the greater the possibility that the result set contains correct results. Based on this, each time the present invention pre-fetches text from the result set, the number of pre-fetched texts is positively correlated with the stability of the result set. That is to say, the more stable the result set is, the more text is pre-fetched from it. Therefore, the number of pre-fetched texts can be automatically adjusted to balance the pre-fetching speed and the recall rate. In a further preferred embodiment, a quantitative index of the stability of the result set is given based on the change of the text position in the result set, and a calculation expression for the number of files pre-fetched each time is further given. This expression comprehensively considers the stability and the recall rate of the result set, and can accurately calculate the most appropriate pre-fetching number for each pre-fetch.

[0036] (4) Based on the quantitative index of the stability of the result set, the present invention provides a calculation expression for the balance factor for balancing the recall rate and the pre-fetching speed, which serves as the basis for the calculation expression of the pre-fetching timing and the pre-fetching quantity, so that the determined pre-fetching timing and the pre-fetching quantity can more accurately balance the pre-fetching speed and the recall rate, thereby ensuring that the retrieval enhancement generation has a higher overall performance.

[0037] (5) Before taking text from the pipeline pool and providing it to the large language model, the present invention will first delete unstable text from the pipeline pool, that is, text whose position in the result set has changed during the retrieval iteration process. This can prevent the large language model from generating erroneous results due to unstable text, thereby ensuring the accuracy of the final knowledge retrieval results.

[0038] (6) During the pre-filling process of the large language model, the present invention will detect in real time whether the text taken out from the pipeline pool contains errors, and if errors are contained, a corresponding error correction strategy will be adopted according to the progress of the pre-filling. Specifically, if an error is found in the early stage of pre-filling, the data in the existing KV cache will be deleted and the key value calculation and KV cache filling operations will be re-executed; if an error is found in the late stage of pre-filling, the erroneous result will be deleted from the KV cache after the current pre-filling process is completed, thereby ensuring that the error can be corrected in a shorter time.

[0039] (7) When taking text from the pipeline pool to start the pre-filling stage of the large language model, the present invention will predict the pre-filling delays corresponding to different pipeline granularities in advance to determine the pipeline granularity that minimizes the pre-filling delay, thereby achieving a dynamic balance of the pipeline granularity and further reducing the overall response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a dynamic pipeline-based search enhancement generation method provided by an embodiment of the present invention;

[0041] Figure 2A schematic diagram comparing the overall response time of different search enhancement generation methods provided by the embodiments of the present invention; wherein (a) is the overall response time of the search enhancement generation method provided by the embodiments of the present invention, and (b) is the overall response time of the traditional search enhancement generation method;

[0042] Figure 3 A schematic diagram of calculating the quantitative index of stability of a result set provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of an error correction strategy provided by an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of dynamic adjustment of pipeline granularity provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0047] Before explaining the technical solution of the present invention in detail, a process of generating text by a large language model (LLM) is briefly introduced.

[0048] The process of generating text by a large language model mainly involves two stages: pre-filling and decoding. The pre-filling stage is mainly used to prepare the initial context, providing a basis for the model to generate subsequent text; the decoding stage is the process in which the model gradually generates subsequent text based on the context prepared in the pre-filling stage. The main operations performed in the two stages are as follows:

[0049] In the pre-filling stage, the user provides an initial text as a prompt, such as a question, sentence, or paragraph. This text will be encoded into a series of tokens. Token is the basic unit used by the model when processing text, depending on the model's word segmentation strategy; the large language model processes all input tokens, generates keys and values, and stores them in the KV cache. These keys and values ​​are used for calculations in the subsequent decoding stage; the probability distribution of the last token is calculated as the starting point for generating the next token.

[0050] In the decoding phase, the model uses autoregression to generate a new token at a time. Each time a new token is generated, the model takes all previous tokens as input; samples a new token from the initial probability distribution in the pre-filling phase as the next input; uses the new token for forward propagation, updates the KV cache, generates new keys and values; generates a new probability distribution for sampling the next token. Repeat the above steps until the maximum length of the generation is reached or the stopping condition is met.

[0051] When knowledge retrieval is integrated into the retrieval enhancement generation system, a knowledge retrieval system based on a text corpus is established in the storage device, and knowledge retrieval methods represented by approximate nearest neighbor search are used to perform knowledge retrieval on the query content input by the user, and the top-k texts most similar to the user input content are found and put into the result set. In traditional retrieval enhancement systems, the pre-filling phase of the large language model is started after the knowledge retrieval process is completed, which makes the retrieval delay significantly affect the overall response time of the retrieval enhancement generation system.

[0052] In order to break through the performance bottleneck faced by knowledge retrieval when it is integrated into a retrieval enhancement generation system, the present invention provides a retrieval enhancement generation method and system based on a dynamic pipeline. The overall idea is to construct a pipeline retrieval enhancement generation execution paradigm, improve efficiency by overlapping the execution time of knowledge retrieval and large language models, thereby jointly optimizing knowledge retrieval and large language models from a system level, and reducing the negative impact of knowledge retrieval represented by approximate nearest neighbor search.

[0053] Specifically, during the knowledge retrieval process, the present invention pre-fetches some results and feeds them to the large language model in advance for pre-filling, thereby shortening the overall response time without significantly reducing the accuracy. On this basis, the present invention further combines the execution characteristics of knowledge retrieval and the large language model, optimizes the pre-fetching timing and pre-fetching quantity, and formulates corresponding error correction strategies and pipeline granularity dynamic adjustment strategies.

[0054] The following are examples.

[0055] Embodiment 1:

[0056] A dynamic pipeline-based retrieval enhancement generation method, such as Figure 1 As shown, including:

[0057] Knowledge retrieval: A pre-established text corpus is searched through a preset retrieval method to obtain the top-k texts that are most similar to the query content entered by the user as retrieval results, and the retrieval results are placed in a result set; the retrieval method includes multiple iterations, and each iteration retrieves the part of the text that is most similar to the query content from the text corpus;

[0058] Pre-fetching of retrieval results: In the process of retrieving a text corpus through a retrieval method, some text is pre-fetched from the result set multiple times and put into a pipeline pool pre-established in memory;

[0059] Pre-filling: Take text from the pipeline pool multiple times and provide it to the large language model so that the large language model can calculate the corresponding keys and values, and fill the calculation results into the KV cache of the large language model for use in the decoding stage of the large language model;

[0060] Among them, the text corpus is stored on the hard disk; top-k is a preset positive integer.

[0061] During the knowledge retrieval process, this embodiment pre-fetches retrieval results from the result set, puts them into a pipeline pool established in the memory, and fetches text from the pipeline pool for use in the pre-filling stage of the large language model. The KV cache in the pre-filling stage of the large language model is calculated in combination with the user's own query to achieve retrieval enhancement generation. This implements a pipeline system with the knowledge retrieval system as the producer, the pre-filling module as the consumer, and the pipeline pool as the pipeline buffer pool, and achieves the time overlap of some stages of knowledge retrieval pre-fetching and some stages of pre-filling performed on the CPU. The overall response time of the traditional retrieval enhancement generation and this embodiment are as follows: Figure 2 As shown in (a) and (b), by comparison, it can be seen that the present invention overlaps part of the time of knowledge retrieval and large language model pre-filling stages by introducing a pipeline, which is equivalent to covering up part of the delay of knowledge retrieval and can effectively reduce the overall response time. The partial overlap of knowledge retrieval tasks and pre-filling saves the time overhead of B1.

[0062] The partial overlap of knowledge retrieval tasks and pre-filling in the pipeline saves the time overhead of B1.

[0063] It should be noted that in the traditional search enhancement generation system, the knowledge retrieval algorithm will be executed according to the default parameters until the search queue converges, while this embodiment interrupts when the knowledge retrieval obtains top-m and top-n (top-m+top-n=top-k), which is equivalent to terminating the knowledge retrieval task in advance. Compared with the traditional search enhancement generation, the overall response time is further shortened. The early termination of the knowledge retrieval task saves the time overhead of B2 while ensuring the quality of data generation.

[0064] The saving of time overhead of B1 and B2 makes this embodiment have greater performance advantages than the traditional retrieval enhancement generation system.

[0065] In addition, this embodiment pre-fetches the knowledge retrieval results in multiple times, which saves the running time of pre-fetching all the results at one time due to the characteristics of the knowledge retrieval itself.

[0066] In order to balance the pre-fetching speed and recall accuracy as much as possible while reducing the overall response time, this embodiment proposes a recall-aware knowledge retrieval pre-fetching strategy to determine the timing and amount of pre-fetching text according to the quality of the user's query request text and the generated text, so as to obtain suitable text and send it to the pipeline pool cache. The recall-aware knowledge retrieval pre-fetching strategy specifically includes the following aspects:

[0067] (1) The time to perform knowledge retrieval prefetching is determined based on the amount of text stored in the pipeline pool in real time.

[0068] Since the recall rate of knowledge retrieval represented by approximate nearest neighbor search increases with the number of iterations at a faster rate in the initial iteration stage and then increases at a slower rate, accordingly, the dynamic adjustment of the pre-fetching frequency follows the principle: increase the pre-fetching frequency in the early stage of the knowledge retrieval iteration represented by approximate nearest neighbor search, and reduce the pre-fetching frequency in the later stage. Based on this principle, the nth operation of pre-fetching text from the result set occurs at the Opt. n iterations, the timing of the nth prefetch operation can be calculated by the following formula:

[0069]

[0070] Among them, Opt n+1 Indicates the number of iterations when the n+1th prefetch occurs; PP.size indicates the number of texts in the pipeline pool; δ indicates the preset increment, δ>0; Balancer n Indicates the balance factor after the nth prefetch is completed; Indicates rounding down.

[0071] In the above calculation formula, PP.size / top-k (less than 1) reflects the real-time prefetch progress. The prefetch progress is multiplied by PP.size to construct a nonlinear fitting function, so that the prefetch timing and the prefetch progress have a quadratic growth relationship, which is consistent with the above frequency adjustment criteria; the preset increment δ ensures that Opt n+1 Greater than Opt n , optional, in this embodiment, δ = 1. In addition, Balancer n As a real-time factor, it plays a role in dynamically adjusting the prefetching speed according to the knowledge retrieval prefetching recall rate. n If it is a positive number, it means that the current knowledge retrieval prefetching has a higher recall precision, and the frequency of prefetching can be increased to further improve the efficiency of the overall system; otherwise, it means that the current knowledge retrieval prefetching has a lower recall precision, and the frequency of prefetching should be reduced to increase the recall precision.

[0072] (2) Determine the amount of pre-fetched text based on the stability of the text in the result set.

[0073] Due to the retrieval characteristics of the knowledge retrieval method, the texts in the result set are sorted in real time from near to far according to the quantified similarity distance. The more stable the texts in the result set, the more likely it is that the result set will contain the correct results. Based on this, in this embodiment, the number of pre-fetched texts from the result set is positively correlated with the stability of the result set, that is, the higher the stability of the result set, the more texts are pre-fetched from the result set.

[0074] In order to quantify stability, this embodiment calculates two indicators by comparing the changes in the text in the result set between two prefetch operations: Stability and Unstability; Stability indicates the number of texts in the result set that have not changed position compared with the previous prefetch; Unstability indicates the number of texts in the result set that have changed position and are prefetched into the pipeline pool compared with the previous prefetch. Based on the above stability quantification indicators, in this embodiment, the number of texts prefetched from the result set for the nth time is MaxNum n for:

[0075]

[0076] Stability indicates the number of texts in the result set that have not changed position compared with the previous prefetch. Balancer n-1 Indicates the balance factor after the n-1th prefetch is completed; D is a preset positive integer.

[0077] In the above calculation formula, Stability-PP.size represents the possible number of potential prefetch operations; Balancer n-1 The current pre-fetch quantity can be rewarded or penalized according to the recall accuracy of the most recent knowledge retrieval pre-fetch; the entire calculation result is divided by a positive integer D as a boundary condition to avoid index out-of-bounds. Optionally, in this embodiment, D=2.

[0078] This embodiment calculates the number of pre-fetched texts by using the above formula before each pre-fetch operation. The calculation result changes with the change of Stability and Balancer, thereby achieving the goal of automatically adjusting the number of pre-fetched texts to balance the pre-fetching speed and recall accuracy.

[0079] (3) Balance the accuracy and prefetching speed of knowledge retrieval based on the accuracy of the text in the pipeline pool.

[0080] Although the size of the pipeline pool and the stability of the result set can dynamically adjust the prefetch speed, the size of the pipeline pool and the stability of the result set are still not sensitive enough to the increase in the recall rate of various types of queries as the number of iterations increases. For queries with faster recall rate growth, the prefetch speed should be increased to build a more efficient pipeline; for queries with slower recall rate growth, the prefetch speed should be reduced to reduce recall loss. This embodiment designs a balance factor Balancer n , to adjust the prefetching speed according to the accuracy of the text in the current pipeline pool. Balancer n The stability quantitative indicators Stability and Unstability of the above result set can be calculated. The specific calculation formula is as follows:

[0081] Balancer n =(Stability-p*Unstability+Balancer n-1 ) / s

[0082] Wherein, p represents a penalty factor, and p∈[1,10], optionally, in this embodiment, p=4; s represents a smoothing factor, and s>1, optionally, in this embodiment, s=2.

[0083] (4) Decide whether to pre-fetch or remove text based on historical location.

[0084] After determining the prefetch time and the number of prefetched texts, you also need to consider whether each text in the result set is worth prefetching. Specifically, after each prefetch, record the historical position of each element in the result set, and retain the prefetch interval for a preset number of times. When prefetching, it will be determined whether the historical position and current position of the prefetched text are equal. If they are equal, the text can be prefetched and placed in the pipeline pool. Otherwise, the text cannot be prefetched, and it will be removed from the text that has been fetched, and the remaining text will be placed in the pipeline pool.

[0085] In addition, during the iteration process, if it is found that the position of any text in the pipeline pool has changed in the result set, it means that the text is unstable and it will be deleted from the pipeline pool to avoid the unstable text causing the large language model to generate incorrect results, thereby ensuring the accuracy of the final knowledge retrieval results.

[0086] The following combination Figure 3 The prefetching example shown further illustrates the execution of the above-mentioned recall-aware prefetching strategy. Figure 3 In, top-k=4, Opt n-2 ~Opt n+1 They respectively represent the corresponding number of knowledge retrieval iterations when the n-2th to n+1th prefetching occurs.

[0087] During the n-1th prefetch, the positions of the four texts in the result set remain unchanged between the n-2th prefetch and the n-1th prefetch. Therefore, during the n-1th prefetch, the Stability of the result set is 4, and the Unstability is 0. The corresponding balancing factor can be calculated as Balancer=2.5, and the prefetch quantity can be further calculated as MaxNum=4.

[0088] During the nth prefetch, the newly inserted text 5 in the result set changes the position of text 4, causing the Unstability of the result set to be equal to 1, while the Stability is reduced to 3. At the same time, after the nth prefetch, since the position of text 4 has changed, the text is unstable and will be deleted from the pipeline.

[0089] In addition to the above recall-aware prefetching strategy, this embodiment further considers that errors will inevitably occur in the pipeline. In order to further ensure the balance between recall rate and efficiency, this embodiment introduces an adaptive error correction strategy and a dynamic pipeline granularity adjustment strategy in the pipeline pool to handle pipeline errors. That is, the text stored in the pipeline pool is provided to the large language model for pre-filling operations after adaptive error correction and dynamic pipeline granularity adjustment, generating the KV cache required for subsequent reasoning of the large language model.

[0090] To achieve adaptive error correction, this embodiment provides the following two error correction solutions:

[0091] (1) Re-prefill: Stop the current prefill task and restart the prefill task based on the correct text;

[0092] (2) Discard after pre-filling: Continue the current pre-filling task and remove the key value calculation results corresponding to the incorrect text from the KV cache after the current pre-filling task is completed.

[0093] This embodiment detects in real time whether the text taken out from the pipeline pool during the pre-filling phase of the large language model contains erroneous text, and when an error is detected, a more appropriate solution is selected from the above two solutions according to the timing of the error detection. Specifically, if the error is detected early and only a few top-level elements are returned, re-prefilling can avoid the subsequent calculation overhead for the erroneous text, and the overall response time is shorter, such as Figure 4 As shown in Case-1 in ; if the error is detected late, many elements have been pre-filled into the KV cache, which will be discarded after pre-filling, thereby avoiding the overhead of repeated calculation of the correct text, and the overall response time is shorter, as shown in Figure 4 As shown in Case-2.

[0094] In accordance with the above-mentioned adaptive error correction strategy, the dynamic pipeline-based retrieval enhancement generation method provided in this embodiment further includes: in the process of calculating the corresponding keys and values ​​of the large language model and filling the calculation results into the KV cache, if it is found that the text taken out from the pipeline pool contains erroneous text, it is determined whether the amount of data filled into the KV cache exceeds the preset threshold. If so, the large language model continues to calculate the keys and values ​​of the currently taken out text, and stores the calculation results in the KV cache. After the calculation of the currently taken out text is completed, the key and value calculation results corresponding to the erroneous text in the KV cache are deleted; otherwise, the data in the KV cache is deleted, and after deleting the erroneous text from the currently taken out text, the large language model recalculates the keys and values ​​of each remaining text and fills them into the KV cache.

[0095] Corresponding to multiple pre-fetches, in this embodiment, the pre-filling process is also performed in multiple times. Therefore, the pre-filling in this embodiment can be regarded as a block pre-filling; block pre-filling will introduce additional costs. As long as the knowledge retrieval pre-fetching text operation is completed, the pre-filling task will be started immediately, which will increase the number of pre-filling sub-tasks and reduce the overall performance. The dynamic pipeline granularity adjustment strategy proposed in this embodiment can better deal with this problem. The dynamic pipeline granularity adjustment strategy proposed in this embodiment specifically includes the following two scheduling filling strategies:

[0096] (1) Fine-grained pipeline: prefilling starts immediately after receiving new prefetch results, which actually overlaps the knowledge retrieval latency but also incurs higher block prefilling costs;

[0097] (2) Coarse-grained pipeline: Wait for a batch of pre-fetched results to start batch pre-filling, reducing the additional overhead of excessive block pre-filling at the cost of waiting time.

[0098] Under a specific hardware and software environment, given a fixed text block size, the delay of each block pre-filling remains reliable and stable. Based on this predictability, the delay of block pre-filling can be predicted in advance, so that there is a clear estimate of the performance of the system under different pipeline granularities; this further makes it possible to optimize the pipeline strategy by dynamically balancing the granularity of the pipeline. In this embodiment, the basic principle followed when dynamically adjusting the pipeline granularity is: calculate the optimal overlapping granularity based on the pre-fetching speed and the delay of block pre-filling.

[0099] Accordingly, in this embodiment, taking text from the pipeline pool and providing it to the large language model includes: predicting the corresponding pre-filling delay when taking different amounts of text from the pipeline pool and taking text from the pipeline pool according to the amount with the minimum pre-filling delay, and providing it to the large language model;

[0100] The pre-fill delay is the time from when the first text to be retrieved enters the pipeline pool to when the large language model completes the calculation of keys and values ​​for all retrieved texts and fills the corresponding calculation results into the KV cache.

[0101] by Figure 5 For example, in Figure 5 In Case-1, the coarse-grained pipeline will have a shorter overall delay than the fine-grained pipeline, so the coarse-grained pipeline will be used; Figure 5 In Case-2, using fine-grained pipeline will have shorter overall latency compared to coarse-grained pipeline, so fine-grained pipeline will be adopted.

[0102] In general, this embodiment adopts a recall-aware prefetching strategy to prefetch the results of knowledge retrieval, and then returns some retrieval results in advance to start the pre-filling stage of the large language model. It can fold the knowledge retrieval and pre-filling processes of the large language model and reduce the overall response time. It is supplemented by an adaptive error correction strategy to ensure the correct execution of the pipeline. At the same time, based on the dynamic pipeline granularity adjustment strategy, it can balance the pipeline efficiency and GPU utilization.

[0103] This embodiment proves that this embodiment can effectively improve the response speed of retrieval enhancement generation through the actual measurement results of TTFT. Specifically, this embodiment first tests the time overhead B1 and B2 that can be saved under various conditions. Then, this embodiment tests the overall performance improvement compared to the traditional retrieval enhancement generation method, which combines these two advantages.

[0104] The existing retrieval enhancement generation system Disk-RAG implemented on the hard disk and the retrieval enhancement generation system DRAM-RAG implemented in DRAM are used for comparison. The experimental results show that compared with Disk-RAG, the present embodiment shortens the response time of retrieval enhancement generation by 1.3 to 2.6 times. Related experimental results also show that despite the use of knowledge retrieval represented by approximate nearest neighbor search, in most cases, the present embodiment is very close to or even better than DRAM-RAG in terms of response speed. This means that the present embodiment can not only introduce knowledge retrieval in the retrieval enhancement generation system to increase the capacity of the knowledge base, but also effectively control the growth of response delay.

[0105] In order to further analyze and verify the improvement in knowledge retrieval performance that this embodiment can bring, this embodiment defines the non-overlapping running time of knowledge retrieval as "independent delay". Experiments show that: this embodiment effectively reduces the knowledge retrieval delay represented by approximate nearest neighbor search under various conditions, and compared with Disk-RAG, this embodiment optimizes the independent delay of knowledge retrieval represented by approximate nearest neighbor search from 68% to 96% in the case of DiskANN, and from 56% to 99% in the case of SPANN. In addition, in most cases, the independent delay of this embodiment is even smaller than that of DRAM-RAG. These results further show that this embodiment has made knowledge retrieval no longer the main bottleneck for retrieval enhancement generation.

[0106] Embodiment 2:

[0107] A dynamic pipeline-based retrieval enhancement generation system includes: a knowledge retrieval module, a pre-fetching module and a pre-filling module;

[0108] The knowledge retrieval module is used to retrieve a pre-established text corpus through a preset retrieval method to obtain the top-k texts most similar to the query content input by the user as retrieval results, and put the retrieval results into a result set; the retrieval method includes multiple iterations, and each iteration retrieves the part of the text most similar to the query content from the text corpus;

[0109] A pre-fetch module is used to pre-fetch part of the text from the result set and put it into a pipeline pool pre-established in memory during the process of retrieving the text corpus through the retrieval method;

[0110] The pre-filling module is used to take text from the pipeline pool and provide it to the large language model so that the large language model can calculate the corresponding keys and values, and fill the calculation results into the KV cache of the large language model for use in the decoding stage of the large language model;

[0111] Among them, the text corpus is stored on the hard disk; top-k is a preset positive integer.

[0112] In this embodiment, the specific implementation of each module can refer to the description in the above embodiment 1 and will not be repeated here.

[0113] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dynamic pipeline-based retrieval enhancement generation method, characterized in that: include: Knowledge retrieval: a pre-established text corpus is searched by a preset search method to obtain the top-k texts most similar to the query content input by the user as search results, and the search results are placed in a result set; the search method includes multiple iterations, and each iteration retrieves the partial text most similar to the query content from the text corpus; Pre-fetching of retrieval results: in the process of searching the text corpus by the retrieval method, pre-fetching part of the text from the result set multiple times and putting it into a pipeline pool pre-established in the memory; Pre-filling: taking text from the pipeline pool multiple times and providing it to the large language model, so that the large language model calculates the corresponding keys and values, and fills the calculation results into the KV cache of the large language model for use in the decoding stage of the large language model; Wherein, the text corpus is stored on a hard disk; top-k is a preset positive integer.

2. The dynamic pipeline-based search enhancement generation method according to claim 1, characterized in that: The frequency of pre-fetching text from the result set decreases as the number of iterations in the knowledge retrieval process increases.

3. The dynamic pipeline-based search enhancement generation method according to claim 2, characterized in that: The nth operation of pre-fetching text from the result set occurs at the Opt. n iterations, then, Among them, Opt n+1 Indicates the number of iterations when the n+1th prefetch occurs; PP.size indicates the number of texts in the pipeline pool; δ indicates the preset increment, δ>0; Balancer n Indicates the balance factor after the nth prefetch is completed; Indicates rounding down.

4. The dynamic pipeline-based search enhancement generation method according to claim 3, characterized in that: The amount of pre-fetched text from the result set is positively correlated with the stability of the result set.

5. The dynamic pipeline-based search enhancement generation method according to claim 4, characterized in that: The number of texts to be pre-fetched from the result set for the nth time MaxNum n for: Stability indicates the number of texts in the result set that have not changed position compared with the previous prefetch. Balancer n-1 Indicates the balance factor after the n-1th prefetch is completed; D is a preset positive integer.

6. The dynamic pipeline-based search enhancement generation method according to claim 5, characterized in that: Balancer n =(Stability-p*Unstability+Balancer n-1 ) / s Wherein, Unstability indicates the number of texts that have changed positions in the result set and are prefetched into the pipeline pool compared with the previous prefetch; p indicates the penalty factor, and p∈[1,10]; s indicates the smoothing factor, and s>1.

7. The dynamic pipeline-based search enhancement generation method according to any one of claims 1 to 6, characterized in that: Each time before taking text from the pipeline pool and providing it to the large language model, it also includes: deleting unstable text in the pipeline pool from the pipeline pool; The unstable text refers to the text whose position in the result set has changed during the retrieval iteration process.

8. The dynamic pipeline-based search enhancement generation method according to any one of claims 1 to 6, characterized in that: Also includes: In the process of the large language model calculating the corresponding keys and values ​​and filling the calculation results into the KV cache, if it is found that the text taken out from the pipeline pool contains erroneous text, it is determined whether the amount of data filled into the KV cache exceeds a preset threshold. If so, the large language model continues to calculate the keys and values ​​of the currently taken out text, and stores the calculation results in the KV cache. After the calculation of the currently taken out text is completed, the key and value calculation results corresponding to the erroneous text in the KV cache are deleted; otherwise, the data in the KV cache is deleted, and after deleting the erroneous text from the currently taken out text, the large language model recalculates the keys and values ​​of the remaining texts and fills them into the KV cache.

9. The dynamic pipeline-based search enhancement generation method according to any one of claims 1 to 6, characterized in that: Taking text from the pipeline pool and providing it to the large language model, including: predicting the corresponding pre-filling delay when taking different amounts of text from the pipeline pool and taking text from the pipeline pool according to the amount with the minimum pre-filling delay, and providing it to the large language model; The pre-filling delay is the time from when the first text to be taken out enters the pipeline pool to when the large language model completes the calculation of keys and values ​​for all the taken out texts and fills the corresponding calculation results into the KV cache.

10. A dynamic pipeline-based search enhancement generation system, characterized in that: include: Knowledge retrieval module, pre-fetch module and pre-fill module; The knowledge retrieval module is used to search a pre-established text corpus through a preset retrieval method to obtain top-k texts that are most similar to the query content input by the user as retrieval results, and put the retrieval results into a result set; The retrieval method includes multiple iterations, each of which retrieves a portion of text that is most similar to the query content from the text corpus; The pre-fetching module is used to pre-fetch part of the text from the result set and put it into a pipeline pool pre-established in the memory during the process of searching the text corpus by the retrieval method; The pre-filling module is used to take text from the pipeline pool and provide it to the large language model, so that the large language model calculates the corresponding key and value, and fills the calculation result into the KV cache of the large language model for use in the decoding stage of the large language model; Wherein, the text corpus is stored on a hard disk; top-k is a preset positive integer.

Images