A dynamic pipeline-based retrieval augmentation generation method and system

Abstract

The application discloses a retrieval enhancement generation method and system based on a dynamic pipeline, belongs to the cross field of solid-state storage and large language models, and comprises the following steps: retrieving a pre-established text corpus by a preset retrieval method to obtain top-k texts most similar to user input query content as retrieval results, and putting the retrieval results into a result set; the retrieval method comprises multiple iterations, and each iteration retrieves part of the texts most similar to the query content; in the process of retrieving the text corpus by the retrieval method, part of the texts are pre-fetched from the result set in multiple times and put into a pipeline pool pre-established in the memory; the texts are taken from the pipeline pool in multiple times and provided to a large language model, so that the large language model calculates corresponding keys and values, and the calculation results are filled into a KV cache of the large language model for use in a decoding stage of the large language model. The application can mask the delay of knowledge retrieval, so that the response time of the retrieval enhancement generation system is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross field of solid-state storage and large language models, and more particularly, relates to a retrieval-augmented generation method and system based on a dynamic pipeline. BACKGROUND

[0002] Large language models have shown excellent capabilities in understanding and generating human language, but their performance is limited by the limitations of training data sets, which manifest as hallucination, outdated knowledge, and difficulty in tracing reasoning processes. To address these issues, retrieval-augmented generation (RAG) has emerged as a promising solution, which enriches user input by retrieving relevant information from external databases to improve the generation quality of large language models. In this process, knowledge retrieval represented by approximate nearest neighbor search (ANNS) as a key technology to balance retrieval accuracy and speed has received extensive attention. However, as the database size expands to hundreds of billions, it becomes impractical to store the knowledge retrieval index represented by approximate nearest neighbor search entirely in DRAM, prompting research into knowledge retrieval algorithms based on other storage devices.

[0003] However, knowledge retrieval needs to face efficiency bottlenecks when integrated into retrieval-augmented generation systems. Taking a specific knowledge retrieval system DiskANN (Disk-based Approximate Nearest Neighbor) as an example, by comparing the performance of the DRAM-based knowledge retrieval system FAISS and the knowledge retrieval system DiskANN on the OpenWebText and Wikipedia data sets, it can be found that the retrieval delay of DiskANN significantly affects the overall response time of the retrieval-augmented generation system, i.e., the entire process time (TTFT) from user input to knowledge retrieval completion, and then to the generation of the first word by the large language model. Although DiskANN optimizes the performance of graph-based knowledge retrieval, the growth of database size, the high demand of large language models for external knowledge, and the expansion of graph index search range all exacerbate retrieval delay.

[0004] Overall, when implementing retrieval-augmented generation systems based on storage devices, how to reduce the overall response time is a problem that needs to be solved. SUMMARY

[0005] In view of the defects and improvement needs of the prior art, the present application provides a retrieval-augmented generation method and system based on a dynamic pipeline, which aims to mask the delay of knowledge retrieval to reduce the response time of the retrieval-augmented generation system.

[0006] To achieve the above object, according to one aspect of the present application, a dynamic pipeline-based search enhancement generation method is provided, comprising:

[0007] Knowledge search: searching a pre-established text corpus through a preset search method to obtain top-k texts most similar to the user input query content as search results, and putting the search results into a result set; the search method includes multiple iterations, and each iteration retrieves part of the text most similar to the query content from the text corpus;

[0008] Search result prefetching: during the process of searching the text corpus through the search method, part of the text is prefetched from the result set in multiple times and put into a pipeline pool pre-established in the memory;

[0009] Pre-filling: taking the text from the pipeline pool in multiple times and providing it to the large language model to calculate the corresponding key and value by the large language model, and filling the calculation results to the KV cache of the large language model for use in the decoding stage of the large language model;

[0010] Wherein, the text corpus is stored on a hard disk; top-k is a preset positive integer.

[0011] Further, the frequency of prefetching the text from the result set decreases with the increase of the number of iterations in the knowledge search process.

[0012] Further, the n-th operation of prefetching the text from the result set occurs in the Opt n th iteration in the knowledge search process, then,

[0013]

[0014] Wherein, Opt n+1 represents the number of iterations when the n+1-th prefetching occurs; PP.size represents the number of texts in the pipeline pool; δ represents a preset increment, δ>0; Balancer n represents the balancing factor after the n-th prefetching ends; represents the floor function.

[0015] Further, the number of texts prefetched from the result set is positively correlated with the stability of the result set.

[0016] Further, the number of texts prefetched from the result set in the n-th time MaxNum n is:

[0017]

[0018] Wherein, Stability represents the number of texts in the result set that have not changed in position compared with the previous prefetching; Balancern-1 represents the balance factor after the end of the n-1th prefetch; D is a preset positive integer.

[0019] Further,

[0020] Balancer n =(Stability-p*Unstability+Balancer n-1 ) / s

[0021] Wherein, Unstability represents that the number of texts in the result set that have changed positions and are prefetched into the pipeline pool compared with the previous prefetch; p represents the penalty factor, and p [1, 10]; s represents the smoothing factor, and s > 1.

[0022] Further, before taking the text from the pipeline pool each time and providing it to the large language model, it further includes: deleting the unstable text in the pipeline pool from the pipeline pool;

[0023] Wherein, the unstable text represents the text whose position in the result set has changed in the process of retrieval iteration.

[0024] Further, the retrieval enhancement generation method based on dynamic pipeline provided by the application further includes: in the process that the large language model calculates the corresponding key and value and fills the calculation result to the KV cache, if it is found that the text taken from the pipeline pool contains error text, it is judged whether the amount of data filled to the KV cache exceeds the preset threshold, if yes, the large language model is made to continue calculating the key and value of the current taken text, and the calculation result is stored to the KV cache, and after the calculation of the current taken text is completed, the key and value calculation result corresponding to the error text in the KV cache is deleted; otherwise, the data in the KV cache is deleted, and after the error text is deleted from the current taken text, the large language model is made to recalculate the key and value of each remaining text and fill it to the KV cache.

[0025] Further, taking the text from the pipeline pool and providing it to the large language model includes: predicting the prefilling delay when taking different number of texts from the pipeline pool, and taking the text from the pipeline pool according to the number of the smallest prefilling delay and providing it to the large language model.

[0026] Wherein, the prefilling delay is the time from the first text taken into the pipeline pool to the completion of the calculation of the key and value of all the taken texts by the large language model and the filling of the corresponding calculation result to the KV cache.

[0027] According to another aspect of the application, a retrieval enhancement generation system based on dynamic pipeline is provided, which includes a knowledge retrieval module, a prefetch module and a prefilling module.

[0028] The knowledge retrieval module is configured to retrieve a pre-established text corpus by a preset retrieval method to obtain top-k texts most similar to the query content input by the user as retrieval results from the text corpus, and put the retrieval results into a result set; the retrieval method comprises multiple iterations, and each iteration retrieves part of the texts most similar to the query content from the text corpus;

[0029] The prefetching module is configured to prefetch part of the texts from the result set and put them into a pipeline pool pre-established in the memory during the retrieval of the text corpus by the retrieval method;

[0030] The pre-filling module is configured to take the texts from the pipeline pool and provide them to the large language model to calculate corresponding keys and values by the large language model, and fill the calculation results to the KV cache of the large language model for use in the decoding stage of the large language model;

[0031] The text corpus is stored on a hard disk; and the top-k is a preset positive integer.

[0032] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0033] (1) The present application introduces a pipeline pool, pre-fetches part of the retrieval results during the iteration process of knowledge retrieval, and then returns part of the retrieval results in advance to start the pre-filling stage of the large language model. Compared with the traditional retrieval-enhanced generation system which starts the pre-filling stage of the large language model after the knowledge retrieval is completed, the present application effectively masks the delay of knowledge retrieval and reduces the overall response time of retrieval-enhanced generation.

[0034] (2) The recall rate of knowledge retrieval increases at a faster rate in the initial iteration stage, and then increases at a slower rate. The present application controls the frequency of prefetching texts from the result set to decrease with the increase of the number of iterations of knowledge retrieval during the process of knowledge retrieval, thereby realizing a recall-aware prefetching. When the recall rate increases faster, the prefetching is performed at a higher frequency; when the recall rate increases slowly, the prefetching is performed at a lower frequency. Thus, while reducing the overall response delay, the additional recall loss is kept low, and the accuracy of the final retrieval-enhanced generation is ensured. In a further preferred embodiment, a non-linear fitting function is constructed for the prefetching mechanism, which takes into account the real-time prefetching progress and the recall rate. Based on the fitting function, the prefetching timing is determined, which can accurately adjust the prefetching speed according to the knowledge retrieval recall rate.

[0035] (3) The more stable the text in the result set is, the greater the possibility of containing correct results in the result set is. Based on this, the number of pre-fetched texts each time is positively correlated with the stability of the result set, that is, the more stable the result set is, the more texts are pre-fetched therefrom, so that the number of pre-fetched texts can be automatically adjusted to balance the pre-fetching speed and the recall rate. In a further preferred solution, a quantitative indicator of the stability of the result set is given based on the change in the position of the text in the result set, and a calculation expression of the number of pre-fetched files each time is further given, which comprehensively considers the stability and the recall rate of the result set, and can accurately calculate the most appropriate number of pre-fetching each time.

[0036] (4) Based on the quantitative indicator of the stability of the result set, the present application gives a calculation expression of a balancing factor for balancing the recall rate and the pre-fetching speed, which serves as the basis for the calculation expression of the pre-fetching time and the number of pre-fetching, so that the determined pre-fetching time and the number of pre-fetching can more accurately balance the pre-fetching speed and the recall rate, and ensure that the retrieval enhancement generation has a high overall performance.

[0037] (5) Before the text is taken from the pipeline pool and provided to the large language model, the present application will first delete unstable texts from the pipeline pool, that is, the texts whose positions in the result set have changed in the process of retrieval iteration, so that the large language model can be prevented from generating incorrect results due to unstable texts, and the accuracy of the final knowledge retrieval result can be ensured.

[0038] (6) In the process of pre-filling by the large language model, the present application will detect in real time whether the text taken from the pipeline pool contains errors, and when errors are contained, corresponding error correction strategies are taken according to the progress of pre-filling, specifically, when errors are found in the early stage of pre-filling, the existing data in the KV cache is deleted and the operation of key value calculation and filling the KV cache is re-executed; when errors are found in the late stage of pre-filling, the incorrect result is deleted from the KV cache after the current pre-filling process is completed, so that errors can be corrected in a short time.

[0039] (7) When the text is taken from the pipeline pool to start the pre-filling stage of the large language model, the present application will predict the pre-filling delay corresponding to different pipeline granularities in advance to determine the pipeline granularity that minimizes the pre-filling delay, so that dynamic balancing of the pipeline granularity is achieved, and the overall response time is further reduced. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A schematic diagram of a retrieval enhancement generation method based on a dynamic pipeline provided by an embodiment of the present application;

[0041] Figure 2The overall response time comparison schematic diagram of different search enhancement generation methods provided by the embodiment of the present application; wherein (a) is the overall response time of the search enhancement generation method provided by the embodiment of the present application, and (b) is the overall response time of the traditional search enhancement generation method;

[0042] Figure 3 The result set stability quantification index calculation schematic diagram provided by the embodiment of the present application;

[0043] Figure 4 The error correction strategy schematic diagram provided by the embodiment of the present application;

[0044] Figure 5 The pipeline granularity dynamic adjustment schematic diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present application clearer and more understandable, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict between them.

[0046] In the present application, the terms "first", "second", etc. (if any) in the present application and the drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0047] Before explaining the technical scheme of the present application in detail, a brief introduction will be made to the process of generating text by a large language model (LLM).

[0048] The process of generating text by a large language model mainly involves two stages of pre-padding and decoding. The pre-padding stage is mainly used to prepare the initial context to provide a basis for the model to generate subsequent text. The decoding stage is the process of the model generating subsequent text step by step according to the context prepared by the pre-padding stage. The main operations performed in the two stages are as follows:

[0049] In the pre-padding stage, the user provides an initial text as a prompt, such as a question, a sentence or a paragraph. This text will be encoded into a series of tokens, which are the basic units used by the model when processing text, depending on the tokenization strategy of the model. The large language model processes all the input tokens to generate keys and values, which are stored in the KV cache. These keys and values are used for calculation in the subsequent decoding stage. The probability distribution of the last token is calculated as the starting point for generating the next token.

[0050] In the decoding phase, the model generates a new token at a time in an autoregressive manner, and each time a new token is generated, the model takes all previous tokens as input; a new token is sampled from the initial probability distribution in the pre-filling phase as the next input; the new token is used for forward propagation to update the KV cache and generate new keys and values; a new probability distribution is generated for sampling the next token. Repeat the foregoing steps until the maximum length of generation is reached or the stop condition is met.

[0051] When knowledge retrieval is integrated into the retrieval enhancement generation system, a knowledge retrieval system based on a text corpus is established in a storage device, a knowledge retrieval method represented by approximate nearest neighbor search is used to perform knowledge retrieval on the query content input by the user, and top-k texts most similar to the user input content are found out and placed in the result set. In the traditional retrieval enhancement system, the pre-filling phase of the large language model is started after the knowledge retrieval process is completed, which makes the retrieval delay significantly affect the overall response time of the retrieval enhancement generation system.

[0052] In order to break through the performance bottleneck of knowledge retrieval when integrated into the retrieval enhancement generation system, the present application provides a retrieval enhancement generation method and system based on a dynamic pipeline, the overall idea of which is to: build a pipeline retrieval enhancement generation execution paradigm, overlap the execution time of knowledge retrieval and large language model to improve efficiency, and thus jointly optimize knowledge retrieval and large language model from the system level, and reduce the negative impact of knowledge retrieval represented by approximate nearest neighbor search.

[0053] Specifically, in the knowledge retrieval process, the present application pre-fetches part of the results and feeds them to the large language model for pre-filling in advance, thereby shortening the overall response time without significantly reducing accuracy. On this basis, the present application further optimizes the pre-fetching timing and the number of pre-fetching in combination with the execution characteristics of knowledge retrieval and the large language model, and formulates corresponding error correction strategies and pipeline granularity dynamic adjustment strategies.

[0054] The following is an example.

[0055] Example 1:

[0056] A retrieval enhancement generation method based on a dynamic pipeline, as shown in Figure 1 , comprising:

[0057] Knowledge retrieval: retrieve the pre-established text corpus by a predetermined retrieval method to obtain top-k texts most similar to the query content input by the user as retrieval results, and place the retrieval results in the result set; the retrieval method includes multiple iterations, and each iteration retrieves part of the text most similar to the query content from the text corpus;

[0058] Result pre-fetching: during the process of retrieving the text corpus by the retrieval method, part of the text is pre-fetched from the result set and put into the pipeline pool pre-established in the memory in multiple times;

[0059] Pre-filling: the text is taken from the pipeline pool in multiple times and provided to the large language model, so that the corresponding key and value are calculated by the large language model, and the calculation result is filled to the KV cache of the large language model for use in the decoding stage of the large language model;

[0060] wherein the text corpus is stored on a hard disk; top-k is a preset positive integer.

[0061] In the process of knowledge retrieval, the retrieval result is pre-fetched from the result set, put into the pipeline pool established in the memory, and the text is taken from the pipeline pool for use in the pre-filling stage of the large language model, and the calculation of the KV cache in the pre-filling stage of the large language model is combined with the user's own inquiry to realize retrieval enhancement generation; thus, a pipeline system is realized, in which the knowledge retrieval system is the producer, the pre-filling module is the consumer, and the pipeline pool is the pipeline buffer pool, and the time overlap of part of the knowledge retrieval pre-fetching stage and part of the pre-filling stage on the CPU is realized. The overall response time of the traditional retrieval enhancement generation and the embodiment is shown in (a) and (b) of Figure 2 By introducing the pipeline, part of the time of the knowledge retrieval and the pre-filling stage of the large language model is overlapped, which is equivalent to masking part of the delay of the knowledge retrieval, so that the overall response time can be effectively reduced, and part of the overlap of the knowledge retrieval task and the pre-filling saves the time cost of B1.

[0062] Part of the overlap of the knowledge retrieval task and the pre-filling in the pipeline saves the time cost of B1.

[0063] It should be noted that in the traditional retrieval enhancement generation system, the knowledge retrieval algorithm will be executed according to the default parameters until the search queue converges, while the embodiment is interrupted when top-m and top-n are obtained (top-m+top-n=top-k), which is equivalent to ending the knowledge retrieval task in advance. Compared with the traditional retrieval enhancement generation, the overall response time is further shortened. The early termination of the knowledge retrieval task saves the time cost of B2 under the premise of ensuring the quality of data generation.

[0064] The saving of the time cost of B1 and B2 makes the embodiment have a performance advantage compared with the traditional retrieval enhancement generation system.

[0065] In addition, the embodiment pre-fetches the knowledge retrieval result in multiple times, which saves the running time of the one-time pre-fetching of all results due to the characteristics of the knowledge retrieval itself.

[0066] To balance the pre-fetching speed and the recall accuracy as much as possible while reducing the overall response time, the embodiment proposes a recall-aware knowledge retrieval pre-fetching strategy to determine the timing and the number of pre-fetched texts according to the quality of the user's query request text and the generated texts so as to obtain suitable texts and send them to the pipeline pool cache. The recall-aware knowledge retrieval pre-fetching strategy specifically includes the following aspects:

[0067] (1) The timing of performing knowledge retrieval pre-fetching is determined according to the number of texts stored in the pipeline pool in real time.

[0068] Since the recall rate of knowledge retrieval represented by approximate nearest neighbor search increases at a faster speed in the initial iteration stage and then increases at a slower speed, correspondingly, the criterion followed by the dynamic adjustment of the pre-fetching frequency is to increase the pre-fetching frequency in the earlier stage of the iteration of knowledge retrieval represented by approximate nearest neighbor search and to decrease the pre-fetching frequency in the later stage. Based on the criterion, the n th pre-fetching operation from the result set occurs in the Opt n th iteration in the knowledge retrieval process, and the timing of the n th pre-fetching operation can be calculated by the following formula:

[0069]

[0070] wherein Opt n+1 represents the iteration number when the n+1 th pre-fetching occurs; PP.size represents the number of texts in the pipeline pool; δ represents a preset increment, δ>0; Balancer n represents the balancing factor after the n th pre-fetching ends; represents the floor function.

[0071] In the above calculation formula, PP.size / top-k (less than 1) reflects the real-time pre-fetching progress, and multiplying the pre-fetching progress by PP.size constructs a nonlinear fitting function, so that the pre-fetching timing and the pre-fetching progress have a quadratic function growth relationship, which is consistent with the above frequency adjustment criterion; the preset increment δ ensures that Opt n+1 is greater than Opt n . Optionally, δ=1 in the embodiment. In addition, Balancer n , as a real-time factor, plays a role in dynamically adjusting the pre-fetching speed according to the knowledge retrieval pre-fetching recall rate. When Balancer n is positive, it indicates that the current knowledge retrieval pre-fetching has a high recall accuracy, and the pre-fetching frequency can be increased to further improve the overall system efficiency; otherwise, it indicates that the current knowledge retrieval pre-fetching has a low recall accuracy, and the pre-fetching frequency should be reduced to increase the recall accuracy.

[0072] (2) Determine the number of pre-fetching texts according to the stability of the texts in the result set.

[0073] Due to the retrieval characteristics of the knowledge retrieval method, the texts in the result and are sorted in real time according to the quantitative similarity distance from near to far, and the more stable the texts in the result set are, the more likely the result set contains correct results. Based on this, in the embodiment, the number of pre-fetching texts from the result set is positively correlated with the stability of the result set, that is, the higher the stability of the result set is, the more the number of pre-fetching texts from the result set is.

[0074] In order to quantify the stability, the embodiment calculates two indexes: Stability and Unstability by comparing the change of the texts in the result set between two pre-fetching operations; Stability represents the number of texts in the result set that do not change positions compared with the previous pre-fetching; Unstability represents the number of texts in the result set that change positions and are pre-fetched to the pipeline pool compared with the previous pre-fetching. Based on the above stability quantification indexes, in the embodiment, the number of pre-fetching texts from the result set in the nth time is MaxNum n :

[0075]

[0076] Stability represents the number of texts in the result set that do not change positions compared with the previous pre-fetching; Balancer n-1 represents the balancing factor after the end of the (n-1)th pre-fetching; D is a preset positive integer.

[0077] In the above calculation formula, Stability-PP.size represents the possible number of potential pre-fetching operations; Balancer n-1 The current pre-fetching number can be rewarded or punished according to the recall accuracy of the recent knowledge retrieval pre-fetching; the whole calculation result is divided by the positive integer D, which is a boundary condition, which can avoid index out of bounds, and optionally, in the embodiment, D=2.

[0078] The embodiment calculates the pre-fetching text number by the above formula before each pre-fetching operation, and the calculation result changes with the changes of Stability and Balancer, so as to realize the goal of automatically adjusting the pre-fetching text number to balance the pre-fetching speed and the recall accuracy.

[0079] (3) Balance the accuracy of the knowledge retrieval pre-fetching and the pre-fetching speed according to the accuracy of the texts in the pipeline pool.

[0080] Although the size of the pipeline pool and the stability of the result set can dynamically adjust the speed of prefetching, the size of the pipeline pool and the stability of the result set are still not sensitive enough to the recall rate of various types of queries with the increase of the number of iterations. For queries with a faster growth of the recall rate, the prefetching speed should be increased to build a more efficient pipeline; for queries with a slower growth of the recall rate, the prefetching speed should be reduced to reduce the recall loss. The embodiment designs a balancing factor Balancer n to adjust the prefetching speed according to the accuracy of the text in the current pipeline pool. Balancer n can be calculated by the above-mentioned stability quantization indicators Stability and Unstability of the result set, and the specific calculation formula is as follows:

[0081] Balancer n =(Stability-p*Unstability+Balancer n-1 ) / s

[0082] Wherein, p represents a penalty factor, and p∈[1,10], optionally, in the embodiment, p=4; s represents a smoothing factor, and s>1, optionally, in the embodiment, s=2.

[0083] (4) Decide whether to prefetch text or remove text according to historical position.

[0084] After determining the prefetching time and the number of prefetching texts, it is also necessary to consider whether each text in the result set is worth prefetching. Specifically, after each prefetching is completed, the historical position of each element in the result set is recorded, and the retention time is a preset number of prefetching intervals. When prefetching, it is judged whether the historical position and the current position of the prefetched text are equal. If they are equal, the text can be prefetched and put into the pipeline pool. Otherwise, the text cannot be prefetched, and it will be removed from the prefetched texts and put into the pipeline pool.

[0085] In addition, during the iteration process, if it is found that the position of any text in the pipeline pool in the result set has changed, it means that the text is an unstable text, which will be deleted from the pipeline pool to avoid unstable texts leading to incorrect results generated by the large language model, thereby ensuring the accuracy of the final knowledge retrieval result.

[0086] The execution of the above-mentioned recall-aware prefetching strategy is further explained and described in the following prefetching example shown in Figure 3 . Figure 3 In the example, top-k=4, Opt n-2 ~ Opt n+1 respectively represent the corresponding number of knowledge retrieval iterations when the n-2~n+1th prefetching occurs.

[0087] At the n-1th prefetch, the positions of the 4 texts in the result set remain unchanged between the n-2th prefetch and the n-1th prefetch, so the Stability of the result set at the n-1th prefetch is 4, and the Unstability is 0, and the corresponding balance factor is Balancer = 2.5, and the number of prefetches can be further calculated as MaxNum = 4.

[0088] At the n-1th prefetch, the positions of the 4 texts in the result set remain unchanged between the n-2th prefetch and the n-1th prefetch, so the Stability of the result set at the n-1th prefetch is 4, and the Unstability is 0, and the corresponding balance factor is Balancer = 2.5, and the number of prefetches can be further calculated as MaxNum = 4.

[0089] In addition to the above recall-aware prefetching strategy, the embodiment further considers that the pipeline will inevitably produce errors, and in order to further balance the recall rate and efficiency, the embodiment introduces an adaptive error correction strategy and a dynamic pipeline granularity adjustment strategy in the pipeline pool to handle pipeline error conditions. That is, the texts stored in the pipeline pool are provided to the large language model for pre-population operation after adaptive error correction and dynamic pipeline granularity adjustment, and the KV cache required for subsequent inference of the large language model is generated.

[0090] In order to realize adaptive error correction, the embodiment provides the following two error correction solutions:

[0091] (1) Re-populate: stop the current pre-population task and restart the pre-population task based on the correct text;

[0092] (2) Discard after pre-population: continue the current pre-population task, and remove the key-value calculation results corresponding to the incorrect text from the KV cache after the current pre-population task ends.

[0093] The embodiment will detect in real time whether the text taken out of the pipeline pool by the pre-population stage of the large language model contains an error text, and when an error is detected, a more appropriate solution will be selected from the above two solutions according to the timing of the detected error. Specifically, if the error is detected early, only a few top elements are returned, then re-populate, which can avoid the overhead of subsequent calculations for error texts, and the overall response time is less, as shown in Case-1 of Figure 4 If the error is detected late, many elements have been pre-populated into the KV cache, then discard after pre-population, which can avoid the overhead of repeated calculations for correct texts, and the overall response time is less, as shown in Case-2 of Figure 4 .

[0094] Adapted to the adaptive error correction strategy described above, the retrieval enhancement generation method based on the dynamic pipeline provided by the embodiment further comprises: in the process of calculating the corresponding key and value by the large language model and filling the calculation result to the KV cache, if it is found that the text taken from the pipeline pool contains error text, it is judged whether the amount of data filled to the KV cache exceeds the preset threshold, if yes, the large language model continues to calculate the key and value of the current taken text, and the calculation result is stored to the KV cache, and after the calculation of the current taken text is completed, the key and value calculation result corresponding to the error text in the KV cache is deleted; otherwise, the data in the KV cache is deleted, and after the error text is deleted from the current taken text, the large language model recalculates the key and value of each remaining text and fills it to the KV cache.

[0095] Corresponding to multiple prefetching, in the embodiment, the pre-filling process is also performed in multiple times, therefore, the pre-filling in the embodiment can be regarded as a kind of block pre-filling; the block pre-filling will introduce additional cost, and starting the pre-filling task as soon as the knowledge retrieval pre-fetching text operation ends will cause the increase of the number of pre-filling sub-tasks, and the overall performance may also be reduced. The dynamic pipeline granularity adjustment strategy proposed in the embodiment can better cope with this problem. The dynamic pipeline granularity adjustment strategy proposed in the embodiment specifically includes the following two scheduling filling strategies:

[0096] (1) Fine-grained pipeline: start pre-filling immediately after receiving new pre-fetching results, which actually overlaps the knowledge retrieval delay, but at the same time generates higher block pre-filling cost;

[0097] (2) Coarse-grained pipeline: wait for a batch of pre-fetching results to start batch pre-filling, which reduces the additional overhead of excessive block pre-filling at the cost of waiting time.

[0098] In a specific hardware and software environment, given a fixed text block size, the delay of each block pre-filling remains reliable and stable. Based on this predictability, the delay of block pre-filling can be predicted in advance, so that the performance of the system under different pipeline granularities can be clearly estimated. This further makes it possible to optimize the pipeline strategy by dynamically balancing the pipeline granularity. In the embodiment, the basic principle followed when dynamically adjusting the pipeline granularity is to calculate the optimal overlap granularity according to the pre-fetching speed and the delay of block pre-filling.

[0099] Correspondingly, in the embodiment, taking text from the pipeline pool and providing it to the large language model comprises: predicting the corresponding pre-filling delay when taking different amounts of text from the pipeline pool, and taking the text from the pipeline pool in the amount corresponding to the minimum pre-filling delay and providing it to the large language model.

[0100] The pre-filling delay is the time elapsed from the moment the first text to be retrieved enters the pipeline pool until the large language model completes the key-value calculations for all retrieved texts and fills the corresponding calculation results into the KV cache.

[0101] by Figure 5 For example, in Figure 5 In Case-1, a coarse-grained pipeline will have a shorter overall latency compared to a fine-grained pipeline; therefore, a coarse-grained pipeline will be used. Figure 5 In Case-2, a fine-grained pipeline will have a shorter overall latency compared to a coarse-grained pipeline, therefore, a fine-grained pipeline will be adopted.

[0102] Overall, this embodiment employs a recall-aware prefetching strategy to prefetch the results of knowledge retrieval, and then returns a portion of the retrieval results in advance to initiate the pre-filling stage of the large language model. This approach can fold the knowledge retrieval and large language model pre-filling processes, reducing the overall response time. Furthermore, an adaptive error correction strategy is used to ensure the correct execution of the pipeline. At the same time, a dynamic pipeline granularity adjustment strategy can balance pipeline efficiency and GPU utilization.

[0103] This embodiment demonstrates, through actual TTFT measurement results, that it can effectively improve the response speed of search enhancement generation. Specifically, this embodiment first tested the time savings B1 and B2 under various conditions. Then, this embodiment tested the overall performance improvement compared to traditional search enhancement generation methods, which combines the advantages of both methods.

[0104] This paper compares the existing retrieval enhancement generation system Disk-RAG, implemented on hard disk, and the retrieval enhancement generation system DRAM-RAG, implemented in DRAM. Experimental results show that, compared to Disk-RAG, this embodiment reduces the response time of retrieval enhancement generation by 1.3 to 2.6 times. Related experimental results also show that, despite using knowledge retrieval represented by approximate nearest neighbor search, this embodiment achieves response speeds very close to or even better than DRAM-RAG in most cases. This means that this embodiment can not only introduce knowledge retrieval into the retrieval enhancement generation system to increase the capacity of the knowledge base, but also effectively control the increase in response latency.

[0105] To further analyze and verify the knowledge retrieval performance improvement brought by the embodiment, the non-overlapping running time of knowledge retrieval is defined as "independent delay" in the embodiment. Experiments show that the embodiment effectively reduces the knowledge retrieval delay represented by approximate nearest neighbor search under various conditions, and the independent delay of the knowledge retrieval represented by approximate nearest neighbor search is optimized from 68% to 96% in the DiskANN case and from 56% to 99% in the SPANN case compared with Disk-RAG. In addition, in most cases, the independent delay of the embodiment is even less than that of DRAM-RAG. These results further show that the embodiment has made knowledge retrieval no longer the main bottleneck of retrieval enhancement generation.

[0106] Embodiment 2:

[0107] A retrieval enhancement generation system based on a dynamic pipeline, comprising a knowledge retrieval module, a prefetching module and a pre-filling module;

[0108] The knowledge retrieval module is configured to retrieve a pre-established text corpus by a preset retrieval method to obtain top-k texts most similar to the query content input by a user from the text corpus as retrieval results, and put the retrieval results into a result set. The retrieval method comprises multiple iterations, and each iteration retrieves part of the text most similar to the query content from the text corpus.

[0109] The prefetching module is configured to prefetch part of the text from the result set and put it into a pipeline pool pre-established in the memory during the retrieval of the text corpus by the retrieval method.

[0110] The pre-filling module is configured to take the text from the pipeline pool and provide it to the large language model to calculate the corresponding key and value by the large language model, and fill the calculation results to the KV cache of the large language model for use in the decoding stage of the large language model.

[0111] The text corpus is stored on a hard disk, and the top-k is a preset positive integer.

[0112] In the embodiment, the specific implementation of each module can refer to the description in Embodiment 1 described above, which will not be repeated here.

[0113] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for retrieval augmentation based on dynamic pipeline, characterized in that, Including: Knowledge retrieval: This function retrieves information from a pre-built text corpus using preset retrieval methods. top-k The text that is most similar to the query entered by the user is selected as the search result, and the search result is placed into the result set; The retrieval method includes multiple iterations, and each iteration retrieves part of the text most similar to the query content from the text corpus; Retrieval result prefetching: during the retrieval of the text corpus by the retrieval method, part of the text is prefetched from the result set and placed in a pipeline pool established in memory in advance; Pre-filling: taking text from the pipeline pool and providing it to the large language model in multiple times to calculate the corresponding key and value by the large language model, and filling the calculation results to the KV cache of the large language model for use in the decoding stage of the large language model; The text corpus is stored on a hard disk. top-k is a preset positive integer. The frequency of prefetching text from the result set decreases as the number of iterations in the knowledge retrieval process increases; No. n The operation of prefetching text from the result set occurs during the knowledge retrieval process. In the next iteration, then... wherein, represents the number of iterations when the n +1 prefetch occurs; PP.size represents the number of texts in the pipeline pool; δ represents a preset increment, δ > 0; Balancer n represents the number of iterations when the n balance factor after the represents rounding down; The number of texts prefetched from the result set is positively correlated with the stability of the result set; First n Prefetching a number of texts from the result set MaxNum n is: wherein, Stability represents the number of texts in the result set that have not changed position compared to the previous prefetch; Balancer n-1 represents the balance factor after the end of the n -1st prefetch; D is a preset positive integer.

2. The retrieval enhancement generation method based on a dynamic pipeline according to claim 1, wherein, wherein, Unstability represents the number of texts in the result set that have changed position compared to the previous prefetch and that were prefetched into the pipeline pool; p represents a penalty factor, and ; s represents a smoothing factor, and s > 1.

3. The dynamic pipeline based retrieval enhancement generation method of claim 1 or 2, wherein, Before taking text from the pipeline pool and providing it to the large language model each time, further comprising: deleting unstable text in the pipeline pool from the pipeline pool; Wherein, the unstable text represents the text whose position in the result set has changed in the process of retrieval iteration.

4. The dynamic pipeline based retrieval enhancement generation method of claim 1 or 2, wherein, Further comprising: During the process of calculating the corresponding key and value by the large language model and filling the calculation results to the KV cache, if it is found that the text taken from the pipeline pool contains error text, it is judged whether the amount of data filled to the KV cache exceeds a preset threshold, if yes, the large language model is caused to continue calculating the key and value of the currently taken text, and the calculation result is stored to the KV cache, and after the calculation of the current taken text is completed, the key and value calculation result corresponding to the error text in the KV cache is deleted; otherwise, the data in the KV cache is deleted, and after the error text is deleted from the current taken text, the large language model is caused to re-calculate the key and value of each remaining text and fill it to the KV cache.

5. The dynamic pipeline based retrieval enhancement generation method of claim 1 or 2, wherein, Taking text from the pipeline pool and providing it to the large language model includes: predicting the pre-filling delay when taking different amounts of text from the pipeline pool, and taking the text from the pipeline pool according to the minimum amount of pre-filling delay and providing it to the large language model; Wherein, the pre-filling delay is the time elapsed from the entry of the first text to be taken into the pipeline pool to the completion of the calculation of the key and value by the large language model for all taken texts and the filling of the corresponding calculation results to the KV cache.

6. A dynamic pipeline based retrieval augmentation generation system, comprising: Including: A knowledge retrieval module, a prefetching module, and a pre-filling module; The knowledge retrieval module is used to retrieve a pre-established text corpus using a preset retrieval method, in order to obtain... top-k The text that is most similar to the query entered by the user is selected as the search result, and the search result is placed into the result set; The retrieval method includes multiple iterations, and each iteration retrieves part of the text most similar to the query content from the text corpus; The prefetching module is configured to prefetch part of the text from the result set and place it in a pipeline pool established in memory in advance during the retrieval of the text corpus by the retrieval method; The pre-filling module is configured to take text from the pipeline pool and provide the text to a large language model to calculate corresponding keys and values by the large language model, and fill the calculation results to a KV cache of the large language model for use in a decoding stage of the large language model. The text corpus is stored on a hard disk. top-k is a preset positive integer. The frequency of pre-fetching text from the result set decreases as the number of iterations in the knowledge retrieval process increases; No. n The operation of prefetching text from the result set occurs during the knowledge retrieval process. In the next iteration, then... wherein, represents the number of iterations at which the n +1 prefetch occurs; PP.size represents the number of texts in the pipeline pool; δ represents a preset increment, δ > 0; Balancer n represents the number of iterations at which the n balance factor after the represents rounding down; The number of pre-fetched texts from the result set is positively correlated with the stability of the result set; The first n Prefetching the number of texts from the result set MaxNum n is: wherein, Stability represents the number of texts in the result set that have not changed position compared to the previous prefetch; Balancer n-1 represents the balance factor after the n -1st prefetch; D is a preset positive integer.

Images