Retrieval enhancement generation method and device, equipment and storage medium

Abstract

One or more embodiments of the invention provide a retrieval enhancement generation method and device, equipment and a storage medium. According to the method, a plurality of parallel channels are utilized to retrieve a query request from a plurality of dimensions; the recall content of each channel is pre-filled according to the time sequence of the recall content returned by each channel, the pre-filling result of each channel is stored in a cache device, and the pre-filling result of the channel is obtained according to the pre-filling result cached by the previous channel returning the recall content; and finally, obtaining an output result of the large language model according to a pre-filling result of the channel which finally returns the recall content. The pre-filling is performed according to the time sequence of the recall content returned by the channels, so that the waiting time in the process of one-time retrieval enhancement generation request is reduced, and the pre-filling result of the current channel is calculated according to the pre-filling result cached by the previous channel returning the recall content, so that repeated calculation in the pre-filling stage is avoided.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a search enhancement generation method, apparatus, device, and storage medium. Background Art

[0002] In some fields of retrieval-augmented generation (RAG) applications, such as the financial field, they often rely on more professional content, which usually has a large amount of data. At the same time, in order to achieve better generation effects, the parameters of the generation model are often relatively large. In this case, the effect that can be achieved by optimizing the underlying model is limited. In the prior art, dynamic speculation pipelines are usually used to improve the overall throughput of RAG and reduce the generation time. This method retrieves cluster by cluster and starts generation immediately, uses the current topK (optimal top K) results for preliminary generation, and then updates topK with the new cluster retrieval results.

[0003] However, in this method, since the cluster division and retrieval query are unrelated, the hit rate of speculative generation cannot be guaranteed, which can easily lead to multiple invalid generation, thus preempting normal generation requests; and in the top K update process, it strongly depends on the order of retrieval results. In some cases, the prefix cache cannot be hit in the RAG cache, and the generation time cannot be effectively reduced. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a search enhancement generation method, apparatus, device, and storage medium to effectively reduce generation time.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a retrieval enhancement generation method is proposed, including:

[0007] Use multiple parallel channels to retrieve query requests from multiple dimensions;

[0008] Pre-filling the recalled content of each channel according to the time sequence of the recalled content returned by each channel, wherein the pre-filling result of each channel is stored in a cache device, and the pre-filling result of the channel is obtained according to the pre-filling result cached by the previous channel that returned the recalled content;

[0009] The output result of the large language model is obtained based on the pre-filled result of the channel that finally returns the recalled content.

[0010] In some embodiments, the method further comprises:

[0011] Determining the scenario to which the query request belongs;

[0012] The dimensions of each channel for retrieval are determined according to the scenario.

[0013] In some embodiments, pre-filling the recalled content of each channel according to the time sequence in which the recalled content is returned by each channel includes:

[0014] When the time difference between the recalled contents returned by the multiple channels is within a set range, the recalled contents of the multiple channels are merged and pre-filled according to the merged contents.

[0015] In some embodiments, pre-filling the recalled content of each channel according to the time sequence in which the recalled content is returned by each channel includes:

[0016] When the length returned by a channel is less than a first set threshold, the recalled contents of the channels returned after the channel are merged until the merged length symbol is greater than the first set threshold, and the merged contents are pre-filled.

[0017] In some embodiments, the method further comprises:

[0018] storing in a cache device using the recalled content of the channel as a key and the corresponding pre-filled result as a value;

[0019] The recalled content of the channel is merged with the recalled content of the previous channel, and the merged content is used for pre-filling, wherein when a key is retrieved in the merged content, a value corresponding to the key is obtained from the cache device as a part of the pre-filled result.

[0020] In some embodiments, the method further comprises:

[0021] A corresponding pre-filled queue is established for each query request;

[0022] The recalled contents of each channel obtained based on the query request are input into the pre-filled queue;

[0023] Wherein, after the recalled content of one channel is pre-filled, the pre-filled queue pops out the recalled content of the next channel.

[0024] In some embodiments, the method further comprises:

[0025] In case of receiving multiple requests at the same time, the following information is obtained for each request;

[0026] The length L of the recalled content to be pre-filled in the current queue;

[0027] The time consumed for pre-filling per unit length is P;

[0028] The time taken to complete all recalls of the request in Tx;

[0029] The recall time of the recalled content in the current queue is T;

[0030] For each request, the maximum saved time Ts is determined according to the formula Min(L*P,Tx-T);

[0031] The priority of the request is determined according to Ts corresponding to each request.

[0032] In some embodiments, the method further includes requesting that the recalled content not be pre-populated if the following conditions are met:

[0033] The sum of the lengths of the recalled contents of all channels is less than a second set threshold; or,

[0034] The total time to complete the recall of all channels is greater than the third set threshold.

[0035] In some embodiments, the method further comprises:

[0036] The recalled content popped out from the pre-filling queue is transmitted to the module for pre-filling processing by using a set number of pre-filling schedulers, wherein the set number is determined according to the concurrency supported by the large model.

[0037] According to a second aspect of one or more embodiments of this specification, a retrieval enhancement generation device is provided, including:

[0038] A retrieval unit, used to use multiple parallel channels to retrieve the query request from multiple dimensions respectively;

[0039] A pre-filling unit, used to pre-fill the recalled content of each channel according to the time sequence of each channel returning the recalled content, wherein the pre-filling result of each channel is stored in a cache device, and the pre-filling result of the channel is obtained according to the pre-filling result cached by the previous channel returning the recalled content;

[0040] The output unit is used to obtain the output result of the large language model according to the pre-filled result of the channel that finally returns the recalled content.

[0041] In some embodiments, the apparatus further comprises a determining unit configured to:

[0042] Determining the scenario to which the query request belongs;

[0043] The dimensions of each channel for retrieval are determined according to the scenario.

[0044] In some embodiments, the pre-filling unit is specifically used for:

[0045] When the time difference between the recalled contents returned by the multiple channels is within a set range, the recalled contents of the multiple channels are merged and pre-filled according to the merged contents.

[0046] In some embodiments, the pre-filling unit is specifically used for:

[0047] When the length returned by a channel is less than a first set threshold, the recalled contents of the channels returned after the channel are merged until the merged length symbol is greater than the first set threshold, and the merged contents are pre-filled.

[0048] In some embodiments, the device further comprises a cache unit, configured to:

[0049] storing in a cache device using the recalled content of the channel as a key and the corresponding pre-filled result as a value;

[0050] The recalled content of the channel is merged with the recalled content of the previous channel, and the merged content is used for pre-filling, wherein when a key is retrieved in the merged content, a value corresponding to the key is obtained from the cache device as a part of the pre-filled result.

[0051] In some embodiments, the apparatus further comprises a queue unit, configured to:

[0052] A corresponding pre-filled queue is established for each query request;

[0053] The recalled contents of each channel obtained based on the query request are input into the pre-filled queue;

[0054] Wherein, after the recalled content of one channel is pre-filled, the pre-filled queue pops out the recalled content of the next channel.

[0055] In some embodiments, the apparatus further comprises a priority unit configured to:

[0056] In case of receiving multiple requests at the same time, the following information is obtained for each request;

[0057] The length L of the recalled content to be pre-filled in the current queue;

[0058] The time consumed for pre-filling per unit length is P;

[0059] The time taken to complete all recalls of the request in Tx;

[0060] The recall time of the recalled content in the current queue is T;

[0061] For each request, the maximum saved time Ts is determined according to the formula Min(L*P,Tx-T);

[0062] The priority of the request is determined according to Ts corresponding to each request.

[0063] In some embodiments, the device further comprises a screening unit, configured to set the content of the request for recall that meets the following conditions to not be pre-filled:

[0064] The sum of the lengths of the recalled contents of all channels is less than a second set threshold; or,

[0065] The total time to complete the recall of all channels is greater than the third set threshold.

[0066] In some embodiments, the apparatus further comprises a scheduling unit, configured to:

[0067] The recalled content popped out from the pre-filling queue is transmitted to the module for pre-filling processing by using a set number of pre-filling schedulers, wherein the set number is determined according to the concurrency supported by the large model.

[0068] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0069] processor;

[0070] a memory for storing processor-executable instructions;

[0071] The processor implements the steps of the method proposed in the above embodiment by running the executable instructions.

[0072] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method provided in the above embodiment are implemented.

[0073] According to a fifth aspect of one or more embodiments of the present specification, a computer program product is proposed, comprising a computer program / instruction, which implements the steps of the method proposed in the above embodiment when executed by a processor.

[0074] The retrieval enhancement generation method proposed in the embodiment of this specification uses multiple parallel channels to search for query requests from multiple dimensions respectively; then, the recall content of each channel is pre-filled according to the time sequence of the recalled content returned by each channel, wherein the pre-filled result of each channel is stored in a cache device, and the pre-filled result of the channel is obtained based on the pre-filled result cached by the previous channel that returns the recalled content; finally, the output result of the large language model is obtained based on the pre-filled result of the channel that returns the recalled content last. Since the pre-filling is performed according to the time sequence of the recalled content returned by the channel, the waiting time in the process of a retrieval enhancement generation request is reduced, and the pre-filled result of the current channel is calculated based on the pre-filled result cached by the previous channel that returns the recalled content, avoiding repeated calculations in the pre-filling stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is an application scenario diagram of a retrieval enhancement generation method provided by an exemplary embodiment.

[0076] Figure 2 It is a flowchart of a retrieval enhancement generation method provided by an exemplary embodiment.

[0077] Figure 3 is a schematic diagram of a pre-filling method provided by an exemplary embodiment.

[0078] Figure 4 is a schematic diagram of another pre-filling method provided by an exemplary embodiment.

[0079] Figure 5 It is a block diagram of a retrieval enhancement generation device provided by an exemplary embodiment.

[0080] Figure 6 It is a structural schematic diagram of a device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0081] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0082] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0083] To help understand this specification, technical terms related to this specification are first introduced.

[0084] Retrieval-Augmented Generation (RAG): is a model that combines information retrieval and generation techniques. This technique generates answers or content by referencing information from an external knowledge base and can be used in a variety of natural language processing tasks.

[0085] Prefilling: A technique used in the training and reasoning process of a large model. Before the large model runs, a portion of known or pre-calculated content is input into the model to reduce the real-time calculation amount of the model, reduce repeated calculations and speed up the reasoning process.

[0086] Decoding: In natural language processing tasks, the model generates output sequences one by one based on the information it has already processed (such as prompts or context entered by the user). Decoding is usually performed after the prefilling stage.

[0087] In some RAG applications, such as the financial field, they often rely on more professional content, which usually has a large amount of data. At the same time, in order to achieve better generation effects, the parameters of the generation model are often relatively large. In this case, the effect of model bottom-level optimization is limited.

[0088] In the prior art, a dynamic speculative pipeline is usually used to improve the overall throughput of RAG and reduce the generation time. The method is usually implemented according to the following steps:

[0089] 1. Divide the vector search into several clusters and traverse each cluster for search;

[0090] 2. After obtaining the top K returned by the first cluster, send them to the model generation in advance; top K refers to the optimal K recall contents;

[0091] 3. After completing the retrieval of the second cluster, update the top K. If the top K changes, the generation in step 2 is invalid and needs to be regenerated;

[0092] 4. Repeat steps 2 and 3 until the last cluster is retrieved.

[0093] There are two problems in the above method:

[0094] 1. Since the cluster division is not related to the retrieval query, the hit rate of speculative generation cannot be guaranteed, which may easily lead to multiple invalid generation, thus preempting normal generation requests;

[0095] 2. In the top K update process, there is a strong dependence on the order of retrieval results. When a single result is hit but the order is different, for example, the results returned by the first cluster are D1 and D2, and the results returned by the second cluster are D2 and D3, the prefix cache cannot be hit in the RAG cache, and the generation time cannot be effectively reduced.

[0096] In order to solve such a problem, an embodiment of the present application proposes a retrieval enhancement generation solution.

[0097] like Figure 1 The figure shows an application scenario diagram of a retrieval enhancement generation method in an embodiment of the present application. Figure 1 The system includes: a service terminal 10, an application 110, a large language model 120, and a cache device 130 arranged in the service terminal.

[0098] The service terminal 10 is a device or software entity that integrates service provision and request processing functions, and can be run in a client environment or deployed in a server environment. It can receive and process requests from users, execute corresponding service logic, and return the processing results to the requester.

[0099] The application 110 is a software program that enables an end user to interact with the pre-trained large language model 120 by providing a user-friendly interface.

[0100] The Big Language Model 120 is a highly sophisticated artificial intelligence system that uses deep learning technology to learn language knowledge from large amounts of text and can be applied to a variety of natural language processing tasks, such as virtual assistants and chatbots, customer service, education, data analysis and other scenarios.

[0101] The cache device 130 can be used to cache the output, intermediate results or parameters of the large language model 120 to improve the efficiency and response speed of the model service.

[0102] It should be noted that if additional modules are added to the illustrated environment or individual modules are removed therefrom, the underlying concepts of the example embodiments of the present application will not be altered.

[0103] It should be noted that the search enhancement generation method proposed in this application is not only applicable to Figure 1 The application scenario shown is also applicable to any device that has a retrieval enhancement generation requirement.

[0104] Figure 2 A flowchart of a search enhancement generation method provided by an embodiment of the present application is shown. Figure 1 The service terminal 10 in the embodiment is executed. Figure 2 As shown, the method includes:

[0105] Step 201: Use multiple parallel channels to search for query requests from multiple dimensions.

[0106] Parallel channels are independent processing paths that can run simultaneously, and each channel can process data independently. These channels can be physical, such as multiple processors or servers, or logical, such as multiple threads or processes on the same processor.

[0107] By using multiple parallel channels to retrieve query requests from multiple dimensions, query requests can be analyzed from different angles or features at the same time, which can more comprehensively understand the user's intentions.

[0108] In some embodiments, retrieval may be performed from the following dimensions, such as text content, semantic understanding, user behavior, geographic location, time information, etc. Each channel independently retrieves relevant information from the corresponding dimension for the query assigned to it.

[0109] The specific dimensions to be searched can be determined according to the scenario to which the query request belongs, and a mapping relationship between each channel and the search dimension can be pre-established for each scenario. Therefore, when a query request is received, the scenario to which the query request belongs can be determined first, and then the dimensions of each channel to be searched can be determined according to the scenario.

[0110] Taking the financial field as an example, the scenario to which the query request belongs may involve questions and answers between financial users and financial product customer service, investment advice, financial news analysis, financial product recommendations, etc. For example, assuming that the scenario to which the query request belongs is determined to be financial product recommendation, then according to the pre-established mapping relationship between each channel and the search dimension, it can be determined that channel 1 searches from the perspective of financial viewpoints, channel 2 searches from the perspective of financial information, channel 3 searches from the perspective of market dynamics, and so on.

[0111] Step 202 , pre-filling the recalled content of each channel according to the time sequence in which each channel returns the recalled content.

[0112] The pre-filling result of each channel is stored in a cache device (also called a prefix cache), and the pre-filling result of the channel is obtained according to the pre-filling result cached by the previous channel that returns the recalled content.

[0113] In the disclosed embodiment, each channel retrieves the query content from different dimensions. Since the data volume of the data source or the data volume of the recalled content retrieved by each channel may be different, or the retrieval algorithm used by each channel may be different, or the resource allocation of each channel is different, the time for each channel to return the recalled content may have a large difference. For the channel that returns the recalled content first, the content is first added to the prompt content (prompt), and the pre-filling calculation is performed in advance, and the pre-filling result of the channel is stored in the cache device. For the next channel that returns the recalled content, after the previous channel completes the pre-filling, the recalled content of the previous channel is merged with the recalled content of the channel, and the pre-filling calculation is performed together. Since the pre-filling result has been calculated for the previous part of the recalled content of the merged content, the channel only performs the pre-filling calculation for its own recalled content, and retrieves the pre-filling result of the previous channel from the cache device, combined with the pre-filling result corresponding to the recalled content of the channel itself, to obtain the pre-filling result finally output by the channel. For the subsequent channels that return the recalled content, the pre-filling results are obtained in this way.

[0114] Step 203, obtaining the output result of the large language model according to the pre-filled result of the channel that finally returns the recalled content.

[0115] After obtaining the pre-filled result of the channel that finally returns the recalled content, the pre-filled result is input into the large language model. The large language model decodes the pre-filled result to convert the implicit state inside the model into the final text output result.

[0116] See also Figure 3The schematic diagram of the pre-filling method provided by the embodiment of the present disclosure is shown. Assume that a query statement is retrieved from three different dimensions through parallel channels 1, 2 and 3, respectively, and the recalled contents are R1, R2 and R3, respectively, and the order of returning the recalled contents is R1, R3 and R2, respectively. After channel 1 returns the recalled content R1, R1 is immediately pre-filled and calculated to obtain the pre-filled result P1. Next, channel 3 returns the recalled content R3. After the pre-filling calculation of channel 1 is completed, R1 and R3 are merged, and the pre-filling calculation is performed on the merged result. Since the pre-filling result of R1 has been stored in the cache device, the cache result can be directly called, and only the pre-filling result P3 of R3 is calculated. After completing the calculation of the R3 part, the pre-filling result (P1+P3) corresponding to channel 3 is obtained. Afterwards, channel 2 returns the recalled content R2. After the pre-filling calculation of channel 3 is completed, R1, R3 and R2 are merged, and the pre-filling calculation is performed on the merged result. During the calculation process, the pre-filled results (P1+P3) corresponding to R1 and R3 cached in channel 3 are directly called, and then combined with the pre-filled result P2 corresponding to R2, the pre-filled result (P1+P3+P2) corresponding to channel 3 is obtained. After obtaining the pre-filled result of channel 2, the pre-filled result is decoded to obtain the final output of the large model.

[0117] The retrieval enhancement generation method proposed in the embodiment of this specification pre-fills according to the time sequence of the recalled content returned by the channels. After a channel returns the recalled content, the pre-fill calculation is performed in advance, and there is no need to wait for all channels to complete the recall before pre-filling. By integrating the two stages of recall and generation, in the case of large differences in the time consumption of multi-channel recalls, the time difference of multi-channel recalls can be fully utilized to reduce the waiting time during a RAG request; and because the pre-filling is performed according to the time sequence of the recalled content returned by each channel, it is guaranteed that the pre-filling result of the current channel can be obtained by using the pre-filling result cached by the channel that previously returned the recalled content, avoiding repeated calculations in the pre-filling stage.

[0118] In some embodiments, when the time difference between multiple channels returning recalled contents is within a set range, the recalled contents of the multiple channels are merged and pre-filled according to the merged contents.

[0119] If the time it takes for multiple channels to complete recall is very similar, there is no need to wait too long for recall. Therefore, the contents of these channels can be merged first, and then pre-filled together based on the merged content to improve the efficiency of the pre-fill calculation.

[0120] In some embodiments, when the length returned by a channel is less than a set threshold, the recalled contents of the channels returned after the channel are merged until the merged length is greater than a first set threshold, and the merged contents are pre-filled. For example, the first set threshold can be set to 500 characters, and the merged contents are pre-filled when the merged length is greater than 500 characters.

[0121] If the recall content of a channel is very small, then the cost of pre-filling calculations for this channel alone may be higher than the time saved. In order to solve this problem, when it is determined that the length of the recall content of a channel is less than the first set threshold, the pre-filling calculation can be temporarily not performed in advance, but wait for the next channel to return the recall content. After the recall content of the two channels is merged, it is compared with the first set threshold. If it is still less than the first set threshold, the recall content returned by the next channel will continue to be merged until the length of the merged content is greater than the first set threshold, and then the merged content will be pre-filled together to effectively reduce the system overhead in the retrieval enhancement generation process.

[0122] Furthermore, if the sum of the lengths of the recalled contents of all channels is less than the second set threshold, or if the total time to complete the recall of all channels is greater than the third set threshold, the recalled contents are not pre-filled, and are directly encoded to obtain the output result of the large language model. For example, the second set threshold can be set to 2000 characters, and the third set threshold can be set to 2 seconds. That is, by setting the above two thresholds, requests with too short recalled contents or too long recalled times are eliminated to avoid the overhead of pre-filling calculations being higher than the overhead of direct encoding, which has a negative impact on the system.

[0123] In some embodiments, the recalled content of the channel is used as a key and the corresponding pre-filled result is stored in a cache device as a value. When the recalled content of the channel is merged with the recalled content of the previous channel and the merged content is used for pre-filling, when a key is retrieved in the merged content, the value corresponding to the key is obtained from the cache device as a part of the pre-filled result.

[0124] In the disclosed embodiment, pre-filling is performed in the chronological order of the recalled contents returned by each channel and cached with the recalled contents as the key and the corresponding pre-filled results as the value. Therefore, when each channel merges the recalled contents of the previous channel for pre-filling calculation, it is ensured that the pre-filled results corresponding to the recalled contents of the previous channel can be retrieved from the cache device, thereby avoiding repeated calculations in the pre-filling stage and effectively reducing system overhead.

[0125] Since the generation model usually requires a lot of computing resources, additional overhead will be generated when calling it. In order to manage the calling of the generation model, in the embodiment of the present disclosure, a scheduling queue can be introduced to decide the timing of initiating segment pre-filling.

[0126] Specifically, when entering the pre-fill queue, the queue can be designed according to the query request group, that is, a corresponding pre-fill queue is established for each query request, and the recall content of each channel obtained based on the query request is input into the same pre-fill queue. In addition, after the recall content of one channel is pre-filled, the recall content of the next channel is popped out of the pre-fill queue.

[0127] See also Figure 4 The schematic diagram of the prefilling method shown in the figure, each query request corresponds to a prefilling queue, and the content recalled from multiple channels for a query request all enter the same prefilling queue. For the recalled content in each prefilling queue, a prefilling scheduler (prefilling worker) is used to transmit the content popped out of the queue to the module for prefilling processing.

[0128] like Figure 4 As shown, the recalled contents R11, R12, and R13 returned from the three channels for query request 1 all enter pre-filling queue 1; the recalled contents R21, R22, and R23 returned from the three channels for query request 2 all enter pre-filling queue 2; and the recalled contents R31, R32, and R33 returned from the three channels for query request 3 all enter pre-filling queue 3. For each pre-filling queue, after the recalled contents of one channel are pre-filled, the recalled contents of the next channel are popped out. For example, assuming that for query request 1, the recall content R11 of channel 1 has been returned, while other channels are still waiting, then R11 will be sent to pre-filling scheduler 1 for pre-filling calculation at the next scheduling; assuming that for query request 2, the recall content R21 of channel 1 has completed the pre-filling calculation, the recall content R22 of channel 2 has been returned, and the third channel is still waiting, then R21 will be sent to pre-filling scheduler 2 for pre-filling calculation at the next scheduling; assuming that for query request 3, the recall content R21 of channel 1 has completed the pre-filling calculation, the recall content R22 of channel 2 and the recall content R23 of channel 3 have been returned, and the time difference between the two returns is within the set range, for example, the interval between the two is less than 0.1 seconds, then R32 and R33 will be sent together by the pre-filling scheduler 3 for pre-filling calculation at the next scheduling, instead of being scheduled separately.

[0129] Since the recalled contents returned by each channel for a request are sent to the pre-scheduling queue in sequence according to the order of return, and the recalled contents returned first are popped out of the queue first for pre-filling calculation, the priority device in the embodiment of the present disclosure does not change the priority of the recalled contents in the queue, and is actually processed according to the first-in-first-out principle.

[0130] Since each pre-fill queue pops up the recall content of the next channel after the recall content of one channel is pre-filled, it can ensure that only one pre-fill is being calculated for a query request at the same time, avoiding unnecessary parallel calculations, making more efficient use of computing resources, and dynamically assembling prompt content in first-in-first-out order, so that the hit rate of the prefix cache is increased to 100%.

[0131] In some embodiments, the number of pre-populated schedulers may be determined based on the concurrency supported by the large model.

[0132] Assuming that the large model supports a concurrency of n, then n pre-fill schedulers can be set up to extract content from the pre-fill queue to ensure that at most n requests enter the pre-fill settlement at the same time, thereby ensuring that there is sufficient capacity to handle such advance pre-fill requests during the model generation phase.

[0133] In the case of receiving multiple requests at the same time, in order to ensure the cost-effectiveness of pre-filling calculations, it is necessary to make decisions on the recall time and recall length of each request to avoid requests with very small time differences and very little recall content in multiple recalls. The additional overhead of these requests may be higher than the time saved by pre-filling in advance. It should be noted that the "simultaneous" here means that the receiving time difference is within a set range.

[0134] In some embodiments, when multiple requests are received simultaneously, the following information may be obtained for each request: the length L of the recalled content to be pre-filled in the current queue;

[0135] The time consumed for pre-filling per unit length is P;

[0136] The time taken to complete all recalls of the request in Tx;

[0137] The recall time of the recalled content in the current queue is T;

[0138] For each request, the maximum saved time Ts is determined according to the formula Min(L*P,Tx-T);

[0139] Prioritize individual requests based on maximum time savings.

[0140] Among them, L*P is the time taken to pre-fill the recalled content in the current queue, and Tx - T is the maximum time that can be saved by enabling pre-filling, that is, the waiting time that can be avoided if pre-filling is performed immediately. During the actual execution process, since Tx is unpredictable, it can be estimated based on the average time taken for each recall channel and according to the recall channel matched by the current request. When L*P < Tx - T, among multiple requests, if the value of L*P for a certain request is larger, it means that this request has a greater potential time savings. Therefore, the larger the value of L*P, the higher the priority of this request.

[0141] When L*P ≥ Tx - T, the time that can be saved by performing the pre-filling operation in advance is (Tx - T), which means that this request has a greater potential time savings. Therefore, the larger the value of Tx - T, the higher the priority of this request.

[0142] By determining the priority of requests through the above method, when multiple requests are received simultaneously, requests with higher cost performance can be selected for calculation, improving the overall response speed of the system and optimizing resource utilization.

[0143] Figure 5 is a block diagram of a retrieval enhanced generation device provided by an exemplary embodiment. As Figure 5 shown, the device includes:

[0144] A retrieval unit 501, configured to retrieve a query request from multiple dimensions respectively by using multiple parallel channels;

[0145] A pre-filling unit 502, configured to pre-fill the recalled content of each channel in the time sequence of the recalled content returned by each channel, where the pre-filling result of each channel is stored in a cache device, and the pre-filling result of this channel is obtained according to the pre-filling result cached by the previous channel that returned the recalled content;

[0146] An output unit 503, configured to obtain the output result of the large language model according to the pre-filling result of the channel that finally returns the recalled content.

[0147] In some embodiments, the device further includes a determination unit, configured to:

[0148] Determine the scenario to which the query request belongs;

[0149] Determine the dimensions of each channel for retrieval according to the scenario.

[0150] In some embodiments, the pre-filling unit is specifically configured to:

[0151] When the time difference between the recalled contents returned by the multiple channels is within a set range, the recalled contents of the multiple channels are merged and pre-filled according to the merged contents.

[0152] In some embodiments, the pre-filling unit is specifically used for:

[0153] When the length returned by a channel is less than a first set threshold, the recalled contents of the channels returned after the channel are merged until the merged length symbol is greater than the first set threshold, and the merged contents are pre-filled.

[0154] In some embodiments, the device further comprises a cache unit, configured to:

[0155] storing in a cache device using the recalled content of the channel as a key and the corresponding pre-filled result as a value;

[0156] The recalled content of the channel is merged with the recalled content of the previous channel, and the merged content is used for pre-filling, wherein when a key is retrieved in the merged content, a value corresponding to the key is obtained from the cache device as a part of the pre-filled result.

[0157] In some embodiments, the apparatus further comprises a queue unit configured to:

[0158] A corresponding pre-filled queue is established for each query request;

[0159] The recalled contents of each channel obtained based on the query request are input into the pre-filled queue;

[0160] Wherein, after the recalled content of one channel is pre-filled, the pre-filled queue pops out the recalled content of the next channel.

[0161] In some embodiments, the apparatus further comprises a priority unit configured to:

[0162] In case of receiving multiple requests at the same time, the following information is obtained for each request;

[0163] The length L of the recalled content to be pre-filled in the current queue;

[0164] The time consumed for pre-filling per unit length is P;

[0165] The time taken to complete all recalls of the request in Tx;

[0166] The recall time of the recalled content in the current queue is T;

[0167] For each request, the maximum saved time Ts is determined according to the formula Min(L*P,Tx-T);

[0168] The priority of the request is determined according to Ts corresponding to each request.

[0169] In some embodiments, the device further comprises a screening unit, configured to set the content of the request for recall that meets the following conditions to not be pre-filled:

[0170] The sum of the lengths of the recalled contents of all channels is less than a second set threshold; or,

[0171] The total time to complete the recall of all channels is greater than the third set threshold.

[0172] In some embodiments, the apparatus further comprises a scheduling unit, configured to:

[0173] The recalled content popped out from the pre-filling queue is transmitted to the module for pre-filling processing by using a set number of pre-filling schedulers, wherein the set number is determined according to the concurrency supported by the large model.

[0174] Figure 6 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0175] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0176] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0177] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0178] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0179] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0180] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0181] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0182] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0183] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A search enhancement generation method, comprising: Use multiple parallel channels to retrieve query requests from multiple dimensions; Pre-filling the recalled content of each channel according to the time sequence of the recalled content returned by each channel, wherein the pre-filling result of each channel is stored in a cache device, and the pre-filling result of the channel is obtained according to the pre-filling result cached by the previous channel that returned the recalled content; The output result of the large language model is obtained based on the pre-filled result of the channel that finally returns the recalled content.

2. The method according to claim 1, further comprising: Determining the scenario to which the query request belongs; The dimensions of each channel for retrieval are determined according to the scenario.

3. The method according to claim 1, wherein the pre-filling of the recalled content of each channel according to the time sequence of each channel returning the recalled content comprises: When the time difference between the recalled contents returned by the multiple channels is within a set range, the recalled contents of the multiple channels are merged and pre-filled according to the merged contents.

4. The method according to claim 1, wherein the pre-filling of the recalled content of each channel according to the time sequence of each channel returning the recalled content comprises: When the length returned by a channel is less than a first set threshold, the recalled contents of the channels returned after the channel are merged until the merged length symbol is greater than the first set threshold, and the merged contents are pre-filled.

5. The method according to claim 1, further comprising: storing in a cache device using the recalled content of the channel as a key and the corresponding pre-filled result as a value; The recalled content of the channel is merged with the recalled content of the previous channel, and the merged content is used for pre-filling, wherein when a key is retrieved in the merged content, a value corresponding to the key is obtained from the cache device as a part of the pre-filled result.

6. The method according to claim 1, further comprising: A corresponding pre-filled queue is established for each query request; The recalled contents of each channel obtained based on the query request are input into the pre-filled queue; Wherein, after the recalled content of one channel is pre-filled, the pre-filled queue pops out the recalled content of the next channel.

7. The method according to claim 1, further comprising: In case of receiving multiple requests at the same time, the following information is obtained for each request; The length L of the recalled content to be pre-filled in the current queue; The time consumed for pre-filling per unit length is P; The time taken to complete all recalls of the request in Tx; The recall time of the recalled content in the current queue is T; For each request, the maximum saved time Ts is determined according to the formula Min(L*P,Tx-T); The priority of the request is determined according to Ts corresponding to each request.

8. The method according to claim 1, further comprising requesting that the recalled content meet the following conditions without pre-filling: The sum of the lengths of the recalled contents of all channels is less than a second set threshold; or, The total time to complete the recall of all channels is greater than the third set threshold.

9. The method according to claim 1, further comprising: The recalled content popped out from the pre-filling queue is transmitted to the module for pre-filling processing by using a set number of pre-filling schedulers, wherein the set number is determined according to the concurrency supported by the large model.

10. A search enhancement generation device, comprising: A retrieval unit, used to use multiple parallel channels to retrieve the query request from multiple dimensions respectively; A pre-filling unit, used to pre-fill the recalled content of each channel according to the time sequence of each channel returning the recalled content, wherein the pre-filling result of each channel is stored in a cache device, and the pre-filling result of the channel is obtained according to the pre-filling result cached by the previous channel returning the recalled content; The output unit is used to obtain the output result of the large language model according to the pre-filled result of the channel that finally returns the recalled content.

11. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images