Search enhancement generation method and apparatus, device, and storage medium

Abstract

One or more embodiments of the specification provide a retrieval enhancement generation method, device, equipment and storage medium. The method uses multiple parallel channels to retrieve query requests from multiple dimensions respectively; then, according to the time sequence of the return recall content of each channel, the recall content of each channel is pre-filled, wherein the pre-filled result of each channel is stored in the cache device, and the pre-filled result of the channel is obtained according to the pre-filled result cached by the previous channel returning the recall content; finally, according to the pre-filled result of the channel returning the recall content last, the output result of the large language model is obtained. Since the pre-filling is performed according to the time sequence of the return recall content of the channel, the waiting time in the retrieval enhancement generation request process is reduced, and the pre-filled result of the current channel is calculated according to the pre-filled result cached by the previous channel returning the recall content, avoiding repeated calculation in the pre-filling stage.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to a search enhancement generation method, apparatus, device, and storage medium. Background Technology

[0002] In some domain-specific Retrieval-Augmented Generation (RAG) applications, such as in finance, there is often a significant reliance on highly specialized content, which typically involves large datasets. Furthermore, to achieve optimal generation results, the parameters of the generation model are often quite large. In such cases, the effectiveness of low-level model optimization is limited. Existing techniques typically utilize dynamic speculative pipelines to improve the overall throughput of RAG and reduce generation time. This method involves cluster-by-cluster retrieval and immediate generation, using the current top K results for initial generation, and then updating the top K with the results of new cluster retrievals.

[0003] However, in this method, since the cluster partitioning and retrieval query are unrelated, the hit rate of speculative generation cannot be guaranteed, which can easily lead to multiple invalid generation and thus preempt normal generation requests. Furthermore, during the update of the top K, there is a strong dependence on the order of retrieval results, and in some cases, the prefix cache cannot be hit in the RAG cache, which cannot effectively reduce the generation time. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a retrieval enhancement generation method, apparatus, device, and storage medium to effectively reduce generation time.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a retrieval enhancement generation method is proposed, comprising:

[0007] The query request is retrieved from multiple dimensions using multiple parallel channels.

[0008] According to the time order of the recall content returned by each channel, the recall content of each channel is pre-filled. The pre-filling result of each channel is stored in the cache device, and the pre-filling result of the channel is obtained based on the pre-filling result cached by the previous channel that returned the recall content.

[0009] The output of the large language model is obtained based on the pre-filled results of the channel that returned the last recalled content.

[0010] In some embodiments, the method further includes:

[0011] Determine the scenario to which the query request belongs;

[0012] The dimensions of each channel for retrieval are determined based on the scenario.

[0013] In some embodiments, the step of pre-filling the recall content of each channel according to the time sequence of the recall content returned by each channel includes:

[0014] If the time difference between the return of recall content from multiple channels is within a set range, the recall content from the multiple channels is merged and then pre-filled based on the merged content.

[0015] In some embodiments, the step of pre-filling the recall content of each channel according to the time sequence of the recall content returned by each channel includes:

[0016] When the length returned by a channel is less than a first set threshold, the recall content of the channels returned after that channel will be merged until the length of the merged content is greater than the first set threshold, at which point the merged content will be pre-filled.

[0017] In some embodiments, the method further includes:

[0018] The recalled content of the channel is used as the key and the corresponding pre-filled result is used as the value to store the data in the cache device;

[0019] The recall content of the channel is merged with the recall content of the previous channel, and the merged content is used for pre-filling. When a key is found in the merged content, the value corresponding to the key is obtained from the cache device as part of the pre-filling result.

[0020] In some embodiments, the method further includes:

[0021] Establish a corresponding pre-populated queue for each query request;

[0022] The retrieved content for each channel obtained based on the query request is input into the pre-filled queue;

[0023] The pre-filled queue pops up the recall content for the next channel after the recall content for one channel has been pre-filled.

[0024] In some embodiments, the method further includes:

[0025] When multiple requests are received simultaneously, the following information is obtained for each request;

[0026] The length L of the recall content to be pre-filled in the current queue;

[0027] The time P for pre-filling per unit length;

[0028] The time Tx for completing all recalls requested;

[0029] The recall time T for the content currently being recalled in the queue;

[0030] For each request, the maximum time saving Ts is determined according to the formula Min(L*P,Tx-T);

[0031] The priority of each request is determined based on its corresponding Ts.

[0032] In some embodiments, the method further includes not pre-populating request recall content if the following conditions are met:

[0033] The sum of the lengths of the recalled content from all channels is less than a second set threshold; or,

[0034] The total time for completing the recall across all channels exceeds the third set threshold.

[0035] In some embodiments, the method further includes:

[0036] A set number of pre-fill schedulers are used to transmit the recalled content popped from the pre-fill queue to the module for pre-filling processing, wherein the set number is determined according to the concurrency supported by the large model.

[0037] According to a second aspect of one or more embodiments of this specification, a retrieval enhancement generation apparatus is provided, comprising:

[0038] The retrieval unit is used to retrieve query requests from multiple dimensions using multiple parallel channels;

[0039] The pre-filling unit is used to pre-fill the recall content of each channel according to the time order of the recall content returned by each channel. The pre-filling result of each channel is stored in the cache device, and the pre-filling result of the channel is obtained based on the pre-filling result cached by the previous channel that returned the recall content.

[0040] The output unit is used to obtain the output of the large language model based on the pre-filled results of the channel that returned the last recalled content.

[0041] In some embodiments, the apparatus further includes a determining unit for:

[0042] Determine the scenario to which the query request belongs;

[0043] The dimensions of each channel for retrieval are determined based on the scenario.

[0044] In some embodiments, the pre-filling unit is specifically used for:

[0045] If the time difference between the return of recall content from multiple channels is within a set range, the recall content from the multiple channels is merged and then pre-filled based on the merged content.

[0046] In some embodiments, the pre-filling unit is specifically used for:

[0047] When the length returned by a channel is less than a first set threshold, the recall content of the channels returned after that channel will be merged until the length of the merged content is greater than the first set threshold, at which point the merged content will be pre-filled.

[0048] In some embodiments, the apparatus further includes a cache unit for:

[0049] The recalled content of the channel is used as the key and the corresponding pre-filled result is used as the value to store the data in the cache device;

[0050] The recall content of the channel is merged with the recall content of the previous channel, and the merged content is used for pre-filling. When a key is found in the merged content, the value corresponding to the key is obtained from the cache device as part of the pre-filling result.

[0051] In some embodiments, the apparatus further includes a queue unit for:

[0052] Establish a corresponding pre-populated queue for each query request;

[0053] The retrieved content for each channel obtained based on the query request is input into the pre-filled queue;

[0054] The pre-filled queue pops up the recall content for the next channel after the recall content for one channel has been pre-filled.

[0055] In some embodiments, the apparatus further includes a priority unit for:

[0056] When multiple requests are received simultaneously, the following information is obtained for each request;

[0057] The length L of the recall content to be pre-filled in the current queue;

[0058] The time P for pre-filling per unit length;

[0059] The time Tx for completing all recalls requested;

[0060] The recall time T for the content currently being recalled in the queue;

[0061] For each request, the maximum time saving Ts is determined according to the formula Min(L*P,Tx-T);

[0062] The priority of each request is determined based on its corresponding Ts.

[0063] In some embodiments, the apparatus further includes a filtering unit for setting request recall content that meets the following conditions to not be pre-populated:

[0064] The sum of the lengths of the recalled content from all channels is less than a second set threshold; or,

[0065] The total time for completing the recall across all channels exceeds the third set threshold.

[0066] In some embodiments, the apparatus further includes a scheduling unit for:

[0067] A set number of pre-fill schedulers are used to transmit the recalled content popped from the pre-fill queue to the module for pre-filling processing, wherein the set number is determined according to the concurrency supported by the large model.

[0068] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0069] processor;

[0070] Memory used to store processor-executable instructions;

[0071] The processor implements the steps of the method proposed in the above embodiments by running the executable instructions.

[0072] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method proposed in the above embodiments.

[0073] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method proposed in the above embodiments.

[0074] The retrieval enhancement generation method proposed in this specification utilizes multiple parallel channels to retrieve query requests from multiple dimensions. Then, according to the chronological order of the recalled content returned by each channel, the recalled content of each channel is pre-filled. The pre-filled result of each channel is stored in a cache device, and the pre-filled result of that channel is obtained based on the pre-filled result cached by the channel that returned the recalled content previously. Finally, the output result of the large language model is obtained based on the pre-filled result of the channel that returned the last recalled content. Because pre-filling is performed according to the chronological order of the recalled content returned by the channels, the waiting time in a single retrieval enhancement generation request process is reduced, and the pre-filled result of the current channel is calculated based on the pre-filled result cached by the channel that returned the recalled content previously, avoiding redundant calculations in the pre-filling stage. Attached Figure Description

[0075] Figure 1 This is an exemplary embodiment of an application scenario diagram of a retrieval enhancement generation method.

[0076] Figure 2 This is a flowchart of a retrieval enhancement generation method provided in an exemplary embodiment.

[0077] Figure 3 This is a schematic diagram of a pre-filling method provided in an exemplary embodiment.

[0078] Figure 4 This is a schematic diagram of another pre-filling method provided in an exemplary embodiment.

[0079] Figure 5 This is a block diagram of a retrieval enhancement generation apparatus provided in an exemplary embodiment.

[0080] Figure 6 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0081] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0082] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0083] To aid in understanding this manual, the technical terms related to this manual will be introduced first.

[0084] Retrieval-Augmented Generation (RAG) is a model that combines information retrieval and generation techniques. This technology generates answers or content by referencing information from external knowledge bases and can be used in various natural language processing tasks.

[0085] Prefilling: A technique used in the training and inference of large models. Before running a large model, some known or pre-computable information is input into the model to reduce the amount of real-time computation, reduce redundant calculations, and accelerate the inference process.

[0086] Decoding: In natural language processing tasks, this is the process by which the model generates an output sequence based on the information it has already processed (such as user input prompts or context). Decoding is typically performed after the prefilling stage.

[0087] In some RAG applications, such as in the financial sector, there is a heavy reliance on highly specialized content, which typically involves large datasets. Furthermore, to achieve good generation results, the parameters of the generation model are often quite large. In such cases, the effectiveness of optimizing the underlying model is limited.

[0088] Existing technologies typically utilize dynamic speculative pipelines to improve overall RAG throughput and reduce generation time. This method is generally implemented through the following steps:

[0089] 1. Divide the vector search into several clusters, and traverse each cluster to perform the search;

[0090] 2. Once the top K results from the first cluster are obtained, they are sent to the model for generation in advance; where top K refers to the K best recalled results.

[0091] 3. After completing the retrieval of the second cluster, update the top K. If the top K changes, the generation in step 2 is invalid and needs to be regenerated.

[0092] 4. Repeat steps 2 and 3 until the retrieval of the last cluster is completed.

[0093] The above method has two problems:

[0094] 1. Since cluster partitioning and retrieval queries are unrelated, the hit rate of speculative generation cannot be guaranteed, which can easily lead to multiple invalid generation attempts, thereby preempting normal generation requests.

[0095] 2. During the top K update process, there is a strong dependence on the order of the search results. If a single result is hit but the order is different, for example, the first cluster returns D1 and D2, and the second cluster returns D2 and D3, then the prefix cache cannot be hit in the RAG cache, and the generation time cannot be effectively reduced.

[0096] To address this problem, this application proposes a search enhancement generation scheme.

[0097] like Figure 1 The diagram shown illustrates an application scenario of a retrieval enhancement generation method as described in this application. Figure 1 It includes: a service terminal 10 and an application 110, a large language model 120, and a cache device 130 installed in the service terminal.

[0098] The service terminal 10 is a device or software entity that integrates service provision and request processing functions. It can run in both client and server environments. It can receive and process requests from users, execute corresponding service logic, and return the processing results to the requester.

[0099] Application 110 is a software program that enables end users to interact with a pre-trained large language model 120 by providing a user-friendly interface.

[0100] Big Language Model 120 is a highly complex artificial intelligence system that learns language knowledge from large amounts of text using deep learning technology and can be applied to a variety of natural language processing tasks, such as virtual assistants and chatbots, customer service, education, and data analysis.

[0101] The caching device 130 can be used to cache the output, intermediate results, or parameters of the large language model 120 to improve the efficiency and response speed of the model service.

[0102] It should be noted that adding additional modules to or removing individual modules from the illustrated environment does not change the underlying concept of the example embodiments of this application.

[0103] It should be noted that the retrieval enhancement generation method proposed in this application is not only applicable to... Figure 1 The application scenarios shown can also be applied to any device that has a need for enhanced retrieval generation.

[0104] Figure 2 The flowchart shown is a retrieval enhancement generation method provided in an embodiment of this application. The method consists of... Figure 1 The service terminal 10 in the middle executes. For example... Figure 2 As shown, the method includes:

[0105] Step 201: Use multiple parallel channels to retrieve the query request from multiple dimensions.

[0106] Parallel channels refer to independent processing paths that can run concurrently, with each channel processing data independently. These channels can be physical, such as multiple processors or servers, or logical, such as multiple threads or processes on the same processor.

[0107] By using multiple parallel channels to retrieve query requests from multiple dimensions, the query requests can be analyzed from different angles or features simultaneously, which allows for a more comprehensive understanding of the user's intent.

[0108] In some embodiments, retrieval can be performed from dimensions such as text content, semantic understanding, user behavior, geographic location, time information, etc. Each channel independently retrieves relevant information from the corresponding dimension for the query assigned to it.

[0109] The specific dimensions to be searched can be determined based on the context of the query request. For each context, a mapping relationship between each channel and the search dimensions can be pre-established. Therefore, upon receiving a query request, the context of the query request can be determined first, and then the dimensions of each channel to be searched can be determined based on the context.

[0110] Taking the financial sector as an example, the scenario to which a query request pertains may involve Q&A between financial users and financial product customer service, investment advice, financial news analysis, financial product recommendations, and so on. For instance, assuming the scenario to which the query request pertains is determined to be financial product recommendations, then based on the pre-established mapping relationship between various channels and search dimensions, it can be determined that channel 1 searches from the perspective of financial viewpoints, channel 2 searches from the perspective of financial information, channel 3 searches from the perspective of market dynamics, and so on.

[0111] Step 202: Pre-fill the recall content for each channel according to the time sequence of the recall content returned by each channel.

[0112] The pre-filled result for each channel is stored in a cache device (also known as a prefix cache), and the pre-filled result for that channel is obtained based on the pre-filled result cached by the channel that returned the recall content.

[0113] In this embodiment, each channel retrieves the query content from different dimensions. Since the data volume of the data source or the amount of recalled content retrieved by each channel may differ, or the retrieval algorithms used by each channel may differ, or the resource allocation of each channel may differ, the time it takes for each channel to return the recalled content may vary significantly. For the channel that returns the recalled content first, the content is added to the prompt for pre-filling calculation, and the pre-filling result of that channel is stored in the cache. For the next channel that returns the recalled content, after the previous channel completes its pre-filling, the recalled content of the previous channel is merged with the recalled content of this channel for joint pre-filling calculation. Since the pre-filling result of the first part of the merged recalled content has already been calculated, this channel only performs pre-filling calculation on its own recalled content, retrieves the pre-filling result of the previous channel from the cache, and combines it with the pre-filling result corresponding to its own recalled content to obtain the final pre-filling result output by that channel. Subsequent channels returning recalled content all obtain their pre-filling results in this manner.

[0114] Step 203: Based on the pre-filling results of the channel that returned the final recalled content, obtain the output results of the large language model.

[0115] After obtaining the pre-filled result of the channel that returned the final recalled content, the pre-filled result is input into the large language model. The large language model decodes the pre-filled result and converts the implicit state inside the model into the final text output result.

[0116] See Figure 3The diagram illustrates the pre-filling method provided in this embodiment of the present disclosure. Assume that a query is retrieved from three different dimensions using parallel channels 1, 2, and 3, with retrieved contents R1, R2, and R3 respectively, and the order in which the retrieved contents are returned is R1, R3, R2. After channel 1 returns the retrieved content R1, pre-filling calculation is immediately performed on R1 to obtain the pre-filled result P1. Next, channel 3 returns the retrieved content R3. After the pre-filling calculation of channel 1 is completed, R1 and R3 are merged, and pre-filling calculation is performed on the merged result. Since the pre-filled result of R1 is already stored in the cache device, the cached result can be directly called, and only the pre-filled result P3 of R3 needs to be calculated. After completing the calculation of R3, the pre-filled result (P1+P3) corresponding to channel 3 is obtained. Then, channel 2 returns the retrieved content R2. After the pre-filling calculation of channel 3 is completed, R1, R3, and R2 are merged, and pre-filling calculation is performed on the merged result. During the calculation, the pre-filled results (P1+P3) corresponding to R1 and R3 cached in channel 3 are directly called, and then combined with the pre-filled result P2 corresponding to R2 to obtain the pre-filled result (P1+P3+P2) corresponding to channel 3. After obtaining the pre-filled result of channel 2, the pre-filled result is decoded to obtain the final output of the large model.

[0117] The retrieval enhancement generation method proposed in this specification pre-fills the data according to the chronological order of the recalled content returned by each channel. This allows for pre-filling calculations to be performed as soon as a channel returns recalled content, eliminating the need to wait for all channels to complete their recall before pre-filling. By merging the recall and generation stages, this method can fully utilize the time difference between multiple recalls, reducing the waiting time in a single RAG request, especially when there are significant differences in recall time. Furthermore, because pre-filling is performed according to the chronological order of the recalled content returned by each channel, it ensures that the pre-filling result of the current channel can be obtained using the pre-filling result cached by the channel that returned recalled content previously, avoiding redundant calculations in the pre-filling stage.

[0118] In some embodiments, when the time difference between the return of recalled content from multiple channels is within a set range, the recalled content from the multiple channels is merged and then pre-filled based on the merged content.

[0119] If the recall times of multiple channels are very close, and there is no need to wait too long for the recall, the contents of these channels can be merged first, and then pre-filled together based on the merged contents to improve the efficiency of pre-filling calculation.

[0120] In some embodiments, when the length returned by a channel is less than a set threshold, the recalled content of channels returned after that channel is merged until the length of the merged content exceeds a first set threshold, at which point the merged content is pre-padded. For example, the first set threshold can be set to 500 characters, and the merged content is pre-padded only when the length of the merged content exceeds 500 characters.

[0121] If the recalled content for a channel is very limited, the overhead of pre-filling calculations for that channel alone may outweigh the time saved. To address this issue, if the length of the recalled content for a channel is determined to be less than a first preset threshold, pre-filling calculations can be temporarily withheld, and the system can wait for the recalled content from the next channel. After merging the recalled content from these two channels, it is compared with the first preset threshold. If it is still less than the first preset threshold, the system continues to merge the recalled content from the next channel until the length of the merged content exceeds the first preset threshold. The merged content is then used for pre-filling, effectively reducing the system overhead during the search enhancement generation process.

[0122] Furthermore, if the sum of the lengths of the recalled content across all channels is less than a second set threshold, or if the total recall time across all channels exceeds a third set threshold, then the recalled content is not pre-filled and is directly encoded to obtain the output of the large language model. For example, the second set threshold can be set to 2000 characters, and the third set threshold can be set to 2 seconds. That is, by setting the above two thresholds, requests with too short recalled content or too long recall time are eliminated to avoid the overhead of pre-filling calculations exceeding the overhead of direct encoding, thus negatively impacting the system.

[0123] In some embodiments, the recalled content of the channel is stored in a cache device as the key and the corresponding pre-filled result is stored as the value. When the recalled content of the channel is merged with the recalled content of the previous channel and the merged content is used for pre-filling, when a key is found in the merged content, the value corresponding to the key is obtained from the cache device as part of the pre-filled result.

[0124] In this embodiment, the pre-filling is performed according to the time sequence of the recall content returned by each channel, and the recall content is used as the key and the corresponding pre-filling result is used as the value for caching. Therefore, when each channel merges the recall content of the previous channel for pre-filling calculation, it can ensure that the pre-filling result corresponding to the recall content of the previous channel can be retrieved from the cache device, avoiding repeated calculations in the pre-filling stage and effectively reducing system overhead.

[0125] Since generative models typically require significant computational resources, invoking them incurs additional overhead. To manage generative model invocations, a scheduling queue can be introduced in this embodiment to determine when to initiate segmented pre-filling.

[0126] Specifically, when entering the pre-filled queue, the queue can be designed according to the query requests. That is, a corresponding pre-filled queue is established for each query request, and the recall content of each channel obtained based on the query request is input into the same pre-filled queue. Furthermore, after the recall content of one channel is pre-filled, the recall content of the next channel is popped out of the pre-filled queue.

[0127] See Figure 4 The diagram illustrates the pre-filling method. Each query request corresponds to a pre-filling queue, and content retrieved from multiple channels for a single query request enters the same pre-filling queue. For the retrieved content in each pre-filling queue, a pre-filling worker is used to transfer the content popped from the queue to the module for pre-filling processing.

[0128] like Figure 4 As shown, for query request 1, the recall content R11, R12, and R13 returned from the three channels respectively enter pre-fill queue 1; for query request 2, the recall content R21, R22, and R23 returned from the three channels respectively enter pre-fill queue 2; and for query request 3, the recall content R31, R32, and R33 returned from the three channels respectively enter pre-fill queue 3. For each pre-fill queue, after the recall content of one channel is pre-filled, the recall content of the next channel is popped. For example, suppose that for query request 1, the recall content R11 of channel 1 has been returned, while other channels are still waiting, then R11 will be sent to the pre-fill scheduler 1 for pre-fill calculation in the next scheduling. Suppose that for query request 2, the recall content R21 of channel 1 has been pre-filled, the recall content R22 of channel 2 has been returned, while the third channel is still waiting, then R21 will be sent to the pre-fill scheduler 2 for pre-fill calculation in the next scheduling. Suppose that for query request 3, the recall content R21 of channel 1 has been pre-filled, the recall content R22 of channel 2 and the recall content R23 of channel 3 have been returned, and the time difference between their returns is within a set range, for example, the interval is less than 0.1 seconds, then R32 and R33 will be sent to the pre-fill scheduler 3 together for pre-fill calculation in the next scheduling, instead of being scheduled separately.

[0129] Since the recalled content returned by each channel for a request is sent into the pre-scheduling queue in the order of return, and the recalled content returned first is popped out of the queue for pre-filling calculation, the priority device in this embodiment does not change the priority of the recalled content in the queue, and is processed according to the first-in-first-out principle in actual execution.

[0130] Since each pre-filled queue pops up the next channel's recall content only after the recall content of one channel has been pre-filled, it can ensure that only one pre-filled content is being calculated for a query request at a time, avoiding unnecessary parallel computation, making more efficient use of computing resources, and dynamically assembling the prompt content according to the first-in-first-out order, thus increasing the hit rate of the prefix cache to 100%.

[0131] In some embodiments, the number of pre-filled schedulers can be determined based on the concurrency supported by the large model.

[0132] Assuming the large model supports a concurrency of n, then n pre-fill schedulers can be set up to extract content from the pre-fill queue, ensuring that at most n requests enter the pre-fill settlement at the same time. This guarantees that there is sufficient capacity to handle such pre-fill requests during the model generation phase.

[0133] When multiple requests are received simultaneously, to ensure the cost-effectiveness of pre-filling computation, decisions need to be made regarding the recall time and recall length for each request. This avoids requests with extremely small time differences and very limited recall content in multi-path recall, as the additional overhead of these requests may outweigh the time saved by pre-filling. It's important to note that "simultaneously" here refers to the receiving time difference being within a set range.

[0134] In some embodiments, when multiple requests are received simultaneously, the following information can be obtained for each request: the length L of the recall content to be pre-filled in the current queue;

[0135] The time P for pre-filling per unit length;

[0136] The time Tx for completing all recalls requested;

[0137] The recall time T for the content currently being recalled in the queue;

[0138] For each request, the maximum time saving Ts is determined according to the formula Min(L*P,Tx-T);

[0139] Prioritize each request based on the maximum time saved.

[0140] Among them, L*P is the time taken to pre-fill the recalled content in the current queue, and Tx - T is the maximum time that can be saved by enabling pre-filling, that is, the waiting time that can be avoided if pre-filling is performed immediately. In the actual execution process, since Tx is unpredictable, it can be estimated based on the average time consumption of each recall channel and according to the recall channel matched by the current request. When L*P < Tx - T, among multiple requests, if the value of L*P of a certain request is larger, it means that this request has a greater potential time savings. Therefore, the larger the value of L*P, the higher the priority of this request.

[0141] When L*P ≥ Tx - T, the time that can be saved by performing the pre-filling operation in advance is (Tx - T), which means that this request has a greater potential time savings. Therefore, the larger the value of Tx - T, the higher the priority of this request.

[0142] By determining the priority of requests through the above method, when multiple requests are received simultaneously, requests with higher cost performance can be selected for calculation, improving the overall response speed of the system and optimizing resource utilization.

[0143] Figure 5 is a block diagram of a retrieval enhanced generation device provided by an exemplary embodiment. As Figure 5 shown, the device includes:

[0144] A retrieval unit 501, configured to retrieve the query request from multiple dimensions respectively by using multiple parallel channels;

[0145] A pre-filling unit 502, configured to pre-fill the recalled content of each channel in the time sequence of the recalled content returned by each channel, wherein the pre-filling result of each channel is stored in the cache device, and the pre-filling result of this channel is obtained according to the pre-filling result cached by the previous channel that returns the recalled content;

[0146] An output unit 503, configured to obtain the output result of the large language model according to the pre-filling result of the channel that finally returns the recalled content.

[0147] In some embodiments, the device further includes a determination unit, configured to:

[0148] Determine the scenario to which the query request belongs;

[0149] Determine the dimensions of each channel for retrieval according to the scenario.

[0150] In some embodiments, the pre-filling unit is specifically configured to: [[ID=3C]]

[0151] If the time difference between the return of recall content from multiple channels is within a set range, the recall content from the multiple channels is merged and then pre-filled based on the merged content.

[0152] In some embodiments, the pre-filling unit is specifically used for:

[0153] When the length returned by a channel is less than a first set threshold, the recall content of the channels returned after that channel will be merged until the length of the merged content is greater than the first set threshold, at which point the merged content will be pre-filled.

[0154] In some embodiments, the apparatus further includes a cache unit for:

[0155] The recalled content of the channel is used as the key and the corresponding pre-filled result is used as the value to store the data in the cache device;

[0156] The recall content of the channel is merged with the recall content of the previous channel, and the merged content is used for pre-filling. When a key is found in the merged content, the value corresponding to the key is obtained from the cache device as part of the pre-filling result.

[0157] In some embodiments, the apparatus further includes a queue unit for:

[0158] Establish a corresponding pre-populated queue for each query request;

[0159] The retrieved content for each channel obtained based on the query request is input into the pre-filled queue;

[0160] The pre-filled queue pops up the recall content for the next channel after the recall content for one channel has been pre-filled.

[0161] In some embodiments, the apparatus further includes a priority unit for:

[0162] When multiple requests are received simultaneously, the following information is obtained for each request;

[0163] The length L of the recall content to be pre-filled in the current queue;

[0164] The time P for pre-filling per unit length;

[0165] The time Tx for completing all recalls requested;

[0166] The recall time T for the content currently being recalled in the queue;

[0167] For each request, the maximum time saving Ts is determined according to the formula Min(L*P,Tx-T);

[0168] The priority of each request is determined based on its corresponding Ts.

[0169] In some embodiments, the apparatus further includes a filtering unit for setting request recall content that meets the following conditions to not be pre-populated:

[0170] The sum of the lengths of the recalled content from all channels is less than a second set threshold; or,

[0171] The total time for completing the recall across all channels exceeds the third set threshold.

[0172] In some embodiments, the apparatus further includes a scheduling unit for:

[0173] A set number of pre-fill schedulers are used to transmit the recalled content popped from the pre-fill queue to the module for pre-filling processing, wherein the set number is determined according to the concurrency supported by the large model.

[0174] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, memory 608, and non-volatile memory 610, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into memory 608 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0175] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0176] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0177] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of computer-readable media.

[0178] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0179] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0181] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0182] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0183] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

Claims

1. A method for retrieving and enhancing generation, comprising: retrieving a query request from multiple dimensions using multiple parallel channels respectively; pre-populating the retrieval results of each channel in time order of the retrieval results returned by each channel, wherein the pre-population result of each channel is stored in a cache device, and the pre-population result of the channel is obtained according to the pre-population result cached by the channel that returns the retrieval result first; obtaining an output result of a large language model according to the pre-population result of the channel that returns the retrieval result last.

2. The method of claim 1, further comprising: determining a scenario to which the query request belongs; determining the dimensions of each channel for retrieval according to the scenario.

3. The method of claim 1, wherein the pre-populating the retrieval results of each channel in time order of the retrieval results returned by each channel comprises: in a case where the time difference of the retrieval results returned by multiple channels is within a set range, merging the retrieval results of the multiple channels, and then pre-populating according to the merged results.

4. The method of claim 1, wherein the pre-populating the retrieval results of each channel in time order of the retrieval results returned by each channel comprises: when the length of the retrieval result returned by a channel is less than a first set threshold, merging the retrieval results of the channels returned after the channel until the length of the merged results is greater than the first set threshold, and then pre-populating the merged results.

5. The method of claim 1, further comprising: storing in the cache device the retrieval results of the channels as keys and the corresponding pre-population results as values; merging the retrieval results of the channels with the retrieval results of the previous channel, and pre-populating using the merged results, wherein when a key is searched in the merged results, the value corresponding to the key is obtained from the cache device as part of the pre-population result.

6. The method of claim 1, further comprising: establishing a corresponding pre-population queue for each query request; inputting the retrieval results of each channel obtained based on the query request into the pre-population queue; wherein the pre-population queue pops out the retrieval results of the next channel after the retrieval results of a channel are completed pre-population.

7. The method of claim 1, further comprising: in a case where multiple requests are received simultaneously, obtaining the following information for each request; the length L of the retrieval results to be pre-populated in the current queue; the time P consumed for pre-population per unit length; the time Tx consumed for completing all retrievals of the request; the retrieval time T of the retrieval results in the current queue; for each request, determining the maximum time saved Ts according to the formula Min(L*P, Tx-T); determining the priority of the request according to Ts corresponding to each request.

8. The method of claim 1, further comprising not pre-populating the retrieval results of a request that meets the following conditions: the sum of the lengths of the retrieval results of all channels is less than a second set threshold; or the total time for completing the retrievals of all channels is greater than a third set threshold.

9. The method of claim 1, further comprising: The recall content popped from the pre-population queue is transmitted to a module for pre-population processing by a set number of pre-population schedulers, wherein the set number is determined according to a concurrency supported by the large language model.

10. A retrieval enhancement generation apparatus, comprising: a retrieval unit configured to retrieve a query request from multiple dimensions respectively by using multiple parallel channels; a pre-population unit configured to pre-populate the recall content of each channel in a time sequence of returning the recall content of each channel, wherein the pre-population result of each channel is stored in a cache apparatus, and the pre-population result of the channel is obtained according to the pre-population result cached by a previous channel returning the recall content; an output unit configured to obtain an output result of a large language model according to the pre-population result of a last channel returning the recall content.

11. An electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any one of claims 1-9 by running the executable instructions.

12. A computer-readable storage medium having stored thereon computer instructions, which when executed by a processor, implement the steps of the method of any one of claims 1-9.

13. A computer program product, comprising computer program / instructions, which when executed by a processor, implement the steps of the method of any one of claims 1-9.

Images