Method and apparatus for determining prefill path, electronic device, storage medium and product

Abstract

The application discloses a prefill path determination method and device, electronic equipment, storage medium and product, and relates to the technical field of communication. The method comprises the following steps: detecting whether first cache data exists in a plurality of cache data and is the same as a first prompt word input by a target object; in the case where it is determined that the first cache data exists, the first time for prefilling the first prompt word and the second time for loading the first cache data are predicted respectively, and the storage position of the first cache data is determined; and the prefill path of the first prompt word is determined according to the size relationship between the first time and the second time, or according to the storage position. Therefore, the problem that the prefill efficiency of the prompt word is not high due to the use of fixed logic to prefill the prompt word input by the user in the related art can be solved, the limitation of the traditional fixed multiplexing cache strategy is broken through, and the prefill efficiency is effectively improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, apparatus, electronic device, storage medium and product for determining a pre-filled path. Background Technology

[0002] With the widespread application of large models in dialogue systems, intelligent customer service and other scenarios, the length of prompts input by users has increased significantly, and multi-turn interactions and long context requests are becoming more common, leading to a sharp increase in the computational overhead of the pre-filling stage.

[0003] Existing large model inference engines generally adopt a fixed pre-filling logic of "direct loading when the cache is hit and forced recalculation when the cache is missed". This results in blind loading even when the cache is unloaded to a low-speed medium, which takes much longer than recalculation. Or, in long prompt word scenarios, high-complexity O(N²) attention calculations are repeatedly executed due to ignoring the cache, resulting in a significant increase in inference latency and waste of GPU memory and bandwidth resources.

[0004] There is no effective solution yet to address the problem of low efficiency in pre-filling prompts by using fixed logic in related technologies. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, storage medium, and product for determining a pre-filled path, to at least solve the problem in the related art where the use of fixed logic to pre-fill prompts input by the user results in low efficiency in pre-filling prompts.

[0006] This application provides a method for determining a pre-filled path, comprising: detecting whether there is first cached data among multiple cached data that is the same as a first prompt word input by a target object; if the existence of the first cached data is determined, predicting a first time for pre-filling the first prompt word and a second time for loading the first cached data, and determining the storage location of the first cached data; determining the pre-filled path of the first prompt word according to the relationship between the first time and the second time, or according to the storage location, wherein the pre-filled path includes one of the following: pre-filling the first prompt word and loading the first cached data.

[0007] This application also provides a pre-filling path determination device, comprising: a detection module, configured to detect whether there is first cached data among multiple cached data that is the same as a first prompt word input by a target object; a prediction module, configured to, when it is determined that the first cached data exists, predict a first time for pre-filling the first prompt word and a second time for loading the first cached data, and determine the storage location of the first cached data; and a determination module, configured to determine the pre-filling path of the first prompt word based on the relationship between the first time and the second time, or based on the storage location, wherein the pre-filling path includes one of the following: pre-filling the first prompt word and loading the first cached data.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described methods for determining a pre-filled path when executing the computer program.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described methods for determining a pre-filled path.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for determining a pre-filled path.

[0011] This application detects whether there is a first cached data in multiple cached data sets that is identical to the first prompt word input by the target object; if the existence of the first cached data is determined, the first time for pre-filling the first prompt word and the second time for loading the first cached data are predicted respectively, and the storage location of the first cached data is determined; the pre-filling path of the first prompt word is determined according to the relationship between the first time and the second time, or according to the storage location, wherein the pre-filling path includes one of the following: pre-filling the first prompt word and loading the first cached data. Therefore, it can solve the problem in related technologies where the use of fixed logic to pre-fill the prompt word input by the user results in low pre-filling efficiency, breaks through the limitations of the traditional fixed reuse caching strategy, and effectively improves the pre-filling efficiency. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1A hardware structure block diagram of a method for determining a pre-filled path provided in an embodiment of this application;

[0014] Figure 2 A flowchart illustrating a method for determining a pre-filled path, as provided in an embodiment of this application;

[0015] Figure 3 A schematic diagram illustrating a method for determining a pre-filled path provided in an embodiment of this application;

[0016] Figure 4 This is a structural block diagram of a pre-filled path determination device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] The methods and embodiments provided in this application can be executed in an inference engine or a similar computing device. Taking running on an inference engine as an example, Figure 1 This is a hardware structure block diagram of a method for determining a pre-filled path according to an embodiment of this application. For example... Figure 1 As shown, an inference engine may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or field-programmable gate array (FPGA)) and a memory 104 for storing data are also shown. The inference engine may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the inference engine described above. For example, the inference engine may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0020] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the pre-filled path determination method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the aforementioned pre-filled path determination method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the inference engine via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0021] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the inference engine. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0022] The embodiments of this application provide a method for determining a pre-filled path, which is applied to the inference engine described above. Figure 2 This is a flowchart of a method for determining a pre-filled path according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0023] Step S202: Detect whether there is a first cached data that is the same as the first prompt word input by the target object among multiple cached data;

[0024] Step S204: If it is determined that the first cached data exists, predict the first time for pre-filling the first prompt word and the second time for loading the first cached data, and determine the storage location of the first cached data.

[0025] Step S206: Determine the pre-filling path of the first prompt word based on the size relationship between the first time and the second time, or based on the storage location, wherein the pre-filling path includes one of the following: pre-filling the first prompt word, loading the first cached data.

[0026] Through the above steps, it is detected whether there is first cached data in multiple cached data that is the same as the first prompt word input by the target object; if it is determined that there is first cached data, the first time for pre-filling the first prompt word and the second time for loading the first cached data are predicted respectively, and the storage location of the first cached data is determined; according to the size relationship between the first time and the second time, or according to the storage location, the pre-filling path of the first prompt word is determined, wherein the pre-filling path includes one of the following: pre-filling the first prompt word and loading the first cached data. Therefore, it can solve the problem in related technologies where fixed logic is used to pre-fill the prompt words input by the user, resulting in low pre-filling efficiency of the prompt words, and break through the limitations of the traditional fixed reuse caching strategy, effectively improving the pre-filling efficiency.

[0027] In an exemplary embodiment, before determining the pre-filled path of the first prompt word based on the relationship between the first time and the second time, or based on the storage location, the method further includes: determining whether the first prompt word conforms to a preset rule based on the storage location and a first length of the first prompt word, wherein the preset rule is used to indicate whether to determine the pre-filled path based on the storage location; if it is determined that the first prompt word conforms to the preset rule, determining the pre-filled path based on the storage location; if it is determined that the first prompt word does not conform to the preset rule, determining the pre-filled path based on the relationship between the first time and the second time.

[0028] In an exemplary embodiment, before determining the pre-filling path based on the relationship between the first time and the second time, the method further includes: determining the time difference between the first time and the second time; if the time difference is greater than a first preset value, determining the pre-filling path based on the relationship between the first time and the second time; if the time difference is less than a second preset value, determining that the pre-filling path is for loading the first cached data, wherein the first preset value is greater than the second preset value.

[0029] In an exemplary embodiment, determining the pre-filling path based on the relationship between the first time and the second time includes: if the first time is less than the second time, determining the pre-filling path to pre-fill the first prompt word; if the first time is greater than the second time, determining the pre-filling path to load the first cached data.

[0030] In an exemplary embodiment, determining the pre-filling path based on the storage location includes: when the storage location is a disk and the first length of the first prompt word is less than a third preset value, determining the pre-filling path to pre-fill the first prompt word; and when the storage location is video memory and the first length is greater than or equal to the third preset value, determining the pre-filling path to load the first cached data.

[0031] When the system detects that the target prompt word has hit the cache, the pre-filling decision module first extracts the storage location of the cached data (e.g., video memory, CPU memory, disk) and the first length of the first prompt word, and inputs them into the preset rule judge. The preset rule judge has built-in multiple sets of lightweight logical judgment conditions summarized from experience and test data. For example, if the first cached data is located on the disk and the first length is less than a third preset value, it is determined to be a short text slow loading scenario, and the first prompt word is directly re-pre-filled; if the first cached data is located in video memory and the first length is greater than or equal to the third preset value, it is determined to be a long text high-speed caching scenario, and the first cached data is directly loaded. When the input parameter (first prompt word) meets any of the above preset rules, the pre-filling path is immediately output according to the rules, achieving millisecond-level deterministic decision-making. If the input parameters do not meet any preset rules (such as the cache being located in CPU memory, or the prompt word length being in the middle range of 1024~3072), it is determined that the rules are not met. The recalculation time predictor (first time prediction model) and the cache loading time predictor (second time prediction model) are then called to calculate the first and second times respectively. The pre-filling path is determined based on the relationship between the first and second times. The specific implementation process is as follows: When the pre-filling decision module receives the first and second times, it calculates the time difference between them. If the time difference is greater than a first preset value (e.g., 50ms), it further judges the magnitude of the first and second times. If the first time is less than the second time, it means that re-prefilling the first prompt word is much faster than loading the first cached data. In this case, the pre-filling path is determined to be pre-filling the first prompt word. If the first time is greater than the second time, it means that directly loading the first cached data is much faster than re-prefilling the first prompt word. In this case, the pre-filling path is determined to be loading the first cached data. If the time difference is less than a second preset value (e.g., 10ms), it indicates that the difference between the first and second times is close. In this case, the pre-filling path is determined to be loading the first cached data.

[0032] Optionally, Example 1: The prompt word length in the user's inference request is 2048, and the first cached data is located in CPU memory. The first time interval is 180ms, and the second time interval is 90ms, with a time difference of 90ms. This time difference is greater than the first preset value of 50ms, and the first time interval is greater than the second time interval. Therefore, the pre-filling path is determined to be loading the first cached data. Example 2: The prompt word length in the user's inference request is 4096, which is greater than the third preset value of 2048. The first cached data is located in video memory, satisfying the preset rule. In this case, the pre-filling path is determined to be loading the first cached data.

[0033] This embodiment uses preset rules to determine and predict the relationship between time intervals. While ensuring decision-making certainty and low latency in extreme scenarios, it retains the prediction model's fine adaptability to complex and transitional scenarios, significantly reducing the system's average decision-making overhead and improving path selection accuracy. Furthermore, by clearly defining the two cases of large and small time differences when comparing the relationship between predicted time intervals, it further refines the method for determining pre-filled paths and improves their accuracy.

[0034] In an exemplary embodiment, before predicting the first time for pre-filling the first prompt word and the second time for loading the first cached data, the method further includes: constructing K groups of second prompt words, wherein each group of second prompt words contains multiple prompt words, the K groups of second prompt words correspond to K prompt word lengths, the multiple prompt words in the i-th group of second prompt words have the same length, K and i are both positive integers, i∈[1,K]; fitting the K groups of second prompt words using the least squares method to obtain the quadratic coefficient, linear coefficient, and constant term of the quadratic function in the first time prediction model, wherein the first time prediction model is used to predict the first time using the quadratic function; and constructing the quadratic function based on the quadratic coefficient, the linear coefficient, and the constant term.

[0035] In an exemplary embodiment, predicting the first time for pre-filling the first prompt word includes: obtaining the first length of the first prompt word; inputting the first length into the first time prediction model; and calculating the first time based on the first length using the quadratic function.

[0036] In the offline calibration phase before deploying the first-time prediction model, K sets of second prompt words (e.g., 10 sets) are constructed. In each set, multiple prompt words have the same length, for example, L1=64, L2=128, ..., L10=4096. The length of multiple prompt words in the first set is 64, and the length of multiple prompt words in the second set is 128. Subsequently, in a unified hardware environment, a complete routine pre-filling process is performed on each set of prompt word samples. A high-precision timer is used to measure the time required to pre-fill each prompt word, and the average time for each set of prompt words is recorded as the training label. K sets of training datasets are obtained based on the lengths of the K prompt words and the K training labels. These K sets of training datasets are fitted using weighted least squares to obtain the quadratic coefficients *a*, *b*, and constant term *c* of the quadratic function in the first-time prediction model. A quadratic function is then constructed based on these coefficients. : Where L is the length of the prompt word. When the first prompt word input by the user is received during the online inference stage, it is first converted into a token sequence by the word segmenter, and its first length is obtained. The first length is input into the quadratic function constructed above to perform lightweight algebraic calculations to obtain the first time.

[0037] Alternatively, assume that the parameters of the quadratic function obtained by offline fitting are a = 1.2 × 10⁻⁶. -5 b = 3.5 × 10 -4 c=0.012, when the first length is 2048, the first time... It is important to clarify that, assuming the prompt word length range used by the quadratic function in the offline calibration phase is [64, 4096], if the first length is less than 64 or greater than 4096, it is considered that the first length exceeds the training range of the quadratic function. In this case, extrapolation is used to expand the applicable range of the quadratic function, and the first time is additionally marked as low confidence. This indicates that the predicted value is an extrapolation result based on the assumption that the trend of the prediction model at the first time is continuously effective. Its accuracy has not been verified by actual testing and there is a large risk of error. After the system receives the first time with low confidence, it will actively trigger the data collection priority improvement mechanism. That is, when the long (or short) prompt word request is actually executed, regardless of the path selection result, its actual time is forcibly recorded and added to the training database as a high-priority sample for subsequent online model optimization.

[0038] This embodiment achieves efficient, accurate, and reproducible prediction of pre-filling time for prompt words of arbitrary length through systematic sampling and mathematical fitting. Furthermore, when the user inputs an excessively long (or excessively short) prompt word that exceeds the historical training range, the system extrapolates from existing trends to provide a preliminary prediction, marks it as low confidence, and immediately initiates a data collection task to actively collect real data to fill the knowledge gaps in the model, thereby achieving continuous self-improvement of predictive capabilities.

[0039] In one exemplary embodiment, predicting a second time for loading the first cached data includes: calculating the cache size and data adaptation time of the first cached data based on the first length of the first prompt word, wherein the data adaptation time is used to characterize the time for preprocessing the first cached data after loading it into video memory; calculating the transmission time of the first cached data based on the cache size, wherein the transmission time is used to characterize the time for transmitting the first cached data from the storage location to the video memory; and determining the second time based on the data adaptation time and the transmission time.

[0040] After obtaining the first length of the first prompt word, the cache size M of the first cached data is calculated based on the first length: M = k × L, where k is the cache size of key-value pairs (KV) per unit token, and the cache size is linearly related to the first length. Subsequently, the transmission time is calculated based on the storage location of the first cached data (video memory, CPU memory, disk). : Where m is the transmission time of a unit of cached data from the storage location to the video memory, and n is the transmission protocol latency. The transmission time is linearly related to the cache size M. Simultaneously, the data adaptation time is calculated based on the first length. : Where h is the adaptation time per unit token (e.g., 0.8μs / token), and i is the fixed initialization delay (e.g., 2ms). Data adaptation time can be understood as the preprocessing overhead necessary for compatibility with the current inference engine's computational flow after the first cached data is loaded from its storage location into GPU memory. This includes: KV data integrity verification (hash comparison), data format conversion (e.g., FP16→FP8), tensor dimension alignment, attention mask remapping, and alignment with the current batch context. Furthermore, the data adaptation time is linearly related to the first length. Finally, the transmission time and data adaptation time are added together to obtain the second time. : It needs to be clarified that if the storage location is video memory, then the second time... .

[0041] This embodiment achieves physical-level quantifiable modeling of the entire KV cache (first cache data) loading process by precisely decomposing the cache loading time (second time) into cache size linearly related to the prompt word length, transmission time linearly related to the cache size, and data adaptation time linearly related to the prompt word length. This significantly improves the accuracy of path decision-making and engineering robustness.

[0042] In an exemplary embodiment, after determining the pre-filling path of the first prompt word based on the relationship between the first time and the second time, or based on the storage location, the method further includes: for a plurality of first prompt words, obtaining a first length and a third time for the plurality of first prompt words, wherein the third time is the time for pre-filling the first prompt word according to the pre-filling path; calculating the average and standard deviation of the plurality of first lengths and the plurality of third times respectively to obtain a plurality of sets of data, wherein the plurality of first prompt words, the plurality of first lengths, and the plurality of third times are all one-to-one correspondences, and the plurality of sets of data includes: the first average and the first standard deviation of the plurality of first lengths, and the second average and the second standard deviation of the plurality of third times; preprocessing the plurality of first lengths and the plurality of third times according to the plurality of sets of data to obtain a plurality of second lengths and a plurality of fourth times; updating a first time prediction model and a second time prediction model according to the plurality of second lengths, the plurality of fourth times, and the plurality of third prompt words, wherein the plurality of second lengths, the plurality of fourth times, and the plurality of third prompt words are all one-to-one correspondences, the plurality of first prompt words include the plurality of third prompt words, the first time prediction model is used to predict the first time, and the second time prediction model is used to predict the second time.

[0043] In one exemplary embodiment, preprocessing multiple first lengths and multiple third times based on the multiple sets of data to obtain multiple second lengths and multiple fourth times includes: determining a length range of multiple first prompt words based on a first average value and a first standard deviation, and determining a time range for pre-filling multiple first prompt words according to the pre-filling path based on a second average value and a second standard deviation; determining multiple second lengths belonging to the length range among the multiple first lengths, and determining multiple fourth times belonging to the time range among the multiple third times.

[0044] After performing several pre-filling operations during the inference process, the dynamic measurement module continuously collects the actual time consumption data for each request, recording the first length and third time (actual pre-filling time) corresponding to each first prompt word, and calculating the first average and first standard values ​​of multiple first lengths, as well as the second average and second standard values ​​of multiple third times. Based on the first average and first standard values, a reasonable length range is defined for multiple first prompt words, and a reasonable pre-filling time range is defined based on the second average and second standard values. These length and time ranges are set according to the 3σ principle in statistics, effectively eliminating extreme values ​​caused by non-model-related anomalies such as system jitter, garbage collection (GC) pauses, GPU preemption, and network latency. Next, the original samples (multiple first lengths and multiple third times) are traversed, retaining only data points falling within the above two ranges to obtain cleaned multiple second lengths and multiple fourth times. The first time prediction model and the second time prediction model are then updated based on the multiple second lengths, multiple fourth times, and multiple third prompt words.

[0045] Optionally, assuming the average length of the prompt words is 1024 and the standard deviation is 256, the length range is [256, 1792]. The average third time is 95ms and the standard deviation is 30ms, so the time range is [5, 185]. If the first length of a prompt word is 5000 or the third time is 200ms, it is discarded. Only prompt words whose first length and third time both fall within the length range are retained, resulting in multiple third prompt words. Multiple second lengths, multiple fourth times, and multiple third prompt words are batch-fed into the online optimization module of the prediction model. Incremental least squares or online gradient descent is used to fine-tune the model parameters, ensuring that the model continuously converges to the true distribution while adapting to hardware fluctuations, load changes, and caching strategy evolution. This embodiment combines the 3σ statistical principle to intelligently identify and remove abnormal samples caused by non-model interference factors such as GC pauses, GPU resource preemption, system jitter, and network latency, effectively filtering out extreme noise values ​​and significantly improving the robustness and long-term convergence accuracy of the first-time prediction model and the second-time prediction model.

[0046] In an exemplary embodiment, after determining the pre-filling path of the first prompt word based on the relationship between the first time and the second time, or based on the storage location, the method further includes: if no prompt word is received from the target object within a preset time, generating N sets of fourth prompt words, wherein each set of fourth prompt words contains M fifth prompt words, the N sets of fourth prompt words correspond to N prompt word lengths, the M fifth prompt words in the j-th set of fourth prompt words have the same length, N, M, and j are all positive integers, j∈[1,N]; randomly setting a second storage location for each fifth prompt word in the N sets of fourth prompt words, and simulating pre-filling of the N sets of fourth prompt words to obtain N M fifth times, wherein the fifth time is the time when the fifth prompt word is pre-filled; according to the N sets of fourth prompt words, the N M fifth times and N M second storage locations update the first time prediction model and the second time prediction model, wherein the first time prediction model is used to predict the first time, and the second time prediction model is used to predict the second time.

[0047] First, N sets of fourth prompt words are generated, where N is the preset number of sample groups (e.g., N=20). Each set corresponds to a unique prompt word length, ensuring coverage of typical inference scenarios ranging from short texts (e.g., 64 tokens) to very long contexts (e.g., 8192 tokens). Each set of fourth prompt words contains M fifth prompt words (e.g., M=10), meaning that the M prompt words in each set have the same length, but their content achieves semantic diversity through randomly generated semantic sequences, avoiding model overfitting to specific text patterns. Then, a second storage location (GPU memory, CPU memory, or disk) is randomly assigned to each fifth prompt word in each set, simulating the distribution of KV caches being unloaded to different media due to hot / cold separation, eviction policies, or GPU memory pressure in real inference, ensuring sampling covers the entire cache lifecycle. Next, the system fully simulates the pre-filling process in the inference engine sandbox environment: instead of triggering real model calculations, the actual loading or recalculation (re-pre-filling) time for each fifth prompt word in the specified storage location is precisely recorded through instrumentation functions, generating N×M fifth times (e.g., 20×10=200 samples). Finally, the N×M fifth time points, the corresponding N×M second storage locations, and the N sets of fourth prompt words as input are used as synthetic training samples and injected into the dynamic optimization module of the prediction model. These samples are then mixed with the data collected from real requests, and the parameters of the first and second time prediction models are updated using incremental least squares or online gradient descent.

[0048] This embodiment introduces an active simulation sampling mechanism. If no real user inference requests are received within a preset time period (e.g., 5 minutes), the system automatically starts the simulation generation module to maintain the online learning capability of the prediction model and avoid the engineering risk of misjudgment after the model fails due to a long period of no requests and the service is restored.

[0049] In an exemplary embodiment, after detecting whether there is first cached data in a plurality of cached data that is the same as the first prompt word input by the target object, the method further includes: if it is determined that there is no first cached data, inputting the first prompt word into a pre-filling model for pre-filling to obtain second cached data corresponding to the first prompt word, and adding the second cached data to the plurality of cached data.

[0050] During the processing of user requests by the large model inference engine, when the management and scheduling module calls the KV cache management module to perform a cache hit check on the input first prompt word, if no first cache data matching the first prompt word is found (i.e., a cache miss), the system will immediately trigger a regular pre-filling process: the first prompt word is converted into a token sequence by a token segmenter, and then sequentially input into the large model's embedding layer, multi-head attention layer, and feedforward network computation modules to perform a complete forward propagation operation, generating the second cache data corresponding to the prompt word, which is a set of self-attention intermediate states composed of key and value tensors. This second cache data completely records the historical context representation of each token in the prompt word sequence in each layer of attention heads, and can be directly used for incremental inference in the subsequent token generation stage, avoiding redundant calculations. To ensure efficient reuse of identical or similar prompts, after pre-filling, the system writes the generated second cache data, using the prompt's unique identifier (such as a hash value or semantic digest) as the key, along with its length, storage location (default GPU memory), generation timestamp, and other metadata, into the memory index table of the KV cache management module, and allocates corresponding GPU memory space for persistent storage. Simultaneously, the system evaluates the current GPU memory usage based on a preset cache eviction policy. If the total GPU memory cache capacity exceeds a threshold, an unloading mechanism is automatically triggered, migrating the least used or lowest-priority cache blocks to CPU memory or disk to maintain a dynamic balance of system memory resources. Through this process, the system achieves a complete closed-loop processing of zero-cache-hit responses to new requests, ensuring not only the correctness and completeness of inference but also laying a data foundation for potential cache reuse scenarios.

[0051] To better understand the process of determining the pre-filled path, the method for determining the pre-filled path will be further described below with reference to optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0052] Figure 3This is a schematic diagram of a method for determining a pre-filled path according to an embodiment of this application, such as... Figure 3 As shown, it specifically includes the following:

[0053] When a user sends an inference request to the large model inference service, the system first receives the request through the inference service entry module. After completing basic protocol verification, request format parsing, and prompt word extraction, the inference request is submitted to the core management and scheduling module. The management and scheduling module acts as the central coordinating hub for the entire inference process, responsible for coordinating the collaborative execution of various sub-modules. The management and scheduling module first calls the KV cache management module to efficiently match multiple historical KV cache data already cached in the cache index table based on the hash value or semantic fingerprint of the prompt words, determining if there is reusable first cache data. If no match is found, the inference execution module is directly instructed to initiate the standard pre-filling process, performing complete word segmentation, embedding, attention calculation, and feedforward propagation on the input prompt words to generate new second cache data and store it in GPU memory, while simultaneously updating the cache index table. If a cache hit occurs, the management and scheduling module uses parameters such as the prompt word length, the storage location of the first cache data (GPU memory / CPU / disk), and the cache size as input to proactively initiate a path decision request to the pre-filling decision module. The prefilling decision module then calls the recalculation time predictor (first-time prediction model) and the cache loading time predictor (second-time prediction model) in parallel to predict the time required to re-execute prefilling (first time) and the time required to load the existing cache (second time), respectively. After obtaining the two sets of prediction results, the decision module performs intelligent comparison based on the time difference between the first and second times and scenario-based rules (such as "disk cache + short text tends to be recalculated" and "video memory cache + long text tends to be reused"), outputs the prefilling path for loading the first cache data or prefilling the first prompt word, and feeds it back to the management and scheduling module. The management and scheduling module sends the prefilling path to the inference execution module to instruct the inference execution module to execute the corresponding path. That is, if the first cache data is selected to be loaded, the cache loading and data adaptation process is triggered; if the first prompt word is selected to be prefilled, the complete forward computation is started and the old cache is overwritten. Throughout the inference execution process, the prediction model dynamic optimization module is continuously embedded in the execution pipeline, collecting real-time data on the actual time consumption, hardware status (such as GPU utilization and memory bandwidth), and cache location information for each request. After 3σ anomaly filtering, this data is injected in batches into the two predictors. Online least squares or incremental gradient descent is used to continuously update model parameters (such as a, b, c, m, h, etc.), enabling the prediction model to adapt to dynamic environmental changes such as hardware performance fluctuations, cache media migration, and load contention. Simultaneously, the KV cache management module runs in the background, dynamically allocating, unloading, and evicting KV caches based on eviction policies and hot / cold tiering strategies. This ensures efficient utilization of GPU memory resources and provides stable and consistent underlying data support for cache hit checks and loading operations.

[0054] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0055] Embodiments of this application also provide a pre-filled path determination device. Figure 4 A structural block diagram of a pre-filled path determination device provided in an embodiment of this application is shown below. Figure 4 As shown, the device includes:

[0056] Detection module 42 is used to detect whether there is a first cached data that is the same as the first prompt word input by the target object among multiple cached data;

[0057] The prediction module 44 is used to predict, when it is determined that the first cached data exists, the first time for pre-filling the first prompt word and the second time for loading the first cached data, and to determine the storage location of the first cached data.

[0058] The determining module 46 is used to determine the pre-filling path of the first prompt word based on the size relationship between the first time and the second time, or based on the storage location, wherein the pre-filling path includes one of the following: pre-filling the first prompt word, loading the first cached data.

[0059] In this embodiment, it is detected whether there is first cached data among multiple cached data that is the same as the first prompt word input by the target object; if it is determined that there is first cached data, the first time for pre-filling the first prompt word and the second time for loading the first cached data are predicted respectively, and the storage location of the first cached data is determined; the pre-filling path of the first prompt word is determined according to the size relationship between the first time and the second time, or according to the storage location, wherein the pre-filling path includes one of the following: pre-filling the first prompt word and loading the first cached data. Therefore, it can solve the problem in the related technology that the pre-filling of prompt words is inefficient due to the use of fixed logic to pre-fill the prompt words input by the user, and break through the limitations of the traditional fixed reuse caching strategy, effectively improving the pre-filling efficiency.

[0060] In an exemplary embodiment, the determining module 46 is further configured to determine whether the first prompt word conforms to a preset rule based on the storage location and the first length of the first prompt word, wherein the preset rule is used to indicate whether to determine the pre-filled path based on the storage location; if the first prompt word conforms to the preset rule, the pre-filled path is determined based on the storage location; if the first prompt word does not conform to the preset rule, the pre-filled path is determined based on the size relationship between the first time and the second time.

[0061] In an exemplary embodiment, the determining module 46 is further configured to determine the time difference between the first time and the second time; if the time difference is greater than a first preset value, determine the pre-filling path according to the size relationship between the first time and the second time; if the time difference is less than a second preset value, determine that the pre-filling path is for loading the first cached data, wherein the first preset value is greater than the second preset value.

[0062] In an exemplary embodiment, the determining module 46 is further configured to determine, when the first time is less than the second time, that the pre-filling path is for pre-filling the first prompt word; and when the first time is greater than the second time, that the pre-filling path is for loading the first cached data.

[0063] In an exemplary embodiment, the determining module 46 is further configured to determine, when the storage location is a disk and the first length of the first prompt word is less than a third preset value, that the pre-filling path is for pre-filling the first prompt word; and when the storage location is video memory and the first length is greater than or equal to the third preset value, that the pre-filling path is for loading the first cached data.

[0064] In an exemplary embodiment, the prediction module 44 is further configured to construct K groups of second prompt words, wherein each group of second prompt words contains multiple prompt words, the K groups of second prompt words correspond to K prompt word lengths, the multiple prompt words in the i-th group of second prompt words have the same length, K and i are both positive integers, i∈[1,K]; fit the K groups of second prompt words using the least squares method to obtain the quadratic coefficient, linear coefficient, and constant term of the quadratic function in the first time prediction model, wherein the first time prediction model is used to predict the first time using the quadratic function; and construct the quadratic function based on the quadratic coefficient, the linear coefficient, and the constant term.

[0065] In an exemplary embodiment, the prediction module 44 is further configured to obtain the first length of the first prompt word; input the first length into the first time prediction model, and calculate the first time based on the first length using the quadratic function.

[0066] In an exemplary embodiment, the prediction module 44 is further configured to calculate the cache size and data adaptation time of the first cached data based on the first length of the first prompt word, wherein the data adaptation time is used to characterize the time for preprocessing the first cached data after loading it into the video memory; calculate the transmission time of the first cached data based on the cache size, wherein the transmission time is used to characterize the time for transmitting the first cached data from the storage location to the video memory; and determine the second time based on the data adaptation time and the transmission time.

[0067] In an exemplary embodiment, the determining module 46 is further configured to: obtain a first length and a third time for a plurality of first prompt words, wherein the third time is the time for pre-filling the first prompt words according to the pre-filling path; calculate the average and standard deviation of the plurality of first lengths and the plurality of third times respectively to obtain a plurality of sets of data, wherein the plurality of first prompt words, the plurality of first lengths, and the plurality of third times are all one-to-one correspondences, and the plurality of sets of data includes: the first average and the first standard deviation of the plurality of first lengths, and the second average and the second standard deviation of the plurality of third times; preprocess the plurality of first lengths and the plurality of third times according to the plurality of sets of data to obtain a plurality of second lengths and a plurality of fourth times; update the first time prediction model and the second time prediction model according to the plurality of second lengths, the plurality of fourth times, and the plurality of third prompt words, wherein the plurality of second lengths, the plurality of fourth times, and the plurality of third prompt words are all one-to-one correspondences, the plurality of first prompt words include the plurality of third prompt words, the first time prediction model is used to predict the first time, and the second time prediction model is used to predict the second time.

[0068] In an exemplary embodiment, the determining module 46 is further configured to determine a length range of a plurality of first prompt words based on the first average value and the first standard deviation, and to determine a time range for pre-filling the plurality of first prompt words according to the pre-filling path based on the second average value and the second standard deviation; to determine a plurality of second lengths belonging to the length range among the plurality of first lengths, and to determine a plurality of fourth times belonging to the time range among the plurality of third times.

[0069] In an exemplary embodiment, the determining module 46 is further configured to, if no prompt word is received from the target object within a preset time, generate N sets of fourth prompt words, wherein each set of fourth prompt words contains M fifth prompt words, the N sets of fourth prompt words correspond to N prompt word lengths, the M fifth prompt words in the j-th set of fourth prompt words have the same length, N, M, and j are all positive integers, j∈[1,N]; randomly set a second storage location for each fifth prompt word in the N sets of fourth prompt words, and simulate pre-filling the N sets of fourth prompt words to obtain N M fifth times, wherein the fifth time is the time when the fifth prompt word is pre-filled; according to the N sets of fourth prompt words, the N M fifth times and N M second storage locations update the first time prediction model and the second time prediction model, wherein the first time prediction model is used to predict the first time, and the second time prediction model is used to predict the second time.

[0070] It should be noted that the description of the features in the embodiment corresponding to the pre-filled path determination device can be found in the relevant description of the embodiment corresponding to the pre-filled path determination method, and will not be repeated here.

[0071] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the method for determining a pre-filled path.

[0072] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the method for determining a pre-filled path.

[0073] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0074] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described methods for determining pre-filled paths.

[0075] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described pre-filled path determination method embodiments.

[0076] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] The foregoing has provided a detailed description of a method, apparatus, electronic device, storage medium, and product for determining a pre-filled path. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for determining a pre-filled path, characterized in that, include: Detect whether there exists a first cached data that is the same as the first prompt word input by the target object among multiple cached data; If the existence of the first cached data is confirmed, the first time for pre-filling the first prompt word and the second time for loading the first cached data are predicted respectively, and the storage location of the first cached data is determined. The pre-filling path of the first prompt word is determined based on the relationship between the first time and the second time, or based on the storage location, wherein the pre-filling path includes one of the following: pre-filling the first prompt word, or loading the first cached data.

2. The method for determining the pre-filled path according to claim 1, characterized in that, Before determining the pre-fill path of the first prompt word based on the relationship between the first time and the second time, or based on the storage location, the method further includes: The first prompt word is determined to conform to a preset rule based on the storage location and the first length of the first prompt word, wherein the preset rule is used to indicate whether the pre-filled path is determined based on the storage location; If the first prompt word is determined to conform to the preset rule, the pre-filled path is determined according to the storage location; If it is determined that the first prompt word does not conform to the preset rule, the pre-filled path is determined according to the relationship between the first time and the second time.

3. The method for determining the pre-filled path according to claim 2, characterized in that, Before determining the pre-filled path based on the relationship between the first time and the second time, the method further includes: Determine the time difference between the first time and the second time; If the time difference is greater than a first preset value, the pre-filled path is determined according to the relationship between the first time and the second time. If the time difference is less than a second preset value, the pre-filled path is determined to load the first cached data, wherein the first preset value is greater than the second preset value.

4. The method for determining the pre-filled path according to claim 3, characterized in that, Determining the pre-filled path based on the relationship between the first time and the second time includes: If the first time is less than the second time, the pre-filled path is determined to be the pre-filling of the first prompt word; If the first time is greater than the second time, the pre-filled path is determined to be for loading the first cached data.

5. The method for determining the pre-filled path according to claim 2, characterized in that, Determining the pre-filled path based on the storage location includes: If the storage location is a disk and the first length of the first prompt word is less than a third preset value, then the pre-filling path is determined to be for pre-filling the first prompt word. If the storage location is video memory and the first length is greater than or equal to the third preset value, the pre-filled path is determined to load the first cached data.

6. The method for determining the pre-filled path according to claim 1, characterized in that, Before predicting the first time for pre-filling the first prompt word and the second time for loading the first cached data, the method further includes: Construct K groups of second prompt words, where each group of second prompt words contains multiple prompt words, and the K groups of second prompt words correspond to K prompt word lengths. The multiple prompt words in the i-th group of second prompt words have the same length, and K and i are both positive integers, i∈[1,K]. By fitting the K groups of second prompt words using the least squares method, the quadratic coefficient, linear coefficient, and constant term of the quadratic function in the first time prediction model are obtained, wherein the first time prediction model is used to predict the first time using the quadratic function; The quadratic function is constructed based on the coefficients of the quadratic term, the coefficients of the linear term, and the constant term.

7. The method for determining the pre-filled path according to claim 6, characterized in that, Predicting the first time to pre-fill the first prompt word includes: Obtain the first length of the first prompt word; The first length is input into the first time prediction model, and the first time is calculated based on the first length using the quadratic function.

8. The method for determining the pre-filled path according to claim 1, characterized in that, Predicting the second time for loading the first cached data includes: The cache size and data adaptation time of the first cached data are calculated based on the first length of the first prompt word, wherein the data adaptation time is used to characterize the time for preprocessing the first cached data after loading it into the video memory; The transmission time of the first cached data is calculated based on the cache size, wherein the transmission time is used to characterize the time to transfer the first cached data from the storage location to the video memory; The second time is determined based on the data adaptation time and the transmission time.

9. The method for determining the pre-filled path according to claim 1, characterized in that, After determining the pre-fill path of the first prompt word based on the relationship between the first time and the second time, or based on the storage location, the method further includes: For multiple first prompt words, obtain the first length and third time of the multiple first prompt words, wherein the third time is the time for pre-filling the first prompt words according to the pre-filled path; The average value and standard deviation of multiple first lengths and multiple third times are calculated respectively to obtain multiple sets of data. Among them, multiple first prompt words, multiple first lengths, and multiple third times are all one-to-one correspondences. The multiple sets of data include: the first average value and the first standard deviation of multiple first lengths, and the second average value and the second standard deviation of multiple third times. Based on the multiple sets of data, multiple first lengths and multiple third times are preprocessed to obtain multiple second lengths and multiple fourth times; The first time prediction model and the second time prediction model are updated based on the plurality of second lengths, the plurality of fourth times, and the plurality of third prompt words, wherein the plurality of second lengths, the plurality of fourth times, and the plurality of third prompt words are all one-to-one correspondences, the plurality of first prompt words include the plurality of third prompt words, the first time prediction model is used to predict the first time, and the second time prediction model is used to predict the second time.

10. The method for determining the pre-filled path according to claim 9, characterized in that, Based on the multiple sets of data, multiple first lengths and multiple third times are preprocessed to obtain multiple second lengths and multiple fourth times, including: The length range of a plurality of first prompt words is determined based on the first average value and the first standard deviation, and the time range for pre-filling the plurality of first prompt words according to the pre-filling path is determined based on the second average value and the second standard deviation. The plurality of second lengths belonging to the length range are determined among the plurality of first lengths, and the plurality of fourth times belonging to the time range are determined among the plurality of third times.

11. The method for determining the pre-filled path according to claim 1, characterized in that, After determining the pre-fill path of the first prompt word based on the relationship between the first time and the second time, or based on the storage location, the method further includes: If no prompt word is received from the target object within a preset time, N sets of fourth prompt words are generated. Each set of fourth prompt words contains M fifth prompt words. The N sets of fourth prompt words correspond to N prompt word lengths. The M fifth prompt words in the j-th set of fourth prompt words have the same length. N, M, and j are all positive integers, and j∈[1,N]. A second storage location is randomly assigned to each fifth prompt word in the N groups of fourth prompt words, and pre-filling of the N groups of fourth prompt words is simulated to obtain N M fifth times, wherein the fifth time is the time when the fifth prompt word is pre-filled; According to the Nth group of fourth prompt words, the N M fifth times and N M second storage locations update the first time prediction model and the second time prediction model, wherein the first time prediction model is used to predict the first time, and the second time prediction model is used to predict the second time.

12. A device for determining a pre-filled path, characterized in that, include: The detection module is used to detect whether there is a first cached data that is the same as the first prompt word input by the target object among multiple cached data; The prediction module is used to predict, when it is determined that the first cached data exists, the first time for pre-filling the first prompt word and the second time for loading the first cached data, and to determine the storage location of the first cached data. The determining module is used to determine the pre-filling path of the first prompt word based on the size relationship between the first time and the second time, or based on the storage location, wherein the pre-filling path includes one of the following: pre-filling the first prompt word, loading the first cached data.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for determining a pre-filled path as described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method for determining the pre-filled path as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for determining the pre-filled path as described in any one of claims 1 to 11.

Images