A data prefetching method and device for a large model inference service, an apparatus, and a medium

Abstract

The application discloses a data prefetching method and device for large model inference services, equipment and a medium. The method comprises the following steps: dividing an input token sequence into multiple token windows, determining a first embedding row in the first token window, and simultaneously performing streaming direct memory access transfer and batch bypass conversion buffer preheating of the first embedding row; starting to perform pre-padding stage calculation of the first token window, and asynchronously and in parallel performing data prefetching operation of a second token window; determining multiple second embedding rows according to a candidate token tree, performing streaming direct memory access transfer and speculative bypass conversion buffer pre-padding according to the second embedding rows, and asynchronously and in parallel performing main model inference calculation; if it is determined that the inference result of the current decoding step hits any candidate path of the candidate token tree, updating the candidate token tree according to the hit candidate path. The above technical solution can improve the throughput efficiency of the pre-padding stage, reduce the remote memory access delay in the decoding stage, and eliminate long tail delay jitter.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence acceleration chips and large model inference technology, and in particular to a data prefetching method, apparatus, device and medium for large model inference services. Background Technology

[0002] With the development of AI (Artificial Intelligence) intelligent agents, thought chains, multimodal technologies, the number of words consumed by large model reasoning has increased significantly. In order to reduce the reasoning cost of large models, a dual-axis architecture of computational sparsity and memory sparsity can be used to optimize large model reasoning.

[0003] Memory sparsity is achieved by introducing a sparse N-tuple memory module. This module stores a large-scale embedding table as static knowledge and accesses it on demand during inference using deterministic hashing to achieve sparse retrieval. However, sparse embedding tables typically contain tens to hundreds of gigabytes of parameters. In existing large-model inference systems that incorporate sparse N-tuple memory modules, these tables cannot reside permanently in the device memory of the AI ​​acceleration chip and usually need to be migrated to host memory or remote pooled memory.

[0004] In existing technologies, embedding tables are accessed on demand during large model inference. During the pre-filling stage of large model inference, long sequence inputs require batch retrieval of a large number of embedding rows. The large number of embedding rows transported from host memory to device memory can easily lead to repeated transmission and bandwidth waste, forming a throughput bottleneck. During the decoding stage of large model inference, each decoding step of word generation requires real-time acquisition of embedding rows, and the latency of remote memory access will be directly superimposed on the word generation latency. At the same time, existing data prefetching schemes still require a complete page table traversal after the prefetched data arrives before it can be accessed by the computation core. However, the row access of the sparse N-tuple memory module is a highly discrete random pattern, which can easily lead to missing bypass conversion buffers and page table traversal, causing long-tail latency jitter. Summary of the Invention

[0005] This invention provides a data prefetching method, apparatus, device, and medium for large model inference services, which can improve throughput efficiency in the pre-filling stage, reduce remote memory access latency in the decoding stage, and effectively reduce bypass conversion buffer misses and page table traversals, eliminating long-tail latency jitter.

[0006] According to one aspect of the present invention, a data prefetching method for a large model inference service is provided, comprising: When the inference request enters the pre-filling stage, the input lexical sequence of the inference request is divided into multiple lexical windows, multiple first embedding lines that do not hit the device memory in the first lexical window are determined, and the streaming direct memory access transport of the first embedding lines and the batch bypass conversion buffer preheating of the target computing core are performed simultaneously. When the data prefetching operation of the first word window is completed, the pre-filling stage calculation of the first word window is started, and the data prefetching operation of the second word window is executed asynchronously and in parallel until the pre-filling stage calculation of all word windows is completed, and then the decoding stage is entered. Based on the candidate lexicon, multiple second embedding lines that did not hit the device memory are identified. Based on the second embedding lines, streaming direct memory access is performed and speculative bypass conversion buffer prefilling is performed. The main model inference calculation is then executed asynchronously and in parallel. If it is determined that the inference result of the current decoding step hits any candidate path of the candidate lexicon, the candidate lexicon is updated according to the hit candidate path for use in the next decoding step, until the inference request is completed.

[0007] According to another aspect of the present invention, a data prefetching apparatus for a large model inference service is provided, comprising: The prefetch module in the prefilling stage is used to divide the input lexical sequence of the inference request into multiple lexical windows when the inference request enters the prefilling stage, determine multiple first embedding lines that do not hit the device memory in the first lexical window, and simultaneously perform streaming direct memory access transport of the first embedding lines and batch bypass conversion buffer warm-up of the target computing core. The parallel computing module is used to start the pre-filling stage calculation of the first word window when the data prefetching operation of the first word window is completed, and to asynchronously and in parallel execute the data prefetching operation of the second word window until the pre-filling stage calculation of all word windows is completed, and then enter the decoding stage. The decoding stage prefetch module is used to determine multiple second embedding lines that have not hit the device memory based on the candidate lexicon tree, perform streaming direct memory access and speculative bypass conversion buffer prefilling based on the second embedding lines, and perform main model inference calculation asynchronously and in parallel. The candidate lexicon tree update module is used to update the candidate lexicon tree according to the candidate path if the inference result of the current decoding step hits any candidate path of the candidate lexicon tree, so that it can be used in the next decoding step until the inference request is completed.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data prefetching method for the large model inference service according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data prefetching method for the large model inference service described in any embodiment of the present invention.

[0010] The technical solution of this invention involves dividing the input lexical sequence of the inference request into multiple lexical windows when the inference request enters the pre-filling stage. Multiple first embedding rows that do not hit the device memory in the first lexical window are identified, and simultaneously, streaming direct memory access (DMMA) transport and batch bypass conversion buffer warm-up of the target computing core are performed on the first embedding rows. When the data prefetching operation of the first lexical window is completed, the pre-filling stage calculation of the first lexical window is initiated, and the data prefetching operation of the second lexical window is performed asynchronously and in parallel until the pre-filling stage calculation of all lexical windows is completed, then entering the decoding stage. Based on the candidate lexical tree, multiple second embedding rows that do not hit the device memory are identified. Streaming direct memory access transport and speculative bypass conversion buffer pre-filling are performed on the second embedding rows, and the main model inference calculation is performed asynchronously and in parallel. If the inference result of the current decoding step hits the candidate lexical tree... For any candidate path, the candidate lexicon tree is updated based on the hit candidate path for use in the next decoding step, until the inference request is completed. This method can significantly improve the throughput efficiency of the pre-filling stage. Through window-driven deterministic batch prefetching and global sorting deduplication, duplicate data transmission can be eliminated, and discrete random memory access can be transformed into continuous streaming direct memory access. This maximizes the bandwidth utilization from host memory to device memory and effectively solves the bandwidth bottleneck and throughput limitation problem under long sequence input. By using the candidate lexicon tree generated by speculative decoding as the basis for memory access prediction, the embedded row prefetching is completed in advance, so that data transportation and backbone model calculation are completely overlapped. The computation masks the communication latency, fundamentally reducing the cumulative latency of remote memory access generated per lexicon. By completing the bypass conversion buffer prefilling at the same time as data transportation, the bypass conversion buffer missing and page table traversal can be significantly reduced, and long-tail latency jitter can be eliminated.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a data prefetching method for a large model inference service provided according to Embodiment 1 of the present invention; Figure 2 This is a hierarchical architecture diagram of a large model inference system provided according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a candidate word tree provided according to an embodiment of the present invention; Figure 4 This is a flowchart of another data prefetching method for large model inference services provided according to Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of the structure of a data prefetching device for a large model inference service provided according to Embodiment 3 of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device that implements the data prefetching method for the large model inference service according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a data prefetching method for a large model inference service provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where embedded table data prefetching is performed separately in the pre-filling stage and the decoding stage of the large model inference request. This method can be executed by a data prefetching device for the large model inference service. This data prefetching device can be implemented in hardware and / or software and is generally configured in a large model inference system with data processing capabilities. Figure 1 As shown, the method includes: S110. When the inference request enters the pre-filling stage, the input lexical sequence of the inference request is divided into multiple lexical windows. Multiple first embedding lines that do not hit the device memory in the first lexical window are determined. At the same time, the streaming direct memory access of the first embedding lines and the batch bypass conversion buffer preheating of the target computing core are performed.

[0017] Figure 2 This is a hierarchical architecture diagram for an optional large-model inference system. For example... Figure 2 The large model inference system shown can be mounted on a computer or processor. The system can include remote pooled memory, a prefetch scheduler, host-side components, and device-side components. The remote pooled memory is an optional type of remote storage. Figure 2 The illustrations are provided for ease of understanding of the optional implementations offered in the specification; the host side includes host memory; the device side includes device memory, on-chip cache, and computing cores.

[0018] Optionally, "host side" can refer to the Central Processing Unit (CPU); "device side" can refer to the AI ​​acceleration device; "host memory" can refer to the CPU's Dynamic Random Access Memory (DRAM), serving as a temporary storage area on the host side; "device memory" can refer to the off-chip high-speed storage unit of the AI ​​acceleration device, acting as a persistent cache and bandwidth carrying layer on the device side, used to handle streaming data transfer from host memory; the computing core is the processing unit that performs inference calculations on the AI ​​acceleration chip; the on-chip cache is the on-chip temporary working area of ​​the computing core, only carrying the embedded table rows to be calculated. Figure 2 The on-chip cache and compute core are represented separately and grouped together in the form of dashed boxes in order to distinguish the storage function of the on-chip cache from the computing function of the compute core, and to clarify the data transfer flow.

[0019] Optionally, the local memory node that is closest to the target AI acceleration device in the host memory can be selected as the staging area to avoid cross-node interconnection overhead. The staging area in the host memory can use large page memory allocation and lock physical memory to reduce the number of pages that the bypass translation buffer needs to cover.

[0020] Optionally, the prefetch scheduler is a scheduling component responsible for scheduling prefetch tasks and coordinating the overlap between direct memory access queues and backbone computation. It can be configured as a dedicated hardware or software functional module in large model inference systems to determine the embedded rows to be moved and trigger streaming direct memory access moves.

[0021] Understandably, the computation process for large model inference needs to be executed by the computing core. The data required by the computing core during computation needs to be moved to the on-chip cache before it can be used for computation. In the prior art, the computing core needs to first determine whether the currently requested embedded row data hits the on-chip cache. If it does, the data is directly retrieved and used for computation; otherwise, it continues to determine whether it hits the device memory. If it hits the device memory, the embedded row data in the device memory is loaded into the on-chip cache before it is used for computation. If it misses, a step-by-step transfer from host memory to device memory and from device memory to on-chip cache is triggered. In contrast, the data prefetching in this application moves the embedded row data from host memory to device memory in advance before computation and performs a translation lookaside buffer (TLB) warm-up. When the computing core needs to use a certain embedded row, it can immediately hit the translation lookaside buffer and load the required embedded row data into the on-chip cache.

[0022] Generally, due to the small capacity of device memory and on-chip cache, embedded row data is unloaded to host memory after the computing core has finished using it. In the case of remote pooled memory, embedded row data may even be unloaded to remote pooled memory. Therefore, if it is moved step by step when computing is needed, it will generate a lot of memory access latency, thereby reducing the performance of large model inference.

[0023] Optionally, an inference request is an execution request initiated by the large model inference service to complete tasks such as text generation, multimodal understanding, or knowledge question answering. Each inference request needs to go through a pre-filling stage and a decoding stage in sequence to finally obtain the output of the inference request.

[0024] Optionally, the pre-filling stage is a parallel batch computation stage for processing input lexical sequences in large model inference. This invention proposes a pre-fetching strategy for the pre-filling stage that focuses on throughput.

[0025] Optionally, the input word sequence is the prompt text or instruction text entered by the user, which is an ordered set of words formed in sequence after word segmentation and encoding. The word window is a continuous processing unit in the pre-filling stage that divides the input word sequence into fixed lengths. For example, when the input word sequence contains 200 words and the fixed length is set to 50, it can start from the first word of the input word sequence, divide 50 consecutive words into the same word window, and then continue to divide from the 51st word, finally generating 4 word windows.

[0026] Optionally, in the pre-filling stage, processing can begin from the first segmented word window and proceed sequentially, with the currently processed word window serving as the first word window.

[0027] The advantage of this setup is that the sparse N-tuple memory module generates a large number of discrete and random embedding line accesses when processing long sequence inputs. If all the input word sequences are processed at once, it will lead to bandwidth congestion and address translation pressure overload. Therefore, the windowed processing method can break down large-scale memory accesses into controllable batch tasks, so that data handling and model calculation can form pipeline overlap.

[0028] Optionally, the first embedded line may refer to the embedded line in the embedded line corresponding to the first lexical window that did not hit the device memory line cache and needs to be moved from the host memory via streaming direct memory access (DMA).

[0029] Optionally, based on the sparse N-tuple memory module, the first word window can be divided into multiple first candidate word sequences by sliding N-tuple expansion. Then, the corresponding embedding line number is calculated for each first candidate word sequence. After global sorting and deduplication of the embedding line numbers, the embedding lines that do not hit the device memory in the remaining embedding lines are taken as the first embedding lines.

[0030] Optionally, streaming direct memory access (DMA) transport refers to a highly efficient data transport method that organizes a large number of embedded rows to be transmitted into a continuous request queue, submits DMA transport in batches according to a fixed block size, and writes data to a continuous buffer while transmitting. By transporting embedded row data through streaming DMA, the overhead of starting and stopping DMA can be effectively reduced, the link utilization can be improved, and streaming DMA does not occupy the computing power of the target computing core. It supports high-speed transmission of batch data, which can effectively reduce data copy overhead and improve throughput.

[0031] Optionally, after determining the first embedded line, the first embedded line can be added to the streaming direct memory access queue. If this is the first time the streaming direct memory access queue has been generated, the streaming direct memory access transfer between host memory and device memory can be triggered by the prefetch scheduler. If the first embedded line is added to an existing streaming direct memory access queue, the streaming direct memory access queue can be sorted and deduplicated again after the first embedded line is added, and the streaming direct memory access transfer can continue to be performed according to the sorted and deduplicated streaming direct memory access queue to reduce duplicate transfers.

[0032] Optionally, the target computing core can refer to the parallel processing unit in the AI ​​acceleration chip responsible for performing attention calculations and feedforward network calculations.

[0033] Optionally, batch bypass translation buffer warm-up refers to pre-filling the page table entries (PTEs) corresponding to each embedded row currently being moved to the device memory into the on-chip bypass translation buffer of the computing core during the data transfer phase. This avoids latency jitter and MMU (Memory Management Unit) congestion caused by page table traversal triggered by missing bypass translation buffers during the computing phase.

[0034] Understandably, in existing technologies, when a computing core needs to retrieve a specific embedded row of data, it must first obtain the virtual address corresponding to that embedded row through hash calculation and then initiate a memory access request using that virtual address. Since existing prefetching schemes only handle data movement and do not process address translation, the memory access request directly leads to a miss in the bypass translation buffer and triggers the memory management unit to perform a complete page table traversal. The memory management unit queries multiple levels of page tables to locate the page table entry corresponding to the virtual address. Only after obtaining the physical address based on the page table entry can the computing core read the prefetched embedded row data from the device memory. Therefore, even if the embedded row data has been prefetched into the cache, the computing core still needs to wait for the complete page table traversal to finish before it can use the data, causing significant pipeline stalls and latency jitter.

[0035] Optionally, the present invention pre-fills the corresponding page table entries into the on-chip bypass translation buffer while data is being transferred, which can avoid the latency jitter caused by triggering a large number of page table traversals during the calculation phase, making subsequent calculation access more predictable.

[0036] S120. When it is determined that the data prefetching operation of the first word window is completed, the pre-filling stage calculation of the first word window is started, and the data prefetching operation of the second word window is executed asynchronously and in parallel until the pre-filling stage calculation of all word windows is completed, and then the decoding stage is entered.

[0037] Optionally, hardware logic circuitry can be used to monitor the completion status of writing the data of the first embedded row to the device memory and writing the page table entry to the on-chip bypass conversion buffer. Only after it is determined that the data of the first embedded row has been completely written to the device memory and the corresponding page table entry has been filled can the data prefetching operation of the first lexical window be determined to be complete and a prefetching completion signal be sent to the computing core. This prevents the computing core from accessing the bypass conversion buffer in advance, resulting in mismatch risks such as missing the bypass conversion buffer or data not arriving.

[0038] Optionally, the pre-filling stage computation may include attention computation and feedforward network computation. Since the computation window is relatively long, it can be used as a communication masking window to asynchronously perform data transfer from host memory to device memory and pre-filling stage computation, thereby maximizing bandwidth utilization and improving end-to-end throughput without changing the computational semantics of the large model.

[0039] Optionally, the second word window can refer to the next word window after the first word window. For example, if the first word window is word 1 to word 50 in the input word sequence, the second word window can be word 51 to word 100 in the input word sequence. This is just an example.

[0040] In an optional example, assuming the first word window is window K and the second word window is window K+1, after completing the data prefetch operation of window K, the data prefetch of window K+1 and the prefilling stage calculation of window K are executed asynchronously. At this time, the computation kernel only accesses the contiguous buffer in the device memory to avoid the loss of bypass conversion buffer caused by random memory access.

[0041] Optionally, the decoding stage is the stage in large model inference where word-by-word autoregression generates output.

[0042] Optionally, the data prefetching in the prefilling stage is window-batch prefetching. When the inference request completes the computation in the prefilling stage and enters the decoding stage, the prefetch scheduler can switch the prefetching strategy of the inference request from window-batch prefetching to speculative prefetching.

[0043] S130. Based on the candidate lexicon tree, determine multiple second embedding lines that did not hit the device memory, perform streaming direct memory access and speculative bypass conversion buffer prefilling based on the second embedding lines, and asynchronously and in parallel execute the main model inference calculation.

[0044] Optionally, the decoding stage of this invention adopts speculative prefetching that emphasizes latency. Based on the speculative decoding, a lightweight draft model can generate a draft sequence of several future lexical units in advance. The main model verifies and accepts the longest correct prefix in batches and backs up and retry when it fails. This invention extends speculative decoding from a purely computational acceleration method to a look-ahead prediction engine for sparse memory access. Since the candidate lexical tree already contains access information for multiple future steps, looking-ahead prefetching using the information of the candidate lexical tree can achieve computationally masked communication.

[0045] Optionally, the candidate lexical tree is a multi-level candidate lexical structure with probability scores pre-generated by a lightweight draft model in the speculative decoding mechanism. It takes the currently generated lexical as the root node and extends downwards to multiple sets of possible future output lexical paths. The candidate lexical tree is used to infer the path of the current decoding step, and after the current decoding step is completed, the candidate lexical tree is updated according to the update result of the current decoding step for use in the next decoding step.

[0046] Figure 3 This is a schematic diagram of an optional candidate lexical tree. For example... Figure 3As shown, the currently generated context is: "I am", and the lightweight draft model uses this as the root node to generate a candidate token tree with a depth of 2 and a width of 2. Starting from the root node, two candidate tokens are generated in the first layer, which are "a" and "an" respectively, where the path probability corresponding to "a" is 0.85, and the path probability corresponding to "an" is 0.15; taking the token "a" as the node for further expansion, two candidate tokens are generated in the second layer, which are "person" and "individual" respectively, the path probability corresponding to "person" is 0.90, and the path probability corresponding to "individual" is 0.10; taking the token "an" as the node for further expansion, two candidate tokens are generated in the second layer, which are "apple" and "American" respectively, the path probability corresponding to "apple" is 0.70, and the path probability corresponding to "American" is 0.30, thus forming four complete candidate paths.

[0047] Optionally, the second embedding row may refer to a target embedding row that belongs to the first priority and is not cached in the device memory during the decoding phase.

[0048] Optionally, speculative translation lookaside buffer pre-filling may refer to pre-storing the page table entry corresponding to the currently moved second embedding row into a dedicated prefetch translation buffer, so that when access is required for calculation, it can be promoted to the on-chip translation lookaside buffer with zero latency, completely eliminating address translation latency.

[0049] Optionally, main model inference calculation may refer to the core calculation process in the decoding phase of a large model, where the main foundational large model performs forward propagation calculation based on the currently generated context tokens to output the next optimal token.

[0050] Wherein, determining a plurality of second embedding rows that miss the device memory according to the candidate token tree may include: Determine a plurality of candidate paths and the path probability corresponding to each candidate path respectively according to the candidate token tree, perform sliding n-gram expansion on each candidate path respectively, and generate a plurality of second candidate token sequences; Determine a plurality of decoding embedding row numbers according to the second candidate token sequence, perform deduplication and sorting operations on the decoding embedding row numbers, and generate a candidate embedding row list; Calculate the score of each candidate embedding row in the candidate embedding row list according to the path probability, the position of the second candidate token sequence in the candidate token tree, and the calculation performance of main model inference calculation; Determine the priority of each candidate embedding row according to the scores, and determine candidate embedding rows that miss the device memory among the candidate embedding rows belonging to the first priority as the second embedding rows.

[0051] Optionally, a candidate path can refer to a complete sequence of lexical nodes from the root node to any leaf node in the candidate lexical tree. Each path represents a possible future output result. The higher the path probability, the greater the likelihood that the sequence will be adopted by the main model. The path probability can refer to the confidence score calculated by the draft model for each candidate path when generating the candidate lexical tree, which is used to quantify the probability that the lexical sequence will become the true output.

[0052] Continuing from the previous example, in such Figure 3 In the candidate word tree shown, the first path is "I am a", with an overall path probability of 0.85 multiplied by 0.90, which equals 0.765; the second path is "I am a", with an overall path probability of 0.85 multiplied by 0.10, which equals 0.085; the third path is "I am a okay", with an overall path probability of 0.105; and the fourth path is "I am a beauty", with an overall path probability of 0.045.

[0053] Optionally, sliding N-tuple expansion can refer to taking each word in the word window as a basis and truncating a continuous context sequence of fixed length N forward to form an N-tuple structure for retrieval. For parts of the sequence that are less than N length at the beginning, padding words are used to complete the sequence, ensuring the uniformity and completeness of the generated addresses.

[0054] Optionally, the value of N in the sliding N-tuple can be a single specified value or multiple specified values; no limitation is made here.

[0055] Optionally, for each candidate path in the candidate lexical tree, sliding N-tuple expansion can be performed separately, and the second candidate lexical sequence can be obtained based on the sliding N-tuple expansion result.

[0056] Continuing from the previous example, in such Figure 3 In the candidate lexicon tree shown, each candidate path is expanded by sliding N-tuples. Taking the first path, "I am a", as an example, when N is 3, the second candidate lexicon sequence can be obtained including [pad, pad, I], [pad, I, is], [I, is, one] and [is, one, name], where pad is the padding lexicon.

[0057] Optionally, the decoded embedding line can refer to the embedding line determined by the second candidate lexical sequence during the decoding stage; the decoded embedding line and the pre-filled embedding line mentioned later are only used to distinguish the embedding lines determined by different inference stages; the embedding line number can refer to the index number calculated after inputting the candidate lexical sequence into the deterministic hash function, which is used to locate the embedding table line; the decoded embedding line number is the index number obtained after hashing the second candidate lexical sequence.

[0058] Optionally, the candidate embedded row list can be an ordered address list formed by deduplicating and sorting the decoded embedded row numbers, used to eliminate duplicate memory access addresses and normalize discrete access sequences.

[0059] Optionally, the score of a candidate embedding line is a prefetch value score calculated by combining factors such as path probability, the depth of the candidate sequence in the candidate lexicon tree, and the inference time slots and bandwidth resources of the main model. This score is used to quantify the urgency and benefit of accessing each candidate embedding line. A higher score indicates a higher prefetch value.

[0060] Optionally, the position weight can be determined based on the depth of the candidate sequence in the candidate lexicon, and the slot coefficient can be determined based on factors such as the main model inference slots and bandwidth resources. The prefetching value score can be obtained by calculating the product of the comprehensive path probability, position weight, and slot coefficient. Among them, the position weight is inversely proportional to the depth of the candidate sequence in the candidate lexicon, that is, the closer the candidate sequence is to the root node, the higher its corresponding position weight. The slot coefficient is calculated based on the remaining slots and transport bandwidth calculated by the current main model inference, and the more abundant the slots, the higher the slot coefficient.

[0061] Optionally, a scoring threshold can be preset, and candidate embedding rows with scores higher than the scoring threshold can be identified as candidate embedding rows belonging to the first priority. Among the candidate embedding rows belonging to the first priority, candidate embedding rows that do not hit device memory can be identified as second embedding rows.

[0062] S140. If it is determined that the inference result of the current decoding step hits any candidate path in the candidate lexicon tree, then the candidate lexicon tree is updated according to the hit candidate path for use in the next decoding step, until the inference request is completed.

[0063] Optionally, during the decoding process, each word-by-word generation constitutes one decoding step.

[0064] Optionally, after the current decoding step completes inference, if the actual output matches a path in the candidate lexicon, the candidate lexicon can be updated based on that hit path, retaining valid prediction branches and discarding invalid branches, and expanding the candidate lexicon based on the valid branches to ensure that the next decoding step can continue to prefetch based on accurate information until the entire inference request is completed.

[0065] Following the previous example, in Figure 3 If the first path is ultimately matched, then the candidate word tree is expanded with "I am one" as the root node.

[0066] Optionally, the data prefetching method for large model inference services may also include: releasing the prefetching resources corresponding to the inference request after determining that the inference request has been completed.

[0067] Optionally, the automatic switching prefetch strategy proposed in this invention, which is throughput-oriented in the pre-filling stage and latency-oriented in the decoding stage, is compatible with mainstream inference system architectures and does not depend on specific hardware vendor platforms. It can be widely adapted to various AI accelerators and sparse inference scenarios. For example, it can be applied to online inference systems that use sparse memory modules in generative inference services, inference systems deployed on single machines with multiple accelerator processors and cross-node pooled interconnects, or inference systems that need to offload sparse embedding tables to the dynamic random access memory of a general host processor or remote pooled memory.

[0068] The technical solution of this invention involves dividing the input lexical sequence of the inference request into multiple lexical windows when the inference request enters the pre-filling stage. Multiple first embedding rows that do not hit the device memory in the first lexical window are identified, and simultaneously, streaming direct memory access (DMMA) transport and batch bypass conversion buffer warm-up of the target computing core are performed on the first embedding rows. When the data prefetching operation of the first lexical window is completed, the pre-filling stage calculation of the first lexical window is initiated, and the data prefetching operation of the second lexical window is performed asynchronously and in parallel until the pre-filling stage calculation of all lexical windows is completed, then entering the decoding stage. Based on the candidate lexical tree, multiple second embedding rows that do not hit the device memory are identified. Streaming direct memory access transport and speculative bypass conversion buffer pre-filling are performed on the second embedding rows, and the main model inference calculation is performed asynchronously and in parallel. If the inference result of the current decoding step hits the candidate lexical tree... For any candidate path, the candidate lexicon tree is updated based on the hit candidate path for use in the next decoding step, until the inference request is completed. This method can significantly improve the throughput efficiency of the pre-filling stage. Through window-driven deterministic batch prefetching and global sorting deduplication, duplicate data transmission can be eliminated, and discrete random memory access can be transformed into continuous streaming direct memory access. This maximizes the bandwidth utilization from host memory to device memory and effectively solves the bandwidth bottleneck and throughput limitation problem under long sequence input. By using the candidate lexicon tree generated by speculative decoding as the basis for memory access prediction, the embedded row prefetching is completed in advance, so that data transportation and backbone model calculation are completely overlapped. The computation masks the communication latency, fundamentally reducing the cumulative latency of remote memory access generated per lexicon. By completing the bypass conversion buffer prefilling at the same time as data transportation, the bypass conversion buffer missing and page table traversal can be significantly reduced, and long-tail latency jitter can be eliminated.

[0069] Example 2 Figure 4 This is a flowchart illustrating a data prefetching method for a large model inference service according to Embodiment 2 of the present invention. Based on the above embodiments, this embodiment specifically describes a data prefetching method for a large model inference service. Figure 4 As shown, the method includes: S210. When the inference request enters the pre-filling stage, the input lexical sequence of the inference request is divided into multiple lexical windows. Multiple first embedding lines that do not hit the device memory in the first lexical window are determined, and the streaming direct memory access transport of the first embedding lines and the batch bypass conversion buffer preheating of the target computing core are performed simultaneously.

[0070] The determination of multiple first embedding lines that do not hit the device memory in the first lexical window may include: Sliding N-tuple expansion is performed on each word in the first word window to generate multiple first candidate word sequences, and multiple pre-filled embedding line numbers are determined based on the first candidate word sequences. The pre-filled embedded row numbers are deduplicated and sorted to generate a request list, and the embedded rows in the request list that do not hit the device memory are determined as the first embedded rows.

[0071] Optionally, the first candidate lexical sequence is a standard N-tuple sequence generated by sliding N-tuple expansion within the first lexical window, and the corresponding pre-filled embedding line number can be calculated for each first candidate lexical sequence.

[0072] In an optional example, assuming N=3, the first word window is: [I, love, playing, chess], and the padding word is represented by pad. Then, for word I, there is no preceding word, and the expansion result is [pad, pad, I]. For word love, there is only 1 preceding word, and the expansion result is [pad, I, love]. For word playing, there are 2 preceding words, and the expansion result is [I, love, playing]. For word chess, there are 2 preceding words, and the expansion result is [love, playing, chess]. Each of the above expansion results can be used as the first candidate word sequence. Each first candidate word sequence can be hashed to obtain the corresponding pre-filled embedding line number.

[0073] Optionally, the pre-filled embedding line can refer to the embedding line determined by the first candidate lexical sequence during the decoding stage, and the pre-filled embedding line number is the index number obtained after hashing the first candidate lexical sequence.

[0074] Optionally, since the first candidate word sequence within each word window may be the same or similar, the pre-filled embedding lines matched by each first candidate word sequence may be duplicated. Therefore, performing deduplication and sorting operations on the pre-filled embedding line numbers can significantly reduce duplicate transport and transform discrete access into a more continuous transport queue, thereby improving the effective bandwidth of streaming direct memory access transport.

[0075] Optionally, the request list is an ordered sequence of addresses formed after deduplication and sorting of the pre-filled embedded row numbers, representing the set of all non-repeating embedded rows that need to be accessed during the pre-filling stage.

[0076] The simultaneous execution of streaming direct memory access transport of the first embedded line and batch bypass conversion buffer warm-up of the target computing core may include: The first embedded row is added to a streaming direct memory access queue, and according to the streaming direct memory access queue, the embedded row data in the host memory is written in batches to a contiguous buffer in the device memory using streaming transmission; and Read the page table entries corresponding to the embedded row data of the current write batch, and batch fill the page table entries into the on-chip bypass translation buffer of the target computing core.

[0077] Furthermore, if the large model inference system includes remote pooled memory, then when some embedded row data is unloaded to remote pooled memory, the first embedded row is divided into first embedded rows that do not hit host memory and first embedded rows that hit host memory. The first embedded rows that hit host memory can be directly added to the streaming direct memory access queue. The first embedded rows that do not hit host memory can be added to the remote direct memory access queue first, and trigger the remote direct memory access transfer from remote pooled memory to host memory. After the remote direct memory access transfer is completed, these first embedded rows that have just been transferred to host memory are added to the streaming direct memory access queue, realizing the step-by-step transfer from remote pooled memory to device memory.

[0078] It is understandable that if the large model inference system also includes other remote storage, the above method can be used for step-by-step transfer. This solution is widely applicable to large model inference systems with various storage architectures.

[0079] Optionally, using streaming to write embedded row data in batches from host memory to a contiguous buffer in device memory may include: calling a hardware reassembly engine to reassemble discrete sparse embedded rows into a contiguous physical address segment and write it to device memory; wherein, the hardware reassembly engine may be a dedicated hardware structure configured within the AI ​​acceleration chip.

[0080] Alternatively, discrete sparse embedded rows can be reassembled into linear contiguous storage by pre-allocating a contiguous buffer in the device memory through software logic.

[0081] The advantage of this setting is that it can eliminate memory fragmentation in subsequent kernel accesses, greatly improve memory access bandwidth utilization, and support efficient access to ultra-large sparse embedded tables that far exceed the device's memory capacity.

[0082] Optionally, a page table entry is the basic unit in the page table that records the mapping from virtual pages to physical pages and access permissions. When the computing core needs to use a certain embedded row data, the actual physical address of the embedded row data in the device memory can be determined according to the page table entry, so that the embedded row data can be directly loaded from the device memory into the on-chip cache to participate in the computation.

[0083] Optionally, based on the write position of the first embedded row in the device memory, the corresponding physical page set can be scanned and the page table entries can be read in batches; a custom broadcast instruction can be issued through the internal bus of the AI ​​acceleration chip to directly fill the page table entries into the on-chip bypass translation buffer of the target computing core, thereby bypassing the page table traversal process of the standard memory management unit.

[0084] Optionally, a dual-issue command queue can be used to simultaneously generate and temporarily store data transfer instructions and page table fill instructions. While embedding row data into the device memory, page table entries are pushed to the on-chip bypass translation buffer to ensure that the two paths proceed in parallel and eliminate address translation latency.

[0085] S220. When it is determined that the data prefetching operation of the first word window is completed, the pre-filling stage calculation of the first word window is started, and the data prefetching operation of the second word window is executed asynchronously and in parallel until the pre-filling stage calculation of all word windows is completed, and then the decoding stage is entered.

[0086] S230. Based on the candidate lexical tree, determine multiple candidate paths and the path probabilities corresponding to each candidate path, and perform sliding N-tuple expansion on each candidate path to generate multiple second candidate lexical sequences.

[0087] S240. Based on the second candidate word sequence, determine multiple decoding embedding line numbers, and perform deduplication and sorting operations on the decoding embedding line numbers to generate a candidate embedding line list.

[0088] S250. Calculate the score of each candidate embedding row in the candidate embedding row list based on the path probability, the position of the second candidate lexical sequence in the candidate lexical tree, and the computational performance of the main model inference calculation.

[0089] S260. Determine the priority of each candidate embedding line based on the score, and identify the candidate embedding line that did not hit the device memory as the second embedding line from the candidate embedding lines belonging to the first priority.

[0090] Furthermore, if the large model inference system includes remote pooled memory, then when some embedded row data is unloaded to remote pooled memory, the method may further include: determining candidate embedded rows belonging to the second priority based on a pre-specified scoring range and the scores of each candidate embedded row; wherein the second priority is lower than the first priority; and determining a third embedded row that did not hit the host memory and device memory among the candidate embedded rows belonging to the second priority.

[0091] Furthermore, it may also include: adding the third embedded row to the remote direct memory access queue, and writing the embedded row data in the remote pooled memory to the host memory in batches according to the remote direct memory access queue.

[0092] Optionally, based on the scores of each candidate embedding row and the minimum value of the score interval, candidate embedding rows belonging to the third priority can be determined and discarded; wherein, the third priority is lower than the second priority.

[0093] The advantage of this setup is that, in the case of a remote storage structure, by using hierarchical data prefetching, high-priority embedded row data can be moved to device memory, while medium-priority embedded row data can be moved from remote pooled memory to host memory, thereby reducing the data movement overhead when it is determined that a third embedded row needs to be used.

[0094] S270, Perform streaming direct memory access transport and speculative bypass translation buffer prefilling based on the second embedded line.

[0095] The streaming direct memory access transport and speculative bypass translation buffer prefilling based on the second embedded line may include: The second embedded row is added to the streaming direct memory access queue. Based on the streaming direct memory access queue, the embedded row data in the host memory is written in batches to the continuous buffer in the device memory using streaming transmission. At the same time, the page table entries corresponding to the embedded row data of the current writing batch are read and the page table entries are filled in batches into the prefetch translation buffer of the target computing core.

[0096] Optionally, an idle memory management unit can be used to query the shadow page table to obtain the physical address and permission bits, thereby reading the page table entry corresponding to the embedded row data.

[0097] Optionally, a shadow page table can refer to a simplified copy of the page table maintained on the device's memory side, used to quickly generate physical addresses and extract page attributes during data prefetching without initiating a full page table traversal.

[0098] S280. If it is determined that the inference result of the current decoding step hits any candidate path of the candidate token tree, the candidate token tree is updated according to the hit candidate path for use in the next decoding step until the inference request is completed.

[0099] The step that if it is determined that the inference result of the current decoding step hits any candidate path of the candidate token tree, the candidate token tree is updated according to the hit candidate path may include: If it is determined that the inference result of the current decoding step hits any candidate path of the candidate token tree, a target tree node is determined according to the currently hit candidate path; A lightweight draft model is invoked to update the candidate token tree according to the target tree node as well as a preset target width and target depth, and path probabilities of each candidate path are generated in the updated candidate token tree.

[0100] Optionally, the target tree node may refer to the tree node where the last token in the currently hit candidate path is located, and the target tree node may serve as the root node for the next update of the candidate token tree, continuing to Figure 3 For example, if the inference result of the current decoding step hits "one", the tree node where "one" is located is taken as the target tree node; if "one person" is hit at the same time, the tree node where "person" is located is taken as the target tree node.

[0101] Optionally, the candidate token tree may be generated in different manners, for example, only 1 token with the highest probability may be retained at each step to form a single path of the target depth; or a plurality of tokens with the highest probability are retained at each step according to the target width and continue to expand, so as to form the candidate token tree that satisfies the target width and target depth.

[0102] After the main model inference calculation is executed asynchronously and in parallel, the method may further include: If it is determined that the inference result of the current decoding step does not hit any candidate path of the candidate token tree, a target continuous buffer that has been used in the current decoding step is determined in the device memory, and the target continuous buffer is released; A candidate token tree is regenerated according to the inference result, and an embedding line prefetch operation is triggered according to a pre-specified storage priority.

[0103] Optionally, the target continuous buffer that has been used in the current decoding step refers to a continuous buffer allocated for the prefetch operation in the current decoding step for storing speculative prefetch embedding lines, which is the actual storage location of the prefetched embedding line data in the device memory.

[0104] Optionally, if no candidate path is hit, the token obtained from the inference result is directly taken as the root node, and the lightweight draft model is invoked to expand the candidate token tree according to the token obtained from the inference result.

[0105] Optionally, storage priority can refer to the predefined priority among various storage units, with the priority from high to low being: device memory, host memory, and remote pooled memory. Based on storage priority, the fastest data source can be selected in the replenishment operation to minimize latency.

[0106] Optionally, when speculative prefetching fails and the actual path is not prefetched, the synchronous transfer operation of the required embedded row can be urgently obtained according to storage priority, starting from the storage unit with the highest priority, to ensure that the computing core does not experience data waiting pauses.

[0107] The advantages of this setup are: releasing invalid buffers can prevent prefetched data from consuming device memory due to failed inferences; rebuilding the candidate tree allows subsequent prefetches to be re-predicted based on the real path, restoring a high hit rate; and priority-based supplementary fetching ensures that the computing cores obtain the correct data with zero waiting time, without increasing inference latency.

[0108] Furthermore, data prefetching methods for large model inference services may also include: Based on the historical reasoning request, determine the high-frequency word sequence and the shared prefix, and write the embedded line data corresponding to the high-frequency word sequence and the shared prefix into the high-frequency cache partition in the device memory; During the processing of the inference request, if the generated lexical sequence matches the target high-frequency lexical sequence or the target shared prefix, the priority of the embedded row data corresponding to the target high-frequency lexical sequence or the target shared prefix is ​​increased.

[0109] Optionally, historical inference requests can be the set of user requests that the service has already processed during its operation.

[0110] Optionally, high-frequency word sequences can be N-gram word sequences that appear more frequently than a certain proportion in historical inference requests; shared prefixes can be identical input prompt prefixes in multiple different inference requests, such as the same system prompt words and task opening phrases.

[0111] Optionally, a high-frequency cache partition can refer to a dedicated independent storage area in the line cache of the device memory, used to store the embedded line data corresponding to high-frequency word sequences and shared prefixes for a long time, and to implement independent priority and eviction policies.

[0112] Optionally, based on the historical inference request, the high-frequency word sequence and shared prefix are statistically analyzed, and the embedded lines corresponding to the high-frequency word sequence and shared prefix are written into the high-frequency cache partition of the device memory in advance for long-term resident protection. This fully utilizes the Zipf distribution characteristics of natural language N-tuples, allowing a small number of high-frequency lines to occupy the high-speed cache and significantly reducing the number of cross-layer transfers.

[0113] Optionally, the Zipf distribution characteristics of natural language N-tuples are that a small number of high-frequency patterns occupy most of the memory access requests, while a large number of low-frequency patterns constitute the long tail.

[0114] Optionally, the target high-frequency word sequence or target shared prefix can refer to a high-frequency word sequence or shared prefix that matches the currently generated word sequence.

[0115] Optionally, the priority of embedded row data can refer to a scoring metric used to determine the cache residency weight of embedded row data; the higher the priority, the less likely it is to be replaced.

[0116] Optionally, when the large model inference system processes a new inference request, the system will monitor the currently generated or input lexical sequence in real time. Once it is detected that the sequence completely matches any high-frequency lexical sequence and shared prefix obtained from historical statistics, the system will immediately increase the cache priority of all embedded line data corresponding to the target high-frequency lexical sequence or the target shared prefix, and apply protection weights in the eviction policy to make it more protected in the high-frequency cache partition and reduce the probability of it being replaced out of the device memory.

[0117] The advantages of this setup are: it enables the inference service to maintain a very high cache hit rate when faced with a large number of similar tasks, repeated prefixes, and high-frequency sentence patterns, reduces the number of times data is read from host memory or remote pooled memory, avoids the address translation pressure caused by random sparse memory access, continuously and stably improves prefetching benefits at the system level, and maximizes the reuse of historical computation and cache resources.

[0118] The technical solution of this invention, by determining the target tree node based on the hit path when the inference result hits a candidate path, and updating the candidate lexicon tree based on this, avoids recalculating the verified correct branch and discards invalid prediction branches, ensuring that the prefetch address pool always maintains a high degree of matching with the actual access. By promptly locating and releasing the target continuous buffer used in the current decoding step when the inference result does not hit any candidate path, it can quickly reclaim device memory resources, avoid cache pollution and space waste caused by invalid prefetch data, and ensure the normal execution of subsequent prefetch and cache operations. By regenerating the candidate lexicon tree based on the actual inference result, it can enable subsequent prefetching to return to the correct prediction path and quickly restore the high hit rate prefetch process. The entire rollback and replenishment mechanism forms a closed loop, which not only ensures the correctness of inference calculation but also minimizes the performance loss caused by speculation failure, improving the robustness of speculation prefetching. By maintaining a high-frequency cache partition, a small amount of hot data with a high access probability can reside in the high-speed storage layer, significantly improving the cache hit rate and effectively reducing the latency jitter and bandwidth pressure caused by random sparse memory access.

[0119] Example 3 Figure 5 This is a schematic diagram of a data prefetching device for a large model inference service provided in Embodiment 3 of the present invention. Figure 5 As shown, the device includes: a prefetching module 310 for the prefilling stage, a parallel computing module 320, a prefetching module 330 for the decoding stage, and a candidate lexical tree update module 340.

[0120] The prefetch module 310 in the prefilling stage is used to divide the input lexical sequence of the inference request into multiple lexical windows when the inference request enters the prefilling stage, determine multiple first embedding lines that do not hit the device memory in the first lexical window, and simultaneously perform streaming direct memory access transport of the first embedding lines and batch bypass conversion buffer preheating of the target computing core.

[0121] The parallel computing module 320 is used to start the pre-filling stage calculation of the first word window when the data pre-fetching operation of the first word window is completed, and asynchronously and in parallel execute the data pre-fetching operation of the second word window until the pre-filling stage calculation of all word windows is completed, and then enter the decoding stage.

[0122] The decoding stage prefetch module 330 is used to determine multiple second embedding lines that have not hit the device memory based on the candidate lexicon tree, perform streaming direct memory access and speculative bypass conversion buffer prefilling based on the second embedding lines, and asynchronously and in parallel execute the main model inference calculation.

[0123] The candidate lexical tree update module 340 is used to update the candidate lexical tree according to the hit candidate path if it is determined that the inference result of the current decoding step hits any candidate path of the candidate lexical tree, so that it can be used in the next decoding step until the inference request is completed.

[0124] The technical solution of this invention involves dividing the input lexical sequence of the inference request into multiple lexical windows when the inference request enters the pre-filling stage. Multiple first embedding rows that do not hit the device memory in the first lexical window are identified, and simultaneously, streaming direct memory access (DMMA) transport and batch bypass conversion buffer warm-up of the target computing core are performed on the first embedding rows. When the data prefetching operation of the first lexical window is completed, the pre-filling stage calculation of the first lexical window is initiated, and the data prefetching operation of the second lexical window is performed asynchronously and in parallel until the pre-filling stage calculation of all lexical windows is completed, then entering the decoding stage. Based on the candidate lexical tree, multiple second embedding rows that do not hit the device memory are identified. Streaming direct memory access transport and speculative bypass conversion buffer pre-filling are performed on the second embedding rows, and the main model inference calculation is performed asynchronously and in parallel. If the inference result of the current decoding step hits the candidate lexical tree... For any candidate path, the candidate lexicon tree is updated based on the hit candidate path for use in the next decoding step, until the inference request is completed. This method can significantly improve the throughput efficiency of the pre-filling stage. Through window-driven deterministic batch prefetching and global sorting deduplication, duplicate data transmission can be eliminated, and discrete random memory access can be transformed into continuous streaming direct memory access. This maximizes the bandwidth utilization from host memory to device memory and effectively solves the bandwidth bottleneck and throughput limitation problem under long sequence input. By using the candidate lexicon tree generated by speculative decoding as the basis for memory access prediction, the embedded row prefetching is completed in advance, so that data transportation and backbone model calculation are completely overlapped. The computation masks the communication latency, fundamentally reducing the cumulative latency of remote memory access generated per lexicon. By completing the bypass conversion buffer prefilling at the same time as data transportation, the bypass conversion buffer missing and page table traversal can be significantly reduced, and long-tail latency jitter can be eliminated.

[0125] Based on the above embodiments, the prefetch module 310 in the prefilling stage may include a first embedded row determination unit and a direct memory access transport unit. The first embedded row determination unit can be specifically used for: Sliding N-tuple expansion is performed on each word in the first word window to generate multiple first candidate word sequences, and multiple pre-filled embedding line numbers are determined based on the first candidate word sequences. The pre-filled embedded row numbers are deduplicated and sorted to generate a request list, and the embedded rows in the request list that do not hit the device memory are determined as the first embedded rows.

[0126] Based on the above embodiments, the direct memory access transport unit can be specifically used for: The first embedded row is added to a streaming direct memory access queue, and according to the streaming direct memory access queue, the embedded row data in the host memory is written in batches to a contiguous buffer in the device memory using streaming transmission; and Read the page table entries corresponding to the embedded row data of the current write batch, and batch fill the page table entries into the on-chip bypass translation buffer of the target computing core.

[0127] Based on the above embodiments, the decoding stage prefetch module 330 may include a second embedded line determination unit and a second embedded line prefetch unit. The second embedded row determination unit can be specifically used for: Based on the candidate lexical tree, multiple candidate paths and their corresponding path probabilities are determined, and each candidate path is expanded by sliding N-tuples to generate multiple second candidate lexical sequences. Based on the second candidate lexical sequence, multiple decoding embedding line numbers are determined, and the decoding embedding line numbers are deduplicated and sorted to generate a candidate embedding line list; Based on the path probability, the position of the second candidate lexical sequence in the candidate lexical tree, and the computational performance of the main model inference calculation, the score of each candidate embedding row in the candidate embedding row list is calculated. The priority of each candidate embedding line is determined based on the score, and the candidate embedding lines that do not hit the device memory are selected as the second embedding lines from the candidate embedding lines belonging to the first priority.

[0128] Based on the above embodiments, the second embedded row prefetch unit can be specifically used for: The second embedded row is added to the streaming direct memory access queue. Based on the streaming direct memory access queue, the embedded row data in the host memory is written in batches to the continuous buffer in the device memory using streaming transmission. At the same time, the page table entries corresponding to the embedded row data of the current writing batch are read and the page table entries are filled in batches into the prefetch translation buffer of the target computing core.

[0129] Based on the above embodiments, a data replenishment module may also be included, specifically for: If it is determined that the inference result of the current decoding step does not hit any candidate path of the candidate lexical tree, then the target continuous buffer used by the current decoding step is determined in the device memory and the target continuous buffer is released. Based on the reasoning results, a candidate lexicon tree is regenerated, and an embedded row completion operation is triggered according to the pre-specified storage priority.

[0130] Based on the above embodiments, the candidate lexical tree update module 340 can be specifically used for: If it is determined that the inference result of the current decoding step hits any candidate path of the candidate word tree, then the target tree node is determined based on the currently hit candidate path; The lightweight draft model is invoked to update the candidate lexicon tree based on the target tree nodes and the preset target width and target depth, and the path probability of each candidate path is generated in the updated candidate lexicon tree.

[0131] Based on the above embodiments, a high-frequency embedded line retention module may also be included, for: Based on the historical reasoning request, determine the high-frequency word sequence and the shared prefix, and write the embedded line data corresponding to the high-frequency word sequence and the shared prefix into the high-frequency cache partition in the device memory; During the processing of the inference request, if the generated lexical sequence matches the target high-frequency lexical sequence or the target shared prefix, the priority of the embedded row data corresponding to the target high-frequency lexical sequence or the target shared prefix is ​​increased.

[0132] The data prefetching device for large model inference services provided in the embodiments of the present invention can execute the data prefetching method for large model inference services provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0133] Example 4 Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0134] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0135] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0136] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the data prefetching method for large model inference services described in embodiments of the present invention. That is: When the inference request enters the pre-filling stage, the input lexical sequence of the inference request is divided into multiple lexical windows, multiple first embedding lines that do not hit the device memory in the first lexical window are determined, and the streaming direct memory access transport of the first embedding lines and the batch bypass conversion buffer preheating of the target computing core are performed simultaneously. When the data prefetching operation of the first word window is completed, the pre-filling stage calculation of the first word window is started, and the data prefetching operation of the second word window is executed asynchronously and in parallel until the pre-filling stage calculation of all word windows is completed, and then the decoding stage is entered. Based on the candidate lexicon, multiple second embedding lines that did not hit the device memory are identified. Based on the second embedding lines, streaming direct memory access is performed and speculative bypass conversion buffer prefilling is performed. The main model inference calculation is then executed asynchronously and in parallel. If it is determined that the inference result of the current decoding step hits any candidate path of the candidate lexicon, the candidate lexicon is updated according to the hit candidate path for use in the next decoding step, until the inference request is completed.

[0137] In some embodiments, the data prefetching method for the large model inference service may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data prefetching method for the large model inference service described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data prefetching method for the large model inference service by any other suitable means (e.g., by means of firmware).

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0143] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0144] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data prefetching method for a large model inference service, characterized in that, include: When the inference request enters the pre-filling stage, the input lexical sequence of the inference request is divided into multiple lexical windows. Multiple first embedding lines that do not hit the device memory in the first lexical window are identified. At the same time, streaming direct memory access transport of the first embedding lines and batch bypass translation buffer warm-up of the target computing core are performed. The batch bypass translation buffer warm-up is to pre-fill the page table entries corresponding to each embedding line currently transported to the device memory into the on-chip bypass translation buffer of the computing core in batches. When the data prefetching operation of the first word window is completed, the pre-filling stage calculation of the first word window is started, and the data prefetching operation of the second word window is executed asynchronously and in parallel until the pre-filling stage calculation of all word windows is completed, and then the decoding stage is entered. Based on the candidate lexicon, multiple second embedding lines that did not hit the device memory are identified. Streaming direct memory access is performed on the second embedding lines, and speculative bypass translation buffer prefilling is performed. The main model inference calculation is executed asynchronously and in parallel. Among them, speculative bypass translation buffer prefilling is to pre-store the page table entry corresponding to the currently moved second embedding line into a dedicated prefetch translation buffer. If it is determined that the inference result of the current decoding step hits any candidate path of the candidate lexicon, the candidate lexicon is updated according to the hit candidate path for use in the next decoding step, until the inference request is completed.

2. The method according to claim 1, characterized in that, Identify multiple first embedding lines that do not hit device memory in the first lexical window, including: Sliding N-tuple expansion is performed on each word in the first word window to generate multiple first candidate word sequences, and multiple pre-filled embedding line numbers are determined based on the first candidate word sequences. The pre-filled embedded row numbers are deduplicated and sorted to generate a request list, and the embedded rows in the request list that do not hit the device memory are determined as the first embedded rows.

3. The method according to claim 1, characterized in that, Simultaneously, the streaming direct memory access transport of the first embedded line and the batch bypass translation buffer warm-up of the target computing core are performed, including: The first embedded row is added to the streaming direct memory access queue, and the embedded row data in the host memory is written in batches to the continuous buffer in the device memory using streaming transmission according to the streaming direct memory access queue. as well as Read the page table entries corresponding to the embedded row data of the current write batch, and batch fill the page table entries into the on-chip bypass translation buffer of the target computing core.

4. The method according to claim 1, characterized in that, Based on the candidate lexicon, identify multiple second embedding lines that did not hit device memory, including: Based on the candidate lexical tree, multiple candidate paths and their corresponding path probabilities are determined, and each candidate path is expanded by sliding N-tuples to generate multiple second candidate lexical sequences. Based on the second candidate lexical sequence, multiple decoding embedding line numbers are determined, and the decoding embedding line numbers are deduplicated and sorted to generate a candidate embedding line list; Based on the path probability, the position of the second candidate lexical sequence in the candidate lexical tree, and the computational performance of the main model inference calculation, the score of each candidate embedding row in the candidate embedding row list is calculated. The priority of each candidate embedding line is determined based on the score, and the candidate embedding lines that do not hit the device memory are selected as the second embedding lines from the candidate embedding lines belonging to the first priority.

5. The method according to claim 4, characterized in that, Based on the second embedded line, streaming direct memory access transport and speculative bypass translation buffer prefilling are performed, including: The second embedded row is added to the streaming direct memory access queue. Based on the streaming direct memory access queue, the embedded row data in the host memory is written in batches to the continuous buffer in the device memory using streaming transmission. At the same time, the page table entries corresponding to the embedded row data of the current writing batch are read and the page table entries are filled in batches into the prefetch translation buffer of the target computing core.

6. The method according to claim 1, characterized in that, Following the asynchronous parallel execution of the main model inference computation, the following is also included: If it is determined that the inference result of the current decoding step does not hit any candidate path of the candidate lexical tree, then the target continuous buffer used by the current decoding step is determined in the device memory and the target continuous buffer is released. Based on the reasoning results, a candidate lexicon tree is regenerated, and an embedded row completion operation is triggered according to the pre-specified storage priority.

7. The method according to claim 1, characterized in that, If it is determined that the inference result of the current decoding step hits any candidate path of the candidate lexicon, then the candidate lexicon is updated according to the hit candidate path, including: If it is determined that the inference result of the current decoding step hits any candidate path of the candidate word tree, then the target tree node is determined based on the currently hit candidate path; The lightweight draft model is invoked to update the candidate lexicon tree based on the target tree nodes and the preset target width and target depth, and the path probability of each candidate path is generated in the updated candidate lexicon tree.

8. The method according to claim 1, characterized in that, Also includes: Based on the historical reasoning request, determine the high-frequency word sequence and the shared prefix, and write the embedded line data corresponding to the high-frequency word sequence and the shared prefix into the high-frequency cache partition in the device memory; During the processing of the inference request, if the generated lexical sequence matches the target high-frequency lexical sequence or the target shared prefix, the priority of the embedded row data corresponding to the target high-frequency lexical sequence or the target shared prefix is ​​increased.

9. A data prefetching device for a large model inference service, characterized in that, include: The prefetch module in the prefilling stage is used to divide the input lexical sequence of the inference request into multiple lexical windows when the inference request enters the prefilling stage, determine multiple first embedding lines that have not hit the device memory in the first lexical window, and simultaneously perform streaming direct memory access transport of the first embedding lines and batch bypass translation buffer warm-up of the target computing core; wherein, batch bypass translation buffer warm-up is to batch prefill the page table entries corresponding to each embedding line currently transported to the device memory into the on-chip bypass translation buffer of the computing core; The parallel computing module is used to start the pre-filling stage calculation of the first word window when the data prefetching operation of the first word window is completed, and to asynchronously and in parallel execute the data prefetching operation of the second word window until the pre-filling stage calculation of all word windows is completed, and then enter the decoding stage. The decoding stage prefetch module is used to determine multiple second embedding lines that have not hit the device memory based on the candidate lexicon tree, perform streaming direct memory access and speculative bypass translation buffer prefilling based on the second embedding lines, and perform main model inference calculation asynchronously and in parallel. Among them, speculative bypass translation buffer prefilling is to prestore the page table entry corresponding to the currently transported second embedding line into a dedicated prefetch translation buffer. The candidate lexicon tree update module is used to update the candidate lexicon tree according to the candidate path if the inference result of the current decoding step hits any candidate path of the candidate lexicon tree, so that it can be used in the next decoding step until the inference request is completed.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the data prefetching method for the large model inference service according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data prefetching method for the large model inference service as described in any one of claims 1-8.

Images