An end-to-end autoregressive large model inference acceleration method, device, medium and product

Abstract

The application discloses an end-side autoregressive large model reasoning acceleration method, equipment, medium and product. The method comprises the following steps: constructing a stable packaged calculation batch including a pre-filled micro-request and a plurality of decoding micro-requests at a first reasoning moment based on a continuous external request sequence; constructing corresponding segment description metadata, generating a cross-request hidden state tensor based on micro-request boundary information of the segment description metadata, the pre-filled micro-request and the plurality of decoding micro-requests; determining a batch segment meta tensor, combining the cross-request hidden state tensor to perform reasoning to obtain batch reasoning results; further updating the output sequence of each micro-request to obtain an output update sequence; performing in-place shift processing of a key-value cache, performing stage advancing rearrangement on slot positions of a unified key-value ring cache to obtain slot position rearrangement results; updating a global cross-request meta state based on each result, iteratively constructing a stable packaged calculation batch at a next reasoning moment, and cyclically performing reasoning processing until all external requests are generated. The efficiency of model reasoning acceleration is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, device, medium, and product for accelerating end-side autoregressive large model inference. Background Technology

[0002] In recent years, large-scale models have been widely applied in various fields. The continuous growth in model size and context length has made performance metrics such as throughput, latency, and memory usage during the inference phase crucial for edge deployment. Autoregressive inference is divided into a pre-filling stage with high computational utilization and a decoding stage with limited memory access and low hardware utilization. Currently, data center scenarios have effectively improved inference throughput and hardware efficiency through techniques such as batch processing scheduling, paginated key-value caching, and optimized attention operators. However, existing technologies are difficult to adapt to edge-side inference scenarios, which often involve continuous requests from a single user and have higher requirements for low jitter and low latency responses. Furthermore, optimizations such as quantization and cache compression only reduce the overhead of a single inference step and do not change the fundamental flaw of serial decoding per token. Various algorithm-level acceleration methods require modification of the model structure and retraining, resulting in high costs and poor compatibility. Currently, the industry lacks batch processing and cache scheduling solutions adapted to continuous edge requests, making it impossible to achieve efficient inference acceleration through system-level optimization without modifying the original model. Summary of the Invention

[0003] This invention provides a method, device, medium, and product for accelerating end-side autoregressive large model inference, in order to solve the problem of poor inference efficiency of end-side autoregressive large model inference.

[0004] According to one aspect of the present invention, a method for accelerating inference in large end-side autoregressive models is provided, comprising: Obtain a continuous external request sequence and construct a stable packaged computation batch at the first inference time based on the continuous external request sequence; wherein, the stable packaged computation batch includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and multiple decoded micro-requests, the multiple decoded micro-requests come from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage. Based on the global cross-request metadata state, construct the segment description metadata corresponding to the stable packaged computation batch. Based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch, and multiple decoded micro-requests, generate the cross-request hidden state tensor. Based on the segment description metadata, the batch segment meta tensor is determined. The cross-request hidden state tensor and the batch segment meta tensor as auxiliary input are jointly input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The autoregressive Transformer decoder performs variable length attention calculation and rotation position encoding writing processing based on the batch segment meta tensor, and directly writes the position encoded key and value into a unified key-value ring cache. The output sequence of each micro-request is updated based on the batch inference results to obtain the output update sequence; the original key-value cache is shifted, and the slots of the unified key-value circular cache are rearranged in stages according to the global cross-request meta state to obtain the slot rearrangement result; Based on the batch inference results, output update sequence, and slot rearrangement results, update the global cross-request meta state, iteratively construct the stable packaged calculation batch for the next inference time, and repeatedly execute inference processing until all external requests are generated.

[0005] Optionally, obtaining a continuous external request sequence and constructing a stable packaged computation batch for the first inference time based on the continuous external request sequence includes: performing task parsing on the continuous external request sequence to distinguish newly arrived external requests to be pre-filled and historical external requests at different decoding stages; converting newly arrived external requests to be pre-filled into first pre-filled micro-requests and mapping historical external requests at different decoding stages to corresponding first decoding micro-requests; using the available computing power resources and available key-value cache resources of the edge device as constraints, selecting second pre-filled micro-requests and second decoding micro-requests to be added to the current batch from the obtained first pre-filled micro-requests and first decoding micro-requests, and forming a micro-request combination based on the second pre-filled micro-requests and second decoding micro-requests; when the micro-request combination meets the preset batch stability conditions, constructing a stable packaged computation batch for the first inference time based on the micro-request combination.

[0006] Optionally, based on the global cross-request meta-state, segment description metadata corresponding to the stable packaged computation batch is constructed. Based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch, and multiple decoded micro-requests, a cross-request hidden state tensor is generated. This includes: obtaining the global cross-request meta-state; extracting information related to the stable packaged computation batch from the global cross-request meta-state; constructing the segment description metadata corresponding to the stable packaged computation batch; obtaining the input lexical sequences of the pre-filled micro-requests and multiple decoded micro-requests within the stable packaged computation batch, and obtaining the hidden features corresponding to each micro-request through embedding processing; and aggregating the hidden features corresponding to each micro-request based on the micro-request boundary information and offset information in the segment description metadata to generate a cross-request hidden state tensor.

[0007] Optionally, based on the segment description metadata, a batch segment meta tensor is determined. The cross-request hidden state tensor and the batch segment meta tensor, which serves as auxiliary input, are input together into an autoregressive Transformer decoder for batch inference to obtain the batch inference result. This includes: performing tensor structure transformation on the micro-request boundaries, offset indices, and cache slot information in the segment description metadata to generate a batch segment meta tensor carrying the position of each micro-request boundary, the effective sequence length, and the cache slot index; inputting the cross-request hidden state tensor and the batch segment meta tensor together into the autoregressive Transformer decoder, so that the decoder performs QKV projection based on the input hidden states in the cross-request hidden state tensor to obtain the query tensor, key tensor, and value tensor; performing rotational position encoding transformation only on the query tensor and key tensor based on the logical position of each micro-request term in the batch segment meta tensor, and directly writing the transformed key tensor and the value tensor without position encoding transformation into the corresponding slots of a unified key-value circular cache; performing variable-length attention calculation based on the micro-request boundary position and effective sequence length of the batch segment meta tensor, and obtaining the final batch inference result through preset subsequent network operations.

[0008] Optionally, the preset subsequent network processing includes cache maintenance operations and feature transformation operations; the feature transformation operations include multi-head attention output projection operations, residual connection operations, and multilayer perceptron feedforward operations.

[0009] Optionally, the output sequence of each micro-request is updated based on the batch inference results to obtain the output update sequence; key-value cache original shift processing is performed, and the slots of the unified key-value circular cache are rearranged in stages according to the global cross-request meta state to obtain the slot rearrangement result, including: parsing the predicted tokens corresponding to each micro-request from the batch inference results, adding the predicted tokens corresponding to each micro-request to the corresponding original output sequence to obtain the output update sequence corresponding to each micro-request; performing key-value cache original shift processing on the key-value data stored in each slot of the unified key-value circular cache, so that the key-value data completes the sequence dimension shift within its slot; reading the decoding iteration stage information and cache slot binding relationship of each micro-request in the global cross-request meta state, and performing stage-advance rearrangement on the slots of the unified key-value circular cache according to the iteration stage advancement of each micro-request after inference to obtain the slot rearrangement result; wherein, the slot rearrangement result is used to update the binding mapping between each micro-request and the unified circular cache slot.

[0010] Optionally, based on the batch inference results, output update sequence, and slot reordering results, the global cross-request meta state is updated, including: updating the output content and generation progress information of each micro-request in the global cross-request meta state based on the output update sequence; updating the current decoding iteration stage of each micro-request in the global cross-request meta state based on the batch inference results; and synchronously updating the binding mapping relationship between each micro-request in the global cross-request meta state and the unified key-value ring cache slot based on the slot reordering results, thereby completing the iterative update of the global cross-request meta state.

[0011] According to another aspect of the present invention, an end-side autoregressive large model inference acceleration device is provided, comprising: The stable packaging calculation batch determination module is used to obtain a continuous external request sequence and construct a stable packaging calculation batch at the first inference time based on the continuous external request sequence. The stable packaging calculation batch includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and multiple decoded micro-requests. The multiple decoded micro-requests come from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage. The cross-request hidden state tensor generation module is used to construct segment description metadata corresponding to stable packaged computation batches based on global cross-request metadata, and generate cross-request hidden state tensors based on micro-request boundary information of segment description metadata, pre-filled micro-requests in stable packaged computation batches and multiple decoded micro-requests. The batch inference result determination module is used to determine the batch segment meta tensor based on the segment description metadata. The cross-request hidden state tensor and the batch segment meta tensor as auxiliary input are jointly input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The autoregressive Transformer decoder performs variable length attention calculation and rotation position encoding writing processing based on the batch segment meta tensor, and directly writes the position encoded key and value into a unified key-value ring cache. The output update sequence and slot rearrangement result determination module is used to update the output sequence of each micro-request based on the batch inference result to obtain the output update sequence; perform key-value cache original shift processing, and perform phased advance rearrangement of the slots of the unified key-value ring cache according to the global cross-request meta state to obtain the slot rearrangement result; The state update and iterative inference module is used to update the global cross-request meta state based on the batch inference results, output update sequence and slot rearrangement results, iteratively construct the stable packaged calculation batch for the next inference time, and cyclically execute inference processing until all external requests are generated.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to execute the end-side autoregressive large model inference acceleration method of any embodiment of the present invention.

[0013] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the edge-side autoregressive large model inference acceleration method of any embodiment of the present invention.

[0014] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the end-side autoregressive large model inference acceleration method of any embodiment of the present invention.

[0015] The technical solution of this invention obtains a continuous external request sequence and constructs a stable batch of computation at the first inference time based on the continuous external request sequence. The stable batch of computation includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and multiple decoding micro-requests. These multiple decoding micro-requests originate from historical external requests at different request times, and each decoding micro-request is at a different decoding iteration stage. This achieves the mixing and batching of pre-filled micro-requests for newly arriving requests with decoding micro-requests from different historical times and different decoding iteration stages, fully utilizing computing power, avoiding idle computing power caused by the separate serial execution of pre-filling and decoding tasks, and mitigating drastic fluctuations in batch size. The process involves constructing segment description metadata corresponding to stable packaged computation batches based on global cross-request metadata. Based on the micro-request boundary information of the segment description metadata, pre-filled micro-requests in the stable packaged computation batches, and multiple decoded micro-requests, a cross-request hidden state tensor is generated. This achieves the reuse of globally maintained micro-request metadata and utilizes micro-request boundary information to complete the unified tensor organization of multi-task hidden features, eliminating the need for repeated parsing of original requests and reducing data copying overhead caused by scattered storage of multiple tensors. The batch segment metadata tensor is determined based on the segment description metadata. The cross-request hidden state tensor and the batch segment metadata tensor, used as auxiliary input, are jointly input into the autoregressive Transformer decoder for batch inference, yielding the batch inference result. The results show that the autoregressive Transformer decoder performs variable-length attention computation and rotational position encoding write processing based on the batch segment tensor. The position-encoded keys and values ​​are directly written to a unified key-value circular cache. By converting batch metadata into tensor inputs that the model can recognize, variable-length attention avoids excessive invalid computation. Key-value encoding is directly written to the cache, eliminating intermediate copying and reducing GPU memory data transfer overhead. The output sequence of each micro-request is updated based on the batch inference results to obtain the output update sequence. Key-value cache original shift processing is performed, and the slots of the unified key-value circular cache are rearranged in stages based on the global cross-request metadata state to obtain the slot rearrangement result, thus realizing batch... The system completes the update of the entire micro-request output sequence. The original shift does not trigger memory copying. The slot rearrangement adjusts the cache mapping according to the decoding progress, reducing key-value cache fragmentation and improving the reuse rate of the circular cache space. Based on the batch inference results, output update sequence and slot rearrangement results, the global cross-request meta state is updated. The system iteratively constructs a stable batch for the next inference time and executes inference processing in a loop until all external requests are generated. This achieves synchronization and alignment of information such as inference output, generation progress and cache slot mapping status, ensuring the accuracy of the next batch metadata and forming a complete autoregressive inference closed loop. This continuously supports the continuous generation scenario with a constant influx of requests and improves the stability of the system during long-term operation and the overall inference throughput.This solution enables hybrid batching of pre-filled micro-requests for newly arriving requests with decoded micro-requests from multiple historical requests and multiple decoding iteration stages. It fully reuses computational resources, leveraging a global cross-request meta-state to uniformly maintain the metadata of all micro-requests. Metadata tensor transformation of batch boundaries, offsets, and cache index information is achieved through segment description metadata and batch segment meta-tensors. Multi-task hidden features are aggregated using cross-request hidden state tensors, eliminating the need for extensive padding. Variable-length attention computation avoids unnecessary computational consumption. Key-value pairs processed by rotational position encoding are directly written to a unified key-value circular cache, reducing intermediate tensor copies. Cache fragmentation is alleviated through key-value cache original shift and slot-stage reordering. Synchronous updates to the global cross-request meta-state ensure consistency between the inference business state and the underlying cache mapping state. A closed-loop iterative scheduling mode continuously processes a constant stream of external requests, suppressing drastic batch size fluctuations, reducing memory usage and data transport overhead, significantly improving the throughput and long-term continuous operation stability of autoregressive Transformer edge-side batch inference, and enhancing the efficiency of edge-side autoregressive large model inference acceleration.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an end-side autoregressive large model inference acceleration method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a stable batch scheduling method for packaging calculation applicable to an embodiment of the present invention; Figure 3 This is a schematic diagram of a cross-request batch processing flow applicable to an embodiment of the present invention; Figure 4 This is a schematic diagram of the execution process of a unified KV caching and packing operator applicable to an embodiment of the present invention; Figure 5 This is a schematic diagram of the module composition of an end-side autoregressive large model inference acceleration system applicable to an embodiment of the present invention; Figure 6 This is a flowchart of an end-side autoregressive large model inference acceleration method provided in Embodiment 2 of the present invention; Figure 7 This is a schematic diagram of the structure of an end-side autoregressive large model inference acceleration device provided in Embodiment 3 of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device that implements the end-side autoregressive large model inference acceleration method of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] Example 1 Figure 1 This is a flowchart of an edge-side autoregressive large model inference acceleration method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of accelerating inference of edge-side autoregressive large models. The method can be executed by an edge-side autoregressive large model inference acceleration device, which can be implemented in hardware and / or software. The edge-side autoregressive large model inference acceleration device can be configured in electronic devices such as computers and servers. Figure 1 As shown, the method includes: S110. Obtain a continuous external request sequence and construct a stable packaged computation batch for the first inference time based on the continuous external request sequence; wherein, the stable packaged computation batch includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and multiple decoded micro-requests, the multiple decoded micro-requests are from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage.

[0022] Specifically, the continuous external request sequence can be understood as a set of external inference requests collected in real time by the autoregressive large model on the edge device and arranged in chronological order. The input port of the edge autoregressive large model can be continuously monitored, and the arriving external inference requests are cached and sorted according to their receiving timestamps. After filtering out invalid requests and discarding them, a time-ordered request queue is generated, thus obtaining the continuous external request sequence. For example, model input can include text prompts, historical context, multimodal features, or application status. The first inference moment can be specifically understood as the time point that triggers batch packaging and inference scheduling. It is the scheduling operation moment on the model side, used to mark the timing of hybrid batch inference. That is, the moment when a newly arrived external request enters the decoding stage for scheduling and inference operations can be determined as the first inference moment. A stable batch of computational requests can be understood as a set of micro-requests generated by the scheduler from a continuous sequence of external requests at the first inference time, intended for inference hardware to perform a single inference operation. At the first inference time, pre-filled micro-requests (also called Prefill micro-requests or simply Prefill) are extracted from newly arriving external requests at that time from the continuous external request sequence. Then, multiple decoding micro-requests (also called Decode micro-requests or simply Decode) from different request times and belonging to different decoding iteration stages are selected from the historical incomplete requests remaining in the sequence. These pre-filled micro-requests are combined with the selected decoding micro-requests. The decoding iteration stages of each decoding micro-request within the batch are constrained to be distinct. After verifying that the total number of micro-requests in the batch is within a reasonable range for hardware adaptation, a stable batch of computational requests is finally output. For example, assuming the current time is t, the computation batch... The current request needs to be processed. The system also retrieves historical requests. The first Decode input The second Decode input, and so on, until... The (K-1)th decode input. Thus, a batch contains one long-sequence Prefill micro-request and multiple single-token or short-token Decode micro-requests. The Prefill portion provides high computational intensity, while the Decode portion is executed within the same computational batch, reducing the low utilization caused by separate Decode batches.

[0023] Specifically, the large model on the edge continuously receives external requests and obtains a continuous sequence of external requests. At the first inference time, various inference micro-requests within this sequence are batch-packaged and aggregated to construct a stable packaged computation batch. This batch includes pre-filled micro-requests derived from external requests arriving at the first inference time that require pre-filling operations, as well as multiple decoding micro-requests derived from external requests at different historical request times. Furthermore, it controls each decoding micro-request to be in a distinct decoding iteration stage, merging decoding micro-requests with different lifecycles and decoding progresses with newly arrived pre-filled micro-requests into the same computation batch and sending it to the inference unit for execution. The stable packaged computation batch can be represented by a set relationship as follows: ; in, This indicates the stable batch calculation corresponding to the t-th request time. This represents the request at time t. This represents a historical request that is s requests later than the current request. The meaning of this formula is: when the request at request time t is set as the request at the current request time, a stable batch of data processing simultaneously includes the prefill micro-requests of the current request and the decode micro-requests of multiple historical requests. This allows for autoregressive processing of the entire stable batch of data processing, rather than waiting for each request to complete all autoregressive processing independently.

[0024] In a specific embodiment, let the external request sequence be: Each request corresponds to a request time, interaction round, or input segment. For the t-th request, the model first processes the input context to obtain the Prefill stage output and the initial KV buffer, then performs K Decode operations to generate an output sequence of length K. Here, K can be a fixed output token length, or an output length determined by the end-of-line character, tool call boundaries, or the maximum length. The generation process is considered as a K-level micro-request pipeline. Level 0 is the prefill for the current request, and levels 1 to K-1 correspond to the first to K-1 decodes of historical requests, respectively. In any stable computation batch, the system selects a prefill for the current request and selects K-1 historical request decode inputs at different decoding stages, combining these inputs into a packaged batch.

[0025] For example, such as Figure 2 The diagram illustrates a stable batch scheduling mechanism for prefilldecode computation in a single-user continuous inference scenario with a large model. The continuous external request stream generates a series of time-series requests. , , , Each complete request is broken down into a Prefill(P) micro-request and multiple Decode(P) micro-requests. (where 's' represents the Decode order) micro-requests, Prefill is used to initialize the KV cache, and Decode is used for autoregressive token generation. In the time series table, the Prefill and Decode micro-tasks of different requests proceed sequentially with inference time. Forming stable packaging calculation batches The system packages the latest request's Prefill computation task with multiple lightweight Decode micro-requests from previous requests into the same batch for parallel execution. This scheduling process involves three phases: pipeline filling, stable execution, and pipeline emptying. During the pipeline filling phase, the system begins receiving continuous requests, but the number of historical requests is less than K-1. This can be filled using empty phases, short batches, or traditional Decode methods. During the stable execution phase, the system outputs a completed historical request for each new request received, achieving continuous throughput. During the pipeline emptying phase, when external requests stop arriving, the system continues to advance the remaining historical requests until output is complete. For long-running edge-side interactive applications, the system can remain in the stable execution phase for extended periods, making the overhead of the pipeline filling and pipeline emptying phases relatively negligible. By mixing and batching Prefill and Decode tasks with significantly different computational characteristics, the system addresses the issue of idle computing power during continuous inference for single users, thereby improving hardware utilization. The same packaging logic continues to process subsequent microtasks until all requests are completed and the pipeline is drained. It's important to note that stable packaging computation batch scheduling must satisfy semantic consistency constraints. The Decode input for each historical request can only use the token already generated for that request and the KV cache belonging to that request; it cannot access the content of other requests. The Prefill for the current request can only establish its state based on the current input context and cannot be contaminated by historical Decode. The output order can be determined according to application requirements: in fixed-delay pipeline mode, the system output... The corresponding generation sequence; in low-latency hybrid mode, the system can also use the traditional method for the first few tokens of a new request, and the batch processing overlap method for the stable part.

[0026] It should also be noted that, to support stable batch scheduling for packaged computation, the system maintains a sequence accumulation buffer. This buffer stores incomplete historical requests in the current pipeline, as well as metadata such as the token sequence already generated for each request, the current decoding step count, the corresponding KV cache slot, and whether it is complete. When each new request arrives, the scheduler places it in level 0; after each packaged forward pass, each historical request advances to the next stage; when the number of tokens generated by a request reaches K or a termination condition is met, the system outputs the complete sequence or committable fragment of that request and releases its state.

[0027] In this embodiment, by mixing and batching decoding micro-requests that span request times and belong to different decoding iteration stages with the current pre-filling micro-requests, the computing power slots of the inference computing unit can be fully utilized, effectively smoothing out the computing power gap between pre-filling and decoding operations, improving the batch filling rate of the inference hardware, reducing hardware idle waste, and increasing the overall inference throughput in scenarios that carry a continuous influx of external inference requests. At the same time, it can alleviate the batch jitter problem caused by the concentrated arrival of new requests, ensure that the inference batch size tends to be stable, and reduce the fluctuation range of inference latency.

[0028] Optionally, obtaining a continuous external request sequence and constructing a stable packaged computation batch for the first inference time based on the continuous external request sequence includes: performing task parsing on the continuous external request sequence to distinguish newly arrived external requests to be pre-filled and historical external requests at different decoding stages; converting newly arrived external requests to be pre-filled into first pre-filled micro-requests and mapping historical external requests at different decoding stages to corresponding first decoding micro-requests; using the available computing power resources and available key-value cache resources of the edge device as constraints, selecting second pre-filled micro-requests and second decoding micro-requests to be added to the current batch from the obtained first pre-filled micro-requests and first decoding micro-requests, and forming a micro-request combination based on the second pre-filled micro-requests and second decoding micro-requests; when the micro-request combination meets the preset batch stability conditions, constructing a stable packaged computation batch for the first inference time based on the micro-request combination.

[0029] Specifically, the first prefill micro-request can be understood as a raw granular task unit used to perform prefill calculations, derived from newly arrived external requests awaiting prefill. After identifying newly arrived external requests awaiting prefill through task parsing, this type of request is split into micro-requests, retaining the complete input context and parameters required for prefill calculations. The resulting prefill granular task, which has not yet undergone resource constraint filtering, is the first prefill micro-request. The first decoding micro-request can be understood as a raw granular task unit used to perform decoding iterative calculations, generated by mapping parsed historical external requests at different decoding stages. This can be achieved by identifying historical external requests that have not completed inference, reading the current decoding iteration stage of each historical external request, the generated token information, and the KV cache context, and mapping the resulting decoding granular task, which has not yet undergone resource constraint filtering, is the first decoding micro-request. The second pre-filled micro-request is a pre-filled micro-request that is allowed to participate in the current batch assembly after being filtered from the first pre-filled micro-requests by constraints on available computing power resources and available key-value cache resources on the edge side. This is achieved by comparing the computing power cost and estimated key-value cache usage of each first pre-filled micro-request with the current remaining available computing power and key-value cache resources on the edge device, eliminating micro-requests that would cause resource overruns. The remaining pre-filled micro-requests that can be included in the current batch are the second pre-filled micro-requests. The second decoding micro-request can be specifically understood as a decoding micro-request that is selected from the first decoding micro-requests by constraints on available computing power resources and available key-value cache resources on the edge side and can participate in the current batch assembly. This can be achieved by filtering based on the single-step decoding computing power consumption, existing key-value cache usage, and incremental cache cost of each first decoding micro-request, combined with the real-time resource availability on the edge side, eliminating decoding micro-requests whose resource costs exceed the remaining capacity, and obtaining the decoding micro-requests that can be used as batch candidates, which are the second decoding micro-requests. The preset batch stability condition can be understood as a set of judgment rules used to verify whether the combination of micro-requests consisting of the second pre-filled micro-request and the second decoding micro-request can formally form a stable batch for computation. The upper and lower limit thresholds of the batch size, the ratio threshold of the number of pre-filled and decoding micro-requests, the differential constraints of the decoding iteration stages between decoding micro-requests, and the upper limit threshold of batch resource consumption can also be configured. When the combination of micro-requests simultaneously meets all or part of the rules that the total number of micro-requests in the batch falls within the upper and lower limit range, the ratio of pre-filled and decoding micro-requests is within the allowable range, the decoding iteration stages of each decoding micro-request do not overlap, and the overall resource consumption does not exceed the limit, it is determined that the preset batch stability condition is met.

[0030] Specifically, when acquiring a continuous external request sequence and constructing a stable packaged computation batch for the first inference time, the continuous external request sequence is parsed to distinguish between newly arrived external requests to be pre-filled and historical external requests at different decoding stages. Then, the newly arrived external requests to be pre-filled are transformed into first pre-filled micro-requests, and historical external requests at different decoding stages are mapped to corresponding first decoding micro-requests. Then, using the available computing power resources and available key-value cache resources of the edge device as hard constraints, the second pre-filled micro-requests and second decoding micro-requests to be added to the current batch are selected from all first pre-filled micro-requests and first decoding micro-requests, and then assembled to obtain a micro-request combination. The micro-request combination is then checked to see if it meets the preset batch stability conditions. After the check passes, the stable packaged computation batch corresponding to the first inference time is generated based on the micro-request combination.

[0031] In this embodiment, pre-filled requests and multi-stage historical decoding requests are classified and decomposed through task parsing, and the original external requests are uniformly transformed into standardized micro-request granularity. At the same time, edge computing power and key-value cache resources are introduced for filtering constraints, which can avoid resource overflow and inference errors caused by batch tasks exceeding the hardware capacity limit. Combined with preset batch stability conditions for final verification, it can realize the mixed batching of pre-filled micro-requests and multi-decoding stage decoding micro-requests, fully utilize hardware computing power and KV cache (key-value cache) to improve hardware resource utilization, and control the batch size within a reasonable range, suppress drastic batch size jitter, reduce edge inference latency fluctuations, and effectively improve the inference throughput and operational stability of edge devices while ensuring the normal scheduling and execution of inference tasks.

[0032] S120. Construct segment description metadata corresponding to the stable packaged computation batch based on the global cross-request metadata state. Generate cross-request hidden state tensor based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch, and multiple decoded micro-requests.

[0033] Specifically, the global cross-request meta-state can be understood as a collection of non-tensor-based global meta-information that manages all active micro-requests and stores during the inference process of each request. It can continuously maintain meta-information such as request identifiers, inference progress, KV cache indexes, resource usage records, and lifecycle markers for all prefilled micro-requests that have not yet completed inference, as well as decoded micro-requests, aggregating them to form the global cross-request meta-state. The segment description metadata can be understood as a descriptive metadata structure specifically generated for the current stable packaged computation batch. It can read the relevant meta-information of all micro-requests within the batch from the global cross-request meta-state, and generate segment description metadata specific to the batch by combining the micro-request arrangement order within the batch. The segment description metadata includes at least: segment start offset, segment length, belonging request identifier, pipeline stage number, logical position, KV cache start offset, KV effective length, attention mask type, and output write-back position. For prefilled micro-requests, their attention mask is usually a causal mask, so that the tokens within the segment can only focus on the current position and previous positions. For Decode micro-requests, the query length is typically 1. The attention context is the existing key-value history of the request plus the current token; the key-value history of other requests cannot be accessed. Micro-request boundary information can be understood as the positional markers within the segment description metadata used to mark the start offset, end offset, request type, and request ID of each micro-request in the tensor. This can be obtained by traversing the pre-filled micro-requests and each decoded micro-request within the stable batch, calculating the start and end offsets of each micro-request sequentially according to their arrangement in the tensor, and associating the micro-request type with the request identifier. The cross-request hidden state tensor can be understood as a unified tensor obtained by regularly arranging the hidden states of the pre-filled micro-requests and multiple decoded micro-requests in the stable batch according to the boundary information. Based on the micro-request boundary information carried by the segment description metadata, the hidden state data corresponding to each micro-request can be read, data filling and alignment can be completed according to the offset position, and the hidden states belonging to different requests can be merged and assembled into a continuous tensor storage space, ultimately resulting in the cross-request hidden state tensor.

[0034] Specifically, based on the global cross-request metadata state, a segment description metadata corresponding to the stable packaged computation batch is constructed. This segment description metadata carries the boundary information corresponding to each micro-request within the batch. Then, the micro-request boundary information in the segment description metadata is read, combined with the pre-filled micro-requests and multiple decoded micro-requests within the stable packaged computation batch, and based on the boundary information such as the request affiliation, sequence segmentation position, and hidden state offset position of each micro-request, the hidden state data corresponding to different requests is segmented, spliced, and aligned, thereby generating a cross-request hidden state tensor adapted to the hybrid batch inference operation.

[0035] In this embodiment, by leveraging the global cross-request meta-state to uniformly maintain the context state information of multiple requests, and by explicitly recording the boundary segmentation information between different micro-requests through segment description metadata, it is possible to uniformly organize the hidden states of micro-requests from different requests within the same batch that have undergone pre-filling and decoding at different iteration stages. This solves the problems of blurred boundaries of hidden states and disordered tensor arrangement in different inference tasks under mixed batch scenarios. It eliminates the need to maintain independent tensors for each micro-request, achieving tensor-based compact storage of hidden states of multiple requests, reducing the data copying overhead caused by tensor splitting and merging, adapting to the inference computation mode of mixed packaging of pre-filled and multi-stage decoding micro-requests, ensuring that subsequent tensor operations can accurately distinguish the hidden state fragments corresponding to each micro-request, improving tensor computation efficiency, reducing memory usage, and ensuring the correctness of the mixed batch inference process.

[0036] S130. Determine the batch segment meta tensor based on the segment description metadata. Input the cross-request hidden state tensor and the batch segment meta tensor as auxiliary input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The autoregressive Transformer decoder performs variable-length attention calculation and rotation position encoding write processing based on the batch segment meta tensor, and directly writes the position-encoded keys and values ​​into a unified key-value circular cache.

[0037] Specifically, the batch segment meta-tensor can be understood as a structured tensor derived from segment description metadata and used as model input. It can be obtained by parsing the segment description metadata corresponding to the stable packaged computation batch, extracting internal micro-request boundary offsets, task types, request indices, and other meta-information, and encoding it into a tensor format recognizable by the model. The batch inference result can be understood as the set of outputs from the autoregressive Transformer decoder after completing inference for the entire stable packaged computation batch. The decoder can perform all forward computations, outputting corresponding token predictions and state outputs for each pre-filled micro-request and decoded micro-request within the batch, and aggregating the outputs of all micro-requests to obtain the batch inference result. Variable-length attention computation can be understood as an attention operation mode adapted to the varying lengths of the context sequences of each micro-request within the batch. The decoder can read the start and end boundaries of the micro-requests carried by the batch segment meta-tensor, divide the effective attention intervals of each independent request according to the boundaries, and perform attention weight calculations only within the effective sequence range of each micro-request, skipping invalid padding positions, thus completing the variable-length attention computation. Rotational position encoding can be understood as an encoding method that injects sequence position information into the input token. Based on the actual sequence position index corresponding to each micro-request in the batch segment tensor, rotational transformation operations are performed on the query and key tensors, binding corresponding position information to each token to obtain the rotated position encoded key and value. The unified key-value circular cache can be understood as a video memory cache area that centrally and cyclically stores the key-value states of all micro-requests within a batch. A pre-allocated, circularly reused, contiguous video memory space receives the rotated position encoded key and value tensors and writes them directly to the corresponding circular storage location according to the cache offset of each micro-request, achieving unified management of the KV states of multiple requests, thus constituting the unified key-value circular cache used in this inference.

[0038] Specifically, a batch segment meta tensor is constructed based on the generated segment description metadata. The cross-request hidden state tensor and the batch segment meta tensor as auxiliary input are fed into the autoregressive Transformer decoder for batch inference operations to obtain batch inference results. During the inference execution phase, the autoregressive Transformer decoder reads the segment boundary, task type and position-related information carried by the batch segment meta tensor, completes the variable-length attention calculation and rotation position encoding writing process for multiple micro-requests within the batch, and then writes the key tensor and value tensor after position encoding directly into a unified key-value circular cache to complete state retention.

[0039] In this embodiment, segment description metadata is converted into batch segment meta-tensors that can participate in model computation, realizing the transformation of metadata into tensor input. This allows the decoder to process pre-filled, multi-stage decoding micro-requests from different requests simultaneously within a single batch. Relying on batch segment meta-tensors to drive variable-length attention computation, it can adapt to scenarios where the context lengths of each micro-request are inconsistent, avoiding unnecessary zero-padding of the input sequence and reducing invalid computation overhead. After the rotation position encoding is processed internally by the decoder, it is directly written to disk to a unified key-value circular cache, eliminating the steps of intermediate copying, transfer, and KV reorganization, reducing data transport overhead. With the help of the circular cache, the KV cache can be reused and managed efficiently. In the batch inference scenario of pre-filled decoding mixed batches, it not only ensures the correctness of multi-task attention and position encoding computation, but also effectively reduces memory usage, improves batch inference throughput, and reduces cache fragmentation problems caused by switching between multiple requests.

[0040] Optionally, based on the segment description metadata, a batch segment meta tensor is determined. The cross-request hidden state tensor and the batch segment meta tensor, which serves as auxiliary input, are input together into an autoregressive Transformer decoder for batch inference to obtain the batch inference result. This includes: performing tensor structure transformation on the micro-request boundaries, offset indices, and cache slot information in the segment description metadata to generate a batch segment meta tensor carrying the position of each micro-request boundary, the effective sequence length, and the cache slot index; inputting the cross-request hidden state tensor and the batch segment meta tensor together into the autoregressive Transformer decoder, so that the decoder performs QKV projection based on the input hidden states in the cross-request hidden state tensor to obtain the query tensor, key tensor, and value tensor; performing rotational position encoding transformation only on the query tensor and key tensor based on the logical position of each micro-request term in the batch segment meta tensor, and directly writing the transformed key tensor and the value tensor without position encoding transformation into the corresponding slots of a unified key-value circular cache; performing variable-length attention calculation based on the micro-request boundary position and effective sequence length of the batch segment meta tensor, and obtaining the final batch inference result through preset subsequent network operations.

[0041] Optionally, the preset subsequent network operations include cache maintenance and feature transformation operations. Feature transformation operations include multi-head attention output projection, residual connection, and multilayer perceptron feedforward. It should be noted that cache maintenance is a key-value cache original shift operation, which performs a sequence-dimensional original shift on the stored key-value sequences within each slot of the unified key-value circular cache in the current Transformer layer, without changing the binding relationship between micro-requests and cache slots. Feature transformation operations include multi-head attention output projection, residual connection, and multilayer perceptron feedforward. Multi-head attention output projection is used to linearly project the output features of variable-length attention to aggregate multi-head features. Residual connection is used to add the attention projection result to the original cross-request hidden state tensor of the current layer, obtaining the residual output features. Multilayer perceptron feedforward is used to perform a non-linear feedforward transformation on the residual output features, outputting the final hidden state of the current Transformer layer.

[0042] Specifically, the batch segment meta tensor is determined based on the segment description metadata. The cross-request hidden state tensor and the batch segment meta tensor, which serves as auxiliary input, are fed into the autoregressive Transformer decoder to perform batch inference and obtain the batch inference result. First, the micro-request boundaries, offset indices, and cache slot information within the segment description metadata are transformed into tensors to generate batch segment meta tensors carrying the positions of each micro-request boundary, the effective sequence length, and the cache slot index. Then, the cross-request hidden state tensor and the batch segment meta tensor are input into the autoregressive Transformer decoder. The decoder performs QKV projection based on the input hidden states in the cross-request hidden state tensor and outputs the query tensor. The batch segment tensor, key tensor, and value tensor are processed. Then, the decoder reads the lexical logical positions of each micro-request in the batch segment tensor and performs rotational position encoding transformation only on the query tensor and key tensor. The transformed key tensor and the value tensor without position encoding transformation are directly written into the corresponding slots of the unified key-value circular cache. The decoder then performs variable-length attention calculation based on the micro-request boundary positions and effective sequence lengths provided by the batch segment tensor. Further processing is carried out through preset subsequent network operations, including cache maintenance operations and feature transformation operations. The feature transformation operations sequentially complete multi-head attention output projection operations, residual connection operations, and multilayer perceptron feedforward operations to finally obtain the batch inference results.

[0043] For example, such as Figure 3The diagram illustrates a cross-request batch processing flow. It shows an edge-side large-model overlapping and packaging inference pipeline architecture. A continuous stream of external requests is input into the batch processing overlapping scheduler. The scheduler performs micro-request splitting, selecting prefill tasks and historical decoding tasks to construct mixed packaging computation batches. Each packaging batch simultaneously carries the current request's Prefill task and multiple historical request Decode iteration tasks. These are then sent to the cross-request state modeling and packaging forward execution module, which sequentially completes unified state packaging, Transformer layer stacking forward computation, and KV cache write-back processing. The processing results flow into the generation sequence accumulation buffer to maintain the state of each stage of the pipeline. Finally, the output and release stages continuously output completed requests and release corresponding resources, achieving one completed request output per batch. Leveraging the temporal predictability of continuous requests from a single user, mixed batch processing of Prefill and Decode tasks is achieved, improving edge-side inference hardware utilization.

[0044] In this embodiment, the boundary, offset, and slot metadata information in the segment description metadata are uniformly transformed into batch segment metadata tensors that can be read and computed by the model, thus achieving the connection between metadata and computation graph. QKV projection is used to separate query, key, and value tensors. Differentiation is achieved by performing rotational position encoding only on query and key tensors, while value tensors retain their original data and directly fall into the corresponding slots of a unified key-value circular cache. This avoids meaningless repeated rotation transformations of value tensors, reducing unnecessary computation. Variable-length attention computation is driven by batch segment metadata tensors, eliminating the need for padding and avoiding unnecessary computational waste. Standardized feature transformation operations such as multi-head attention output projection, residual connections, and multi-layer perceptron feedforward, combined with cache maintenance operations, complete the entire decoder forward link. This is compatible with inference scenarios where pre-filled micro-requests and multi-stage decoding micro-requests are mixed in the same batch, ensuring the correctness of position encoding and attention computation logic, reducing intermediate tensor copying, lowering GPU memory usage, and improving the computational efficiency of batch inference. Simultaneously, targeted writing of KV values ​​is achieved through cache slot indexing, providing an accurate key-value cache foundation for subsequent iterative inference.

[0045] It should be noted that during the forward pass of the packaged process, the cross-request hidden state tensor CRS is passed sequentially through each Transformer layer as a single, unified input. CRS is a unified tensor formed by aggregating the hidden states of multiple pipeline stages within the same computation batch along a specified dimension. For a stable batch containing K stages, CRS includes the Prefill input for the current request. And the Decode input of K-1 historical requests. .in, It can be the length sequence, The token embedding is of length 1, or in some implementations, a short token block, with s taking values ​​from 2 to K-1. For matrix calculations such as linear projection, output projection, and MLP, the number of rows in the input matrix increases after packaging, and the small matrix vector operations that were originally performed in multiple decoding stages are merged into larger matrix operations. For GPUs and tensor computation units, larger matrices can usually better utilize parallelism, shared memory, and tensor cores, thereby improving computational efficiency. For element-wise operations, such as LayerNorm, activation functions, and residual connections, the system can directly execute the entire CRS or execute them in parallel according to segment boundaries, without changing the computational results. The attention layer is a key challenge in the packaged forward pass. Because each segment in the CRS has a different KV history length, it is not possible to simply concatenate all KV values ​​and allow all queries to share the same context. In this embodiment, a variable-length attention mechanism is adopted. The variable-length attention operator reads the KV range corresponding to each segment according to the segment description metadata and calculates the attention according to the context length and mask of each segment. Thus, although the current Prefill segment and each historical Decode segment are executed within the same operator call, logically they are still equivalent to executing their respective attention segments separately. Mathematically speaking, packaged execution does not change the input token sequence of each request, nor does it change the autoregressive dependencies generated by each request. For any historical request... The s-th decode operation retains the same query vector, key / value history, and positional encoding as when executed alone. In this embodiment, the change lies in the hardware organization of these computations, rather than the function definitions of the model; this represents a system-level optimization at the inference runtime.

[0046] In a preferred embodiment, the system performs the following packaging forward process at each layer: First, for Normalization was performed to obtain Secondly, regarding The process involves performing QKV projection to obtain Q, K, and V; then, calling the fused RoPE write operator (fused rotation position encoding write operator) to process position encoding and KV writing; subsequently, calling the variable-length attention operator to obtain attention output; then, performing in-situ KV shift or cached metadata update; finally, performing output projection, residual connections, feedforward network, and next-layer input update. This process can be encapsulated as a cross-request state packaged forward operator, or it can be implemented by combining multiple kernels and schedulers. It should be noted that CRS also supports dynamic strategies. If the current input image is complex, the language instructions are long, or the prefill length is large, the scheduler can reduce the number of decodes in the same batch to avoid exceeding GPU memory or register resources. If the current prefill is short or hardware utilization is insufficient, the scheduler can increase the number of decode stages or use multi-token decode blocks. For models with different numbers of layers, heads, and KV compression formats, the CRS description method can remain consistent; only the underlying operator implementation needs to be adjusted.

[0047] S140. Update the output sequence of each micro-request based on the batch inference results to obtain the output update sequence; perform key-value cache original shift processing, and perform phased advance rearrangement of the slots of the unified key-value ring cache according to the global cross-request meta state to obtain the slot rearrangement result.

[0048] Specifically, the output update sequence can be understood as the latest generated sequence obtained after each micro-request within a batch completes token output iteration based on the batch inference results. It can iterate through each micro-request in the stable batch, extract the predicted token for the corresponding micro-request from the batch inference results, append the predicted token to the end of the original output sequence of that micro-request, and update the latest output update sequence for each micro-request. Key-value cache original shift processing refers to memory operations that adjust the position of key-value data within the cache without moving the key-value data out of the current video memory storage area. This can be achieved by keeping the key and value tensors within the original video memory address of a unified key-value circular cache, and only modifying the pointers to the valid internal data range to complete the shift in the sequence dimension, without triggering data copying or memory reallocation. A slot in a unified key-value circular cache refers to an independent storage subunit within the unified key-value circular cache, used to hold the key-value state of a single micro-request. Based on the pre-defined single-task cache granularity during memory allocation, the entire circular memory space can be divided into multiple independently addressable storage sub-blocks, each of which is a slot in the unified key-value circular cache. The slot reordering result refers to the new cache slot layout obtained after the iteration phase advancement and occupancy adjustment of slots according to the global cross-request meta-state. This involves reading the decoding iteration phase, task completion marker, and slot occupancy record of each active micro-request in the global cross-request meta-state, releasing slots occupied by completed inference tasks, updating the corresponding slot index and phase pointer for micro-requests still performing inference, adjusting the mapping relationship between each micro-request and slot, forming a new slot occupancy layout, and thus obtaining the slot reordering result.

[0049] Specifically, the batch inference results output by the autoregressive Transformer decoder are obtained. The output sequence of each micro-request within the batch is updated using these batch inference results to generate the corresponding output update sequence. Then, the key-value cache original shift processing is performed on the unified key-value circular cache, and the inference iteration progress, cache occupancy index, and lifecycle information of each micro-request in the global cross-request meta-state are read. Based on the above meta-state information, the stage-wise advancement rearrangement of each slot within the unified key-value circular cache is completed, and the slot rearrangement result is finally obtained.

[0050] In this embodiment, the output sequence of each micro-request is synchronously updated based on the batch inference results, realizing the batch synchronous refresh of the inference output of multiple micro-requests. This avoids the performance loss caused by serial updates per request. The use of key-value cache original shift processing eliminates the need to move KV cache data out of video memory for additional copying, reducing the data transfer overhead between video memory. At the same time, combined with the global cross-request meta state driving cache slots to advance and rearrange according to the inference iteration stage, it can adapt to the cache state migration after pre-filling and mixing micro-requests in different decoding iteration stages. It can reasonably schedule each storage slot in the ring cache, release the slot space occupied by completed tasks in a timely manner, adjust the cache storage location of unfinished inference tasks, alleviate cache fragmentation, improve the space reuse efficiency of the unified key-value ring cache, and provide a regular and usable cache storage foundation for the construction and inference of the next round of stable packaged computation batches, ensuring that continuous multi-round mixed batch inference can run stably.

[0051] Optionally, the output sequence of each micro-request is updated based on the batch inference results to obtain the output update sequence; key-value cache original shift processing is performed, and the slots of the unified key-value circular cache are rearranged in stages according to the global cross-request meta state to obtain the slot rearrangement result, including: parsing the predicted tokens corresponding to each micro-request from the batch inference results, adding the predicted tokens corresponding to each micro-request to the corresponding original output sequence to obtain the output update sequence corresponding to each micro-request; performing key-value cache original shift processing on the key-value data stored in each slot of the unified key-value circular cache, so that the key-value data completes the sequence dimension shift within its slot; reading the decoding iteration stage information and cache slot binding relationship of each micro-request in the global cross-request meta state, and performing stage-advance rearrangement on the slots of the unified key-value circular cache according to the iteration stage advancement of each micro-request after inference to obtain the slot rearrangement result; wherein, the slot rearrangement result is used to update the binding mapping between each micro-request and the unified circular cache slot.

[0052] Specifically, the output sequence of each micro-request is updated based on the batch inference results to obtain the output update sequence. Then, the original key-value cache is shifted, and the slots of the unified key-value circular cache are rearranged in stages according to the global cross-request meta state to obtain the slot rearrangement result. Specifically, the predicted tokens corresponding to each micro-request are parsed from the batch inference results. The predicted tokens of each micro-request are appended to their original output sequences to generate the output update sequence corresponding to each micro-request. Then, the original key-value cache is shifted for the key-value data stored in each slot of the unified key-value circular cache, so that the key-value data completes the sequence dimension shift operation within its own slot. Next, the decoding iteration stage information of each micro-request and the cache slot binding relationship saved by the global cross-request meta state are read. Combined with the progress of the iteration stage after the inference of each micro-request is completed, the slots of the unified key-value circular cache are rearranged in stages, and the slot rearrangement result is output to update the binding mapping relationship between each micro-request and the unified circular cache slots.

[0053] In this embodiment, predicted terms are appended to the original output sequence by parsing the batch inference results, and the iterative update of all micro-request output sequences is completed in batches. This avoids the performance loss caused by serial processing of each request. The key-value cache original shift processing completes the sequence dimension adjustment within the slot, without triggering data migration and additional memory copying, reducing the overhead of video memory data movement. Based on the iteration stage and slot binding information carried by the global cross-request meta state, the slot reordering is driven, which can dynamically adjust the mapping and binding relationship between micro-requests and cache slots according to the decoding progress. This adapts to the iterative generation scenario of mixed batching of pre-filled and multi-stage decoding micro-requests, adapts to changes in the inference stage in a timely manner, organizes the internal storage layout of the cache, suppresses the generation of key-value cache fragmentation, and ensures that the mapping relationship between micro-requests and cache slots is accurate and effective. This provides reliable underlying storage support for batch construction and KV cache read and write in the next round of inference, and improves the reuse efficiency of cache resources and the stability of system operation under long-term continuous autoregressive inference.

[0054] It should be noted that after performing cross-request state packaging processing to obtain the cross-request hidden state tensor, KV cache management becomes a key factor determining whether performance gains can be realized. This embodiment proposes a unified KV circular cache to reduce the cost of dynamic collection and copying at each layer. The unified KV circular cache is a physically contiguous cache area maintained for each Transformer layer. This cache area can be divided into several slots by stage or several logical partitions by request. Each slot stores the Key and Value history of a certain active micro-request. The system records the logical request number, current stage, effective length, write position, overwrite flag, and target slot for the next stage for each slot. Since the slots are physically contiguous, the variable-length attention operator can directly read the corresponding range, avoiding the temporary allocation of contiguous buffers at each layer. The lifecycle of the circular cache is synchronized with the batch processing overlapping pipeline. When the current request enters the Prefill stage, its newly generated KV is written to the stage 0 slot. In the next batch, the request enters the Decode1 stage, and the corresponding KV is regarded as the stage 1 history. And so on, the request advances step by step in the pipeline. After the oldest request completes K steps of generation, its slot can be overwritten by new requests. Circular caches achieve phase advancement through physical copying, index rotation, or a combination of both. If the hardware supports efficient address mapping, only metadata needs to be updated; if the variable-length attention kernel requires a strictly continuous physical layout, in-place shifts can be performed on the device side to move the slot data to the next phase position.

[0055] It should also be noted that the fusion operator is invoked immediately after QKV projection to calculate the RoPE for each token based on its logical position in the original request, and the position-encoded Key and Value are directly written to a predetermined address in the unified KV circular cache. For queries, position encoding can also be completed in the same operator and written to the subsequent attention input buffer. This can reduce one or more memory accesses. An in-situ KV shift operator is also proposed. This operator is executed after each calculation batch ends or before the next batch begins, and is used to update the slots of each stage in the unified KV circular cache. In-situ shifting can move the cached content from stage 0 to stage K-2 to stage 1 to stage K-1, or it can only adjust the circular index to map the logical stage to the new physical slot. For attention operators that require physically continuous layout, in-situ shifting is preferably performed on the device side to avoid the CPU initiating multiple small copies and synchronizations. For attention operators that support pagination or block sparse format, in-situ shifting can degenerate into metadata updates and block table rearrangement. The idea of ​​overlapping operator execution is also proposed. Operator overlap is not simply about parallelizing two arbitrary operators; rather, it involves merging and packaging them based on the complementary nature of Prefill and Decode computations. The Prefill stage is typically computationally intensive, while the Decode stage is typically memory-intensive. Integrating Decode into Prefill batches allows for better interleaving of Decode memory access and Prefill computation within the hardware pipeline. Simultaneously, the fusion of RoPE writes, variable-length attention, and in-situ key-value shifts reduces data movement around the attention layer, ensuring that the benefits of batch overlap are not offset by cache reordering overhead.

[0056] In some implementations, the system can further introduce double buffering or multiple CUDA stream mechanisms. For example, when performing attention and MLP in the current batch, the image encoding or input embedding of the next batch can be prepared in advance; when performing computation in the current layer, the cache metadata required by the next layer can be generated in advance; during in-situ KV shifting, it can overlap with projection computation that does not depend on the cache region. The unified KV circular cache differs from ordinary PagedAttention. PagedAttention mainly solves the problem of dynamic growth and fragmentation of KV cache in multi-user LLM services, and it manages non-contiguous KV blocks through a block table. It should be noted that the fixed progression relationship of multiple stages within a single-user continuous control flow emphasizes the use of physically or logically continuous layout to allow variable-length attention to directly read different historical lengths, and avoids dynamic gathering at each layer through device-side writing and shifting. The two can be used in combination: paging blocks are used in the global memory management layer, and circular slots and stage metadata are used within a single continuous control flow. The cooperation relationship between unified KV circular cache, RoPE writing, variable-length attention, and in-situ KV shifting can be found in [reference needed]. Figure 4 , Figure 4This diagram illustrates a unified key-value caching and packing operator execution flow, specifically showcasing the inference computation flow of the Transformer decoder layer equipped with cross-request state CRS input, and the hidden state input from the cross-request state CRS. First, a QKV tensor is obtained through a normalization (Norm) and QKV projection module. Then, it is fed into a RoPE-fused KV writing module to apply rotational position encoding (rotary(Q)) to Q. The rotated rotation(K) and V are then written into the KVCache. The stored KV cache participates in the attention calculation. The cache state is updated using an in-place shift KV module. The attention output is projected onto the O-value and added to the residual. Finally, it is fed into the MLP module for processing, outputting the result for the current layer. The overall approach couples RoPE rotation, KV cache read / write, attention calculation, and in-place KV shift operations. It reuses multiple KVCache (a)(b)(c) sets of caches to store historical key-value tensors for different requests, achieving integrated computation of attention and KV cache across request states. During the stable packaging and computation of batches, the slot relationships between adjacent stages in the circular cache can be abstracted as follows: This formula indicates that the cache state of stage s in the t-th batch will advance to stage s+1 in the next batch; in practice, physical shifting can be used, or logical shifting can be achieved through circular index or block table updates.

[0057] S150. Based on the batch inference results, output update sequence and slot rearrangement results, update the global cross-request meta state, iteratively construct the stable packaged calculation batch for the next inference time, and repeatedly execute inference processing until all external requests are generated.

[0058] Specifically, combining the batch inference results output by the decoder, the output update sequence obtained through inference iteration, and the slot reordering results obtained after cache adjustment, the meta-information such as the inference progress, output content, slot mapping relationship, and task lifecycle of each micro-request stored in the global cross-request meta-state is refreshed and updated as a whole. Then, based on the updated global cross-request meta-state, a stable packaged calculation batch corresponding to the next inference moment is iteratively constructed. The entire inference process of task parsing, micro-request filtering and batching, metadata construction, tensor generation, Transformer batch inference, output update, and key-value cache slot reordering is repeated until all external requests have completed token generation and the inference ends.

[0059] In this embodiment, the inference output, generation sequence state, and cache slot layout are linked together to update the global cross-request meta state, achieving consistency and synchronization between the inference business state and the underlying cache resource state. This ensures that the next batch can obtain accurate and up-to-date task and cache information. Relying on the closed-loop processing mode of iterative loop, it continuously processes the continuous influx of external requests, continuing the scheduling advantages of pre-filling and multi-stage decoding micro-request hybrid batching. It can dynamically adapt to business scenarios where requests are constantly flowing in and some historical requests are continuously decoded and output, reducing scheduling errors caused by inconsistent states, continuously maintaining stable batch size, reducing the accumulation of memory cache fragmentation, improving the overall system's inference throughput and long-term operational stability, and ensuring that all external requests can complete autoregressive generation in an orderly manner.

[0060] Optionally, based on the batch inference results, output update sequence, and slot reordering results, the global cross-request meta state is updated, including: updating the output content and generation progress information of each micro-request in the global cross-request meta state based on the output update sequence; updating the current decoding iteration stage of each micro-request in the global cross-request meta state based on the batch inference results; and synchronously updating the binding mapping relationship between each micro-request in the global cross-request meta state and the unified key-value ring cache slot based on the slot reordering results, thereby completing the iterative update of the global cross-request meta state.

[0061] The generation progress information refers to the state information stored in the global cross-request metastate, used to characterize the total number of tokens generated by each micro-request, whether pre-filling is complete, and the completeness of the output sequence. It can read the output update sequence of the corresponding micro-request, count the number of generated tokens, and combine this with the pre-filling task completion marker and the output sequence termination judgment marker to summarize the generation progress information of that micro-request. The current decoding iteration stage refers to the state parameter used to identify the autoregressive decoding round in which the micro-request is located after pre-filling is completed. It can be obtained by accumulating the decoding round count after the pre-filling stage based on whether the micro-request generates valid predicted tokens in this round according to the batch inference results.

[0062] Specifically, the global cross-request meta-state is iteratively updated based on the batch inference results, output update sequence, and slot reordering results. Specifically, the output content and generation progress information of each micro-request within the global cross-request meta-state are rewritten according to the output update sequence. Then, the current decoding iteration stage corresponding to each micro-request in the global cross-request meta-state is updated according to the batch inference results. At the same time, the binding mapping relationship between each micro-request and the unified key-value ring cache slot within the global cross-request meta-state is synchronously refreshed using the slot reordering results, thereby completing the overall iterative update of the global cross-request meta-state.

[0063] In this embodiment, three key information types—micro-request output generation progress, decoding iteration stage, and cache slot mapping—are linked and updated to the global cross-request meta-state. This achieves real-time alignment of the inference output state, decoding progress state, and underlying cache resource mapping state, avoiding inconsistencies between the business inference state and the underlying cache mapping. It provides accurate and complete global meta-information for the stable construction of batches for the next inference step, ensuring the correct execution of the batch scheduling logic for hybrid pre-filling and multi-stage decoding micro-requests. This reduces scheduling anomalies and cache read / write errors caused by state misalignment, and supports stable and continuous operation of multi-round cyclic autoregressive inference.

[0064] In a specific embodiment, such as Figure 5 The diagram illustrates the module composition of an edge-side autoregressive large model inference acceleration system. This diagram showcases the system architecture for overlapping and packaged inference of large models under continuous requests from a single user on the edge. The request input module receives external requests. For text generation models, external requests can include text prompts, system instructions, and historical context; for multimodal models, external requests can include images, speech, video frames, sensor inputs, or other modal features; for agent systems, external requests can include environmental observations, task descriptions, and tool call history. The request input module is also responsible for embedding, encoding, or preprocessing the input. After completing the embedding, encoding, and preprocessing of text, multimodal inputs, and historical context, the request input module sends the processing results to the batch processing overlap scheduling module. The batch processing overlap scheduling module maintains the pipeline state and determines which micro-requests are included in each computation batch, i.e., selecting the current Prefill and historical Decode micro-requests. This module maintains at least the current request number, historical request stage, generation sequence accumulation buffer, number of tokens generated for each request, completion flag, and cache slot mapping. The scheduling module outputs the stage list for the current batch. The cross-request state construction module reads the input tokens or embeddings for each stage based on the stage list and generates... Simultaneously, this module generates segment description metadata, including the start and end positions of each segment, sequence length, request identifier, logical position, KV cache offset, and mask information. This module ensures that subsequent packaging calculations can be performed on a unified input. The packaging forward execution module is used to... This module, which feeds data into each layer of the large model, can call existing deep learning framework operators or custom fusion operators. Its core requirement is that each segment shares the same forward scheduling, but the attention context and output write-back remain independent. Specifically, it is used to uniformly complete the shared forward scheduling of each Transformer layer while maintaining the isolation of each request context. The packaged forward execution module is provided by the unified KV cache management module, which provides slot allocation, release, and advancement capabilities for the KV circular cache. The fusion operator execution module provides support for fused RoPE write, variable-length attention, and in-situ KV shift operators, used to execute fused RoPE write, variable-length attention, and in-situ KV shift. This module can be implemented using CUDA, Triton, HIP, OpenCL, Metal, NPU instructions, or other forms. For different hardware platforms, this module can use different kernels, but the interface remains consistent: inputting CRS, segment metadata, and KV cache description, and outputting the updated hidden state and cache state. The packaged inference results are sent to the output write-back module, which extracts the corresponding token prediction results from the output of each stage and updates the output sequence of historical requests. When a request reaches the output length K or meets the termination condition, the output write-back module returns the complete generated sequence or a submitable fragment to the upper-layer application. The output content can be text, code, structured instructions, tool call parameters, multimodal descriptions, or other discrete sequences generated by the autoregressive model. When generating tokens and writing back to update the output sequence, the generated output sequence and completion status are fed back to the batch processing overlap scheduling module. The pipeline status and metadata uniformly maintain all scheduling information such as request identifier, stage number, sequence and KV offset, and mask, forming a closed-loop pipeline inference system.

[0065] The technical solution of this embodiment obtains a continuous external request sequence and constructs a stable packaged computation batch at the first inference time based on the continuous external request sequence. The stable packaged computation batch includes pre-filled micro-requests corresponding to the external requests arriving at the first inference time and multiple decoded micro-requests. These multiple decoded micro-requests originate from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage. Segment description metadata corresponding to the stable packaged computation batch is constructed based on the global cross-request metadata. Based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch, and the multiple decoded micro-requests, a cross-request hidden state tensor is generated. The batch segment element tensor is determined based on the segment description metadata, and the cross-request hidden state tensor and the batch segment element tensor, which serves as auxiliary input, are then combined. A common input is used to perform batch inference using an autoregressive Transformer decoder, resulting in batch inference results. The autoregressive Transformer decoder performs variable-length attention computation and rotational position encoding write processing based on the batch segment tensor, directly writing the position-encoded keys and values ​​into a unified key-value circular cache. The output sequence of each micro-request is updated based on the batch inference results, resulting in an output update sequence. In-place shift processing of the key-value cache is performed, and the slots of the unified key-value circular cache are rearranged in stages based on the global cross-request meta-state, resulting in a slot rearrangement result. Based on the batch inference results, the output update sequence, and the slot rearrangement result, the global cross-request meta-state is updated, and a stable batch computation for the next inference time step is iteratively constructed. This inference process is repeated until all external requests are generated. This solution enables hybrid batching of pre-filled micro-requests for newly arriving requests with micro-requests from multiple historical requests and multiple decoding iteration stages. It fully reuses computational resources, leveraging a global cross-request meta-state to uniformly maintain the metadata of all micro-requests. Metadata tensors for batch boundaries, offsets, and cache index information are transformed using segment description metadata and batch segment meta-tensors. Multi-task hidden features are aggregated using cross-request hidden state tensors, eliminating the need for extensive padding. Variable-length attention computation avoids unnecessary computational consumption. Key-value pairs processed by rotational position encoding are directly written to a unified key-value circular cache, reducing intermediate tensor copies. Cache fragmentation is alleviated through key-value cache original shift and slot-stage reordering. Synchronous updates to the global cross-request meta-state ensure consistency between the inference business state and the underlying cache mapping state. A closed-loop iterative scheduling mode continuously processes a constant stream of external requests, suppressing drastic batch size fluctuations, reducing memory usage and data transport overhead, significantly improving the throughput and long-term continuous operation stability of autoregressive Transformer edge-side batch inference, and enhancing the efficiency of edge-side autoregressive large model inference acceleration.

[0066] Example 2 Figure 6This is a flowchart of an edge-side autoregressive large model inference acceleration method provided in Embodiment 2 of the present invention. The method in this embodiment is a further optimization of the method in the above embodiments. Optionally, it involves obtaining the global cross-request metastate, extracting information related to the stable packaged computation batch from the global cross-request metastate, and constructing segment description metadata corresponding to the stable packaged computation batch; obtaining the input lexical sequences of pre-filled micro-requests and multiple decoded micro-requests within the stable packaged computation batch, and obtaining the latent features corresponding to each micro-request through embedding processing; and aggregating the latent features corresponding to each micro-request based on the micro-request boundary information and offset information in the segment description metadata to generate a cross-request latent state tensor. For example... Figure 6 As shown, the method includes: S610. Obtain a continuous external request sequence and construct a stable packaged computation batch for the first inference time based on the continuous external request sequence; wherein, the stable packaged computation batch includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and multiple decoded micro-requests, the multiple decoded micro-requests are respectively from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage.

[0067] S620. Obtain the global cross-request meta state, extract information related to the stable packaging calculation batch from the global cross-request meta state, and construct the segment description metadata corresponding to the stable packaging calculation batch.

[0068] Specifically, the system reads and maintains the global cross-request metadata status, and extracts the associated information such as request identifier, micro-request type, sequence offset of each micro-request, effective sequence length, decoding iteration stage, and cache slot index corresponding to all micro-requests in the current stable packaged computing batch. Based on the extracted information, the system performs structured assembly to construct the segment description metadata corresponding to the current stable packaged computing batch.

[0069] In this embodiment, batch-related metadata is directly extracted from the globally maintained cross-request metadata state, and the full state of each micro-request that has already been maintained is reused. There is no need to repeatedly traverse the original external request sequence for re-parsing and statistics, which reduces the computational overhead of metadata collection. The segment description metadata only focuses on the micro-request set of the current batch to complete the information encapsulation, realizing the efficient conversion of the global metadata state to the local metadata of the batch. This provides accurate segment boundaries and cache index basis for subsequent generation of batch segment metadata tensors, execution of variable-length attention calculation, and targeted reading and writing of unified key-value ring cache. It ensures the consistency of metadata in the mixed batching scenario of pre-filled micro-requests and multi-stage decoded micro-requests, and reduces the risk of information mismatch.

[0070] S630: Obtain the input lexical sequences of pre-filled micro-requests and multiple decoded micro-requests within a stable batch of computational data, and obtain the latent features corresponding to each micro-request through embedding processing.

[0071] The latent features corresponding to each micro-request are high-dimensional semantic vectors obtained by mapping the discrete input word sequence of the micro-request through the embedding layer. They are the basic feature representations that can be fed into the Transformer decoder to participate in subsequent projection and attention operations. The method for determining them is to read the input word sequence of the corresponding micro-request within the stable packaged calculation batch, send the discrete word index into the word embedding layer to complete the vector mapping, and output the vector set corresponding to all words of the micro-request, thus obtaining the latent features corresponding to the micro-request.

[0072] Specifically, the pre-filled micro-requests within the stable packaged computation batch and the input word sequences corresponding to each of the multiple decoding micro-requests are obtained. Embedding processing is performed on the input word sequence of each micro-request to map the discrete words into high-dimensional vectors, thereby obtaining the latent features corresponding to each micro-request.

[0073] In this embodiment, the original input lexical sequences of different types of micro-requests within a batch are read separately. Through embedding processing, the lexical sequences are transformed from discrete symbols into model-computable latent feature vectors. This provides basic vector data for subsequent concatenation to generate cross-request latent state tensors. It can adapt to the characteristics of different lengths of input lexical sequences of pre-filled micro-requests within the same batch and decoded micro-requests at different decoding iteration stages. It preserves the independent semantic information of each micro-request and provides reliable input feature support for the subsequent QKV projection operation of the Transformer decoder, ensuring the correctness of feature extraction in the hybrid batch mode.

[0074] S640. Based on the micro-request boundary information and offset information in the segment description metadata, the latent features corresponding to each micro-request are aggregated to generate a cross-request latent state tensor.

[0075] Specifically, the micro-request boundary and offset information carried within the segment description metadata is read. Based on the start and end offset positions of each micro-request in the tensor space, the latent features corresponding to the pre-filled and decoded micro-requests within the stable batch are arranged to their respective offset positions to complete feature aggregation. These features are then concatenated and assembled to generate the cross-request latent state tensor (CRS). The expression for determining the cross-request latent state tensor CRS is as follows: ; Here, `Aggregate` can be performed by concatenating along the sequence dimension or by concatenating along the batch dimension and then expanding it within the operator. Preferably, to obtain a larger matrix multiplication size and to cooperate with the variable-length attention operator, this invention uses aggregation along the sequence dimension. The total query length after aggregation is... If multiple token blocks are used in certain decoding stages, the total length will be expanded accordingly. Add the sum of the lengths of all Decode blocks.

[0076] Optionally, the latent features corresponding to each micro-request can be aggregated using a sequence-dimensional aggregation approach. In this preferred implementation of sequence-dimensional aggregation, the construction of CRS can be further expressed as follows: ; in, The Prefill hidden state corresponds to the current request. to Corresponding to the hidden state of different historical requests, Concat means tensor concatenation while preserving segment metadata.

[0077] If used The length of the Prefill segment is indicated by... If the length of the s-th Decode segment is given, then the length of the packaged query satisfies the following condition. When each Decode stage uses a single token, This relationship is used to determine the number of matrix rows in the packing forward and the segment boundaries of the variable-length attention.

[0078] For any segment s in CRS, variable-length attention can be abstracted as: ; in, This indicates the original request to which segment s belongs. This indicates the length of the accessible KV history segment. This represents the attention mask for that segment; thus ensuring that different requests within the same batch only access their respective contexts.

[0079] In this embodiment, the regularization and aggregation of latent features of multiple micro-requests are completed based on the boundary and offset information provided by the segment description metadata. There is no need to perform redundant padding and zero-padding operations on latent features of varying lengths, reducing redundant and invalid feature data. Independent latent features belonging to different requests and different task types are integrated into a complete tensor that can be batch-input into the decoder. This achieves unified organization of tensor-based features of multiple tasks, avoiding the overhead of multiple tensor scheduling and frequent data copying caused by multiple independent small tensors. At the same time, the explicit boundary offset constraints ensure that the position of each latent feature of a micro-request is accurate within the tensor, providing regular and effective tensor input for subsequent variable-length attention calculations combined with batch segment metadata tensors, and adapting to the inference mode of mixed batching of pre-filled multi-stage decoding micro-requests.

[0080] S650. Determine the batch segment meta tensor based on the segment description metadata. Input the cross-request hidden state tensor and the batch segment meta tensor as auxiliary input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The autoregressive Transformer decoder performs variable-length attention calculation and rotation position encoding write processing based on the batch segment meta tensor, and directly writes the position-encoded keys and values ​​into a unified key-value circular cache.

[0081] S660. Update the output sequence of each micro-request based on the batch inference results to obtain the output update sequence; perform key-value cache original shift processing, and perform phased advance rearrangement of the slots of the unified key-value ring cache according to the global cross-request meta state to obtain the slot rearrangement result.

[0082] S670: Based on the batch inference results, output update sequence and slot rearrangement results, update the global cross-request meta state, iteratively construct the stable packaged calculation batch for the next inference time, and repeatedly execute inference processing until all external requests are generated.

[0083] The technical solution of this embodiment obtains a continuous sequence of external requests, constructs a stable packaged computation batch at the first inference time based on the continuous sequence of external requests, and includes both pre-filled micro-requests corresponding to the external requests arriving at the first inference time and multiple decoding micro-requests from historical external requests at different request times, each in a different decoding iteration stage. Then, the global cross-request meta-state is obtained, and information related to the stable packaged computation batch is extracted from it to construct the segment description metadata corresponding to the stable packaged computation batch. Next, the input lexical sequences of the pre-filled micro-requests and multiple decoding micro-requests within the batch are obtained, and the latent features corresponding to each micro-request are obtained through embedding processing. The latent features of each micro-request are aggregated based on the micro-request boundary information and offset information in the segment description metadata to generate a cross-request latent state tensor. Finally, based on the segment description metadata, the latent features are determined... A batch segment tensor is used. The cross-request hidden state tensor and the batch segment tensor, which are used as auxiliary inputs, are fed into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The decoder performs variable-length attention calculation and rotation position encoding writing based on the batch segment tensor, and writes the position-encoded keys and values ​​directly into a unified key-value circular cache. Then, the output sequence of each micro-request is updated based on the batch inference result to obtain the output update sequence. The original key-value cache is shifted, and the slots of the unified key-value circular cache are rearranged in stages based on the global cross-request meta state to obtain the slot rearrangement result. The global cross-request meta state is updated again using the batch inference result, the output update sequence, and the slot rearrangement result. The stable batch calculation batch for the next inference time is constructed iteratively, and the entire inference process is run in a loop until all external requests have completed lexical generation. This solution enables hybrid batching of pre-filled micro-requests for newly arriving requests with micro-requests from multiple historical requests and multiple decoding iteration stages. It fully reuses computing resources, uses a global cross-request meta-state to uniformly maintain the metadata of all micro-requests, and completes the metadata tensor transformation of batch boundaries, offsets, and cache index information through segment description metadata and batch segment meta-tensors. It relies on cross-request hidden state tensors to achieve integrated aggregation of hidden features of multiple tasks without the need for large amounts of padding. Combined with variable-length attention computation, it avoids unnecessary computing power consumption. The KV after rotation position encoding is directly written to a unified key-value circular cache, reducing intermediate tensor copying. Then, the key-value cache original shift and slot stage advance rearrangement alleviate cache fragmentation. Synchronous linkage updates of the global cross-request meta-state ensure the consistency between the inference business state and the underlying cache mapping state. With a closed-loop iterative scheduling mode, it continuously processes the continuous stream of external requests, which can suppress drastic fluctuations in batch size, reduce memory usage and data transfer overhead, and significantly improve the throughput and long-term continuous operation stability of autoregressive Transformer edge-side batch inference.

[0084] Example 3 Figure 7This is a schematic diagram of the structure of an end-side autoregressive large model inference acceleration device provided in Embodiment 3 of the present invention. Figure 7 As shown, the device includes: The stable packaged computation batch determination module 710 is used to obtain a continuous external request sequence and construct a stable packaged computation batch at the first inference time based on the continuous external request sequence. The stable packaged computation batch includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and multiple decoded micro-requests. The multiple decoded micro-requests come from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage. The cross-request hidden state tensor generation module 720 is used to construct segment description metadata corresponding to the stable packaged computation batch based on the global cross-request metadata state, and generate cross-request hidden state tensors based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch and multiple decoded micro-requests. The batch inference result determination module 730 is used to determine the batch segment meta tensor based on the segment description metadata. The cross-request hidden state tensor and the batch segment meta tensor as auxiliary input are jointly input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The autoregressive Transformer decoder performs variable length attention calculation and rotation position encoding writing processing based on the batch segment meta tensor, and directly writes the position encoded key and value into a unified key-value ring cache. The output update sequence and slot rearrangement result determination module 740 is used to update the output sequence of each micro-request based on the batch inference result to obtain the output update sequence; perform key-value cache original shift processing, and perform stage-by-stage rearrangement of the slots of the unified key-value ring cache according to the global cross-request meta state to obtain the slot rearrangement result; The state update and iterative inference module 750 is used to update the global cross-request meta state based on the batch inference results, output update sequence and slot rearrangement results, iteratively construct the stable packaged calculation batch for the next inference time, and cyclically execute inference processing until all external requests are generated.

[0085] The technical solution of this embodiment obtains a continuous external request sequence through a stable batching calculation determination module, and constructs a stable batching calculation batch at the first inference time based on the continuous external request sequence. The stable batching calculation batch includes pre-filled micro-requests corresponding to the external requests arriving at the first inference time and multiple decoded micro-requests. These multiple decoded micro-requests originate from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage. A cross-request hidden state tensor generation module constructs segment description metadata corresponding to the stable batching calculation batch based on the global cross-request metadata. Based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable batching calculation batch, and the multiple decoded micro-requests, a cross-request hidden state tensor is generated. A batch inference result determination module determines the batch segment metadata tensor based on the segment description metadata, and combines the cross-request hidden state tensor with the batch segments as auxiliary input. The meta-tensors are input into the autoregressive Transformer decoder for batch inference, yielding batch inference results. The autoregressive Transformer decoder performs variable-length attention calculations and rotational position encoding write processing based on the batch segment meta-tensors, directly writing the position-encoded keys and values ​​into a unified key-value circular cache. The output update sequence and slot rearrangement result determination module updates the output sequence of each micro-request based on the batch inference results, obtaining the output update sequence. It then performs key-value cache original shift processing and performs phased advancement rearrangement of the slots in the unified key-value circular cache based on the global cross-request meta-state, obtaining the slot rearrangement result. The state update and iterative inference module updates the global cross-request meta-state based on the batch inference results, the output update sequence, and the slot rearrangement result, iteratively constructs a stable batch for the next inference time, and continuously executes inference processing until all external requests are generated. This solution enables hybrid batching of pre-filled micro-requests for newly arriving requests with micro-requests from multiple historical requests and multiple decoding iteration stages. It fully reuses computational resources, leveraging a global cross-request meta-state to uniformly maintain the metadata of all micro-requests. Metadata tensors for batch boundaries, offsets, and cache index information are transformed using segment description metadata and batch segment meta-tensors. Multi-task hidden features are aggregated using cross-request hidden state tensors, eliminating the need for extensive padding. Variable-length attention computation avoids unnecessary computational consumption. Key-value pairs processed by rotational position encoding are directly written to a unified key-value circular cache, reducing intermediate tensor copies. Cache fragmentation is alleviated through key-value cache original shift and slot-stage reordering. Synchronous updates to the global cross-request meta-state ensure consistency between the inference business state and the underlying cache mapping state. A closed-loop iterative scheduling mode continuously processes a constant stream of external requests, suppressing drastic batch size fluctuations, reducing memory usage and data transport overhead, significantly improving the throughput and long-term continuous operation stability of autoregressive Transformer edge-side batch inference, and enhancing the efficiency of edge-side autoregressive large model inference acceleration.

[0086] Based on the above embodiments, optionally, a stable packaging computation batch determination module 710 is used to perform task parsing on a continuous external request sequence, distinguishing newly arrived external requests to be pre-filled and historical external requests in different decoding stages; converting newly arrived external requests to be pre-filled into first pre-filled micro-requests, and mapping historical external requests in different decoding stages to corresponding first decoding micro-requests; using the available computing power resources and available key-value cache resources of the end-side device as constraints, selecting second pre-filled micro-requests and second decoding micro-requests to be added to the current batch from the obtained first pre-filled micro-requests and first decoding micro-requests, and forming a micro-request combination based on the second pre-filled micro-requests and second decoding micro-requests; when the micro-request combination meets the preset batch stability conditions, constructing a stable packaging computation batch at the first inference time based on the micro-request combination.

[0087] Optionally, the cross-request hidden state tensor generation module 720 is specifically used to obtain the global cross-request meta-state, extract information related to the stable packaged computation batch from the global cross-request meta-state, construct the segment description metadata corresponding to the stable packaged computation batch; obtain the input lexical sequences of the pre-filled micro-requests and multiple decoded micro-requests within the stable packaged computation batch, and obtain the hidden features corresponding to each micro-request through embedding processing; and aggregate the hidden features corresponding to each micro-request based on the micro-request boundary information and offset information in the segment description metadata to generate the cross-request hidden state tensor.

[0088] Optionally, the batch inference result determination module 730 is specifically used to perform tensor structure transformation on the micro-request boundaries, offset indices, and cache slot information in the segment description metadata, generating a batch segment meta tensor carrying the position of each micro-request boundary, the effective sequence length, and the cache slot index; inputting the cross-request hidden state tensor and the batch segment meta tensor together into the autoregressive Transformer decoder, so that the decoder performs QKV projection based on the input hidden state in the cross-request hidden state tensor to obtain the query tensor, key tensor, and value tensor; the decoder performs rotation position encoding transformation only on the query tensor and key tensor based on the logical position of each micro-request term in the batch segment meta tensor, and directly writes the transformed key tensor and the value tensor without position encoding transformation into the corresponding slot of the unified key-value circular cache; the decoder performs variable-length attention calculation based on the micro-request boundary position and the effective sequence length of the batch segment meta tensor, and obtains the final batch inference result through preset subsequent network operations.

[0089] Optionally, the output update sequence and slot rearrangement result determination module 740 is specifically used to parse the predicted tokens corresponding to each micro-request from the batch inference results, add the predicted tokens corresponding to each micro-request to the corresponding original output sequence to obtain the output update sequence corresponding to each micro-request; perform key-value cache original shift processing on the key-value data stored in each slot of the unified key-value circular cache, so that the key-value data completes the sequence dimension shift within its slot; read the decoding iteration stage information and cache slot binding relationship of each micro-request in the global cross-request meta state, and perform stage advancement rearrangement on the slots of the unified key-value circular cache according to the iteration stage progress of each micro-request after inference, to obtain the slot rearrangement result; wherein, the slot rearrangement result is used to update the binding mapping between each micro-request and the unified circular cache slot.

[0090] Optionally, the state update and iterative inference module 350 is specifically used to update the output content and generation progress information of each micro-request in the global cross-request meta-state based on the output update sequence; update the current decoding iteration stage of each micro-request in the global cross-request meta-state based on the batch inference results; and synchronously update the binding mapping relationship between each micro-request in the global cross-request meta-state and the unified key-value ring cache slot based on the slot rearrangement results, thereby completing the iterative update of the global cross-request meta-state.

[0091] The end-side autoregressive large model inference acceleration device provided in the embodiments of the present invention can execute the end-side autoregressive large model inference acceleration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0092] Example 4 Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0093] like Figure 8As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0094] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0095] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as edge-side autoregressive large model inference acceleration methods.

[0096] In some embodiments, the edge-side autoregressive large model inference acceleration method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the edge-side autoregressive large model inference acceleration method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the edge-side autoregressive large model inference acceleration method by any other suitable means (e.g., by means of firmware).

[0097] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] Computer programs used to implement the end-side autoregressive large model inference acceleration method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0099] Example 5 Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute an edge-side autoregressive large model inference acceleration method, the method comprising: Obtain a continuous external request sequence and construct a stable packaged computation batch at the first inference time based on the continuous external request sequence; wherein, the stable packaged computation batch includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and multiple decoded micro-requests, the multiple decoded micro-requests come from historical external requests at different request times, and each decoded micro-request is in a different decoding iteration stage. Based on the global cross-request metadata state, construct the segment description metadata corresponding to the stable packaged computation batch. Based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch, and multiple decoded micro-requests, generate the cross-request hidden state tensor. Based on the segment description metadata, the batch segment meta tensor is determined. The cross-request hidden state tensor and the batch segment meta tensor as auxiliary input are jointly input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The autoregressive Transformer decoder performs variable length attention calculation and rotation position encoding writing processing based on the batch segment meta tensor, and directly writes the position encoded key and value into a unified key-value ring cache. The output sequence of each micro-request is updated based on the batch inference results to obtain the output update sequence; the original key-value cache is shifted, and the slots of the unified key-value circular cache are rearranged in stages according to the global cross-request meta state to obtain the slot rearrangement result; Based on the batch inference results, output update sequence, and slot rearrangement results, update the global cross-request meta state, iteratively construct the stable packaged calculation batch for the next inference time, and repeatedly execute inference processing until all external requests are generated.

[0100] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0101] To provide interaction with an object, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the object; and a keyboard and pointing device (e.g., a mouse or trackball) through which the object provides input to the electronic device. Other types of devices can also be used to provide interaction with the object; for example, feedback provided to the object can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the object can be received in any form (including sound input, voice input, or tactile input).

[0102] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., a computer with a graphical user interface or web browser through which an item can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0103] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0104] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0105] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements an edge-side autoregressive large model inference acceleration method according to any embodiment of the invention.

[0106] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for accelerating inference in large end-side autoregressive models, characterized in that, include: Obtain a continuous external request sequence, and construct a stable packaged computation batch at the first inference time based on the continuous external request sequence; wherein, the stable packaged computation batch includes a pre-filled micro-request corresponding to the external request arriving at the first inference time and a plurality of decoded micro-requests, wherein the plurality of decoded micro-requests are respectively from historical external requests at different request times, and each of the decoded micro-requests is in a different decoding iteration stage; Based on the global cross-request metadata state, the segment description metadata corresponding to the stable packaged computation batch is constructed. Based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch, and multiple decoded micro-requests, a cross-request hidden state tensor is generated. Based on the segment description metadata, the batch segment meta tensor is determined. The cross-request hidden state tensor and the batch segment meta tensor as auxiliary input are jointly input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result. The autoregressive Transformer decoder performs variable-length attention calculation and rotation position encoding writing processing based on the batch segment meta tensor, and directly writes the position-encoded keys and values ​​into a unified key-value circular cache. The output sequence of each micro-request is updated based on the batch inference result to obtain the output update sequence; the original key-value cache is shifted, and the slots of the unified key-value circular cache are rearranged in stages according to the global cross-request meta state to obtain the slot rearrangement result; Based on the batch inference results, the output update sequence, and the slot rearrangement results, the global cross-request meta state is updated, a stable packaged calculation batch for the next inference time is iteratively constructed, and inference processing is performed repeatedly until all external requests are generated.

2. The method according to claim 1, characterized in that, The step of obtaining a continuous external request sequence and constructing a stable packaged computation batch for the first inference time based on the continuous external request sequence includes: The continuous external request sequence is parsed to distinguish newly arrived external requests to be pre-filled and historical external requests at different decoding stages. The newly arrived external request to be prefilled is converted into a first prefilled micro-request, and historical external requests at different decoding stages are mapped to the corresponding first decoding micro-requests; Using the available computing power resources and available key-value cache resources of the edge device as constraints, the second pre-filled micro-request and the second decoding micro-request to be added to the current batch are selected from the obtained first pre-filled micro-request and first decoding micro-request, and a micro-request combination is formed based on the second pre-filled micro-request and the second decoding micro-request; When the micro-request combination meets the preset batch stability condition, a stable packaged computation batch at the first inference time is constructed based on the micro-request combination.

3. The method according to claim 1, characterized in that, The step of constructing segment description metadata corresponding to the stable packaged computation batch based on the global cross-request metadata state, and generating a cross-request hidden state tensor based on the micro-request boundary information of the segment description metadata, the pre-filled micro-requests in the stable packaged computation batch, and multiple decoded micro-requests, includes: Obtain the global cross-request meta state, extract information related to the stable packaging calculation batch from the global cross-request meta state, and construct the segment description metadata corresponding to the stable packaging calculation batch; The input lexical sequences of pre-filled micro-requests and multiple decoded micro-requests within the stable packaging calculation batch are obtained, and the latent features corresponding to each micro-request are obtained through embedding processing. Based on the micro-request boundary information and offset information in the segment description metadata, the latent features corresponding to each micro-request are aggregated to generate the cross-request latent state tensor.

4. The method according to claim 1, characterized in that, The batch segment meta tensor is determined based on the segment description metadata. The cross-request hidden state tensor and the batch segment meta tensor, which serves as auxiliary input, are jointly input into the autoregressive Transformer decoder for batch inference to obtain the batch inference result, including: The micro-request boundaries, offset indices, and cache slot information in the segment description metadata are transformed into tensors to generate batch segment metadata tensors carrying the position of each micro-request boundary, the effective sequence length, and the cache slot index. The cross-request hidden state tensor and the batch segment tensor are input together into the autoregressive Transformer decoder, so that the decoder performs QKV projection based on the input hidden state in the cross-request hidden state tensor to obtain the query tensor, key tensor and value tensor. The decoder performs rotation position encoding transformation only on the query tensor and the key tensor based on the logical position of each micro-request term in the batch segment tensor, and directly writes the transformed key tensor and the value tensor without position encoding transformation into the corresponding slot of the unified key-value circular cache. The decoder performs variable-length attention computation based on the micro-request boundary position and effective sequence length of the batch segment tensor, and obtains the final batch inference result through preset subsequent network operations.

5. The method according to claim 4, characterized in that, The preset subsequent network operation processing includes cache maintenance operation and feature transformation operation; the feature transformation operation includes multi-head attention output projection operation, residual connection operation and multilayer perceptron feedforward operation.

6. The method according to claim 1, characterized in that, The output sequence of each micro-request is updated based on the batch inference result to obtain the output update sequence; the original key-value cache is shifted, and the slots of the unified key-value circular cache are rearranged in stages according to the global cross-request meta state to obtain the slot rearrangement result, including: The predicted tokens corresponding to each micro-request are parsed from the batch inference results, and the predicted tokens corresponding to each micro-request are added to the corresponding original output sequence to obtain the output update sequence corresponding to each micro-request. Perform key-value cache original shift processing on the key-value data stored in each slot of the unified key-value circular cache, so that the key-value data completes the sequence dimension shift within its slot; Read the decoding iteration stage information and cache slot binding relationship of each micro-request in the global cross-request meta state. Based on the iteration stage progress after inference of each micro-request, perform stage progress rearrangement on the slots of the unified key-value ring cache to obtain the slot rearrangement result. The slot rearrangement result is used to update the binding mapping between each micro-request and the unified ring cache slot.

7. The method according to claim 1, characterized in that, The step of updating the global cross-request meta state based on the batch inference results, the output update sequence, and the slot rearrangement results includes: Based on the output update sequence, update the output content and generation progress information of each micro-request in the global cross-request meta-state; The current decoding iteration stage of each micro-request in the global cross-request meta-state is updated based on the batch inference results; Based on the slot rearrangement result, the binding mapping relationship between each micro-request and the unified key-value ring cache slot in the global cross-request meta state is updated synchronously to complete the iterative update of the global cross-request meta state.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the end-side autoregressive large model inference acceleration method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the edge-side autoregressive large model inference acceleration method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the end-side autoregressive large model inference acceleration method according to any one of claims 1-7.

Images