Method and apparatus for observing request concurrency behavior of large model inference engines

Abstract

The embodiment of the specification provides a method and device for observing request concurrency behavior of a large model inference engine. The method comprises the following steps: in the process that the inference engine concurrently processes a plurality of inference requests, obtaining, for each inference request, performance data generated by the inference engine in an inference process, the performance data comprising time sequence performance data of a token corresponding to the inference request; and analyzing the performance data of the plurality of inference requests to obtain observation indexes corresponding to each processing stage in the inference process of different inference requests. The method can observe request concurrency behavior of the large model inference engine.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of large model technology, specifically to a method and apparatus for observing the request and issuance behavior of a large model inference engine. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, especially the widespread application of large language models (LMs), large model inference engines, as the key infrastructure for carrying online model inference services, are widely deployed in production environments that require high concurrency, low latency, and high throughput to handle inference requests from a large number of users or upstream services simultaneously.

[0003] Due to the large parameter scale, computational intensity, and high memory / GPU memory consumption of large models, inference engines typically involve complex resource scheduling and sharing mechanisms when processing requests, such as compute unit scheduling, GPU memory management, compute pipeline coordination, batch processing, and cache reuse. In multi-request concurrency scenarios, multiple inference requests often share the same set of hardware and software resources, such as compute units, GPUs (Graphics Processing Units), host memory, network bandwidth, and caches, thus forming complex competition and coupling relationships in time and space. The aforementioned concurrent interactions may lead to problems such as memory swapping in and out due to GPU memory contention, long-tail latency due to improper scheduling strategies, and increased response latency for critical requests due to mismatch between batch processing strategies and request priorities. At the same time, optimization mechanisms such as dynamic batch processing, continuous batch processing, key-value (KV) cache sharing, and asynchronous execution further increase the nonlinearity and unpredictability of system behavior, making the localization and analysis of concurrency problems more difficult.

[0004] Existing performance monitoring and debugging tools mostly focus on performance profiling at the single request or operator level, or on statistical analysis of hardware resource utilization. They lack the ability to uniformly observe, correlate, and visualize concurrent behavior across the lifecycle of multiple requests at the overall inference engine level, making it difficult to accurately depict the concurrent overlap between requests. Therefore, when issues such as sudden increases in inference latency, decreased throughput, or abnormal resource contention occur, operations personnel or system developers often need to rely on experience to troubleshoot layer by layer, resulting in low efficiency in locating the root cause and difficulty in quickly identifying it.

[0005] Therefore, there is an urgent need for a method that can observe the request and issuance behavior of large model inference engines in order to improve the observability and problem diagnosis efficiency of large model inference engines in concurrent scenarios. Summary of the Invention

[0006] This specification provides a scheme for observing the request and issuance behavior of a large model inference engine, which can improve the observability of the large model inference engine in concurrent scenarios.

[0007] In a first aspect, embodiments of this specification provide a method for observing the concurrent request behavior of a large model inference engine, comprising: during the concurrent processing of multiple inference requests by the inference engine, acquiring performance data generated by the inference engine during the inference process for each inference request, wherein the performance data includes time-series performance data of the keyword token corresponding to the inference request; and analyzing the performance data of the multiple inference requests to obtain observation indicators corresponding to each processing stage of the different inference requests during the inference process.

[0008] In one optional implementation, the timing performance data of the key token includes at least one of the following: the key token's in iteration time, the key token's out iteration time, the key token's execution time, the key token's scheduling delay, the key token's total execution time, and the key token's preprocessing time and queuing time.

[0009] In one optional implementation, the performance data further includes at least one of the following: the time when the inference request enters the inference engine, the time when the inference engine completes processing the inference request, the number of concurrent inference requests in the execution batch, and the number of concurrent tokens in the execution batch.

[0010] In one alternative implementation, the key token includes any one of the following: a first token generated by the inference engine based on the inference request, or a second token generated by the inference engine based on the inference request under a pre-filled decode separation architecture.

[0011] In one optional implementation, the processing stages include a pre-filling stage and a decoding stage; the pre-filling stage includes the following sub-stages: inference request preprocessing, queuing, and generation of the first token; the decoding stage includes the following sub-stages: queuing and subsequent token generation.

[0012] In one optional implementation, the step of obtaining performance data generated by the inference engine during the inference process for each inference request while the inference engine concurrently processes multiple inference requests includes: setting multiple data points in the code of the inference engine, the multiple data points corresponding to multiple processing stages in the inference process of the inference request; and obtaining performance data generated by the inference engine during the inference process for each inference request through the multiple data points while the inference engine concurrently processes multiple inference requests.

[0013] In an optional implementation, after obtaining the performance data generated by the inference engine during the inference process through the plurality of tracking points for each inference request, the method further includes: when the inference request is processed, attaching the performance data as an attribute to the tracking record unit of the inference request, and reporting the tracking record unit to the storage system.

[0014] In one optional implementation, the step of analyzing the performance data of multiple inference requests to obtain the observation metrics corresponding to each processing stage of the inference request includes: in response to receiving a user query instruction, determining the performance data to be analyzed based on the time range specified by the user query instruction and the inference engine identifier, and analyzing the performance data to be analyzed to obtain the observation metrics corresponding to each processing stage of the inference request.

[0015] In an optional implementation, after analyzing the performance data of the multiple inference requests to obtain the observation metrics corresponding to each processing stage of the inference request during the inference process, the method further includes: generating a multi-track waterfall chart based on the observation metrics, wherein each track in the multi-track waterfall chart is used to display different processes in the inference engine, and each track displays the observation metrics corresponding to at least one processing stage of several inference requests in chronological order.

[0016] In one alternative implementation, each processing stage consists of one or more color blocks, the length of which represents the execution time of the corresponding processing stage, and the color blocks of different processing stages are distinguished by different colors.

[0017] In one optional implementation, multiple tracks corresponding to the same inference request in the multi-track waterfall graph are displayed together using connection identifiers to indicate that the multiple tracks belong to the same inference request.

[0018] In an optional implementation, the method further includes: in response to receiving a selection instruction for any of the orbits, displaying a detailed graph of the observation indicators of the processing stage corresponding to the orbit, the processing stage including multiple sub-stages, and the detailed graph including an identifier for each sub-stage, a corresponding color block color, and an execution duration.

[0019] Secondly, embodiments of this specification provide an apparatus for observing the request and concurrent behavior of a large model inference engine, comprising: a data acquisition module, configured to acquire performance data generated by the inference engine during the inference process for each inference request during the concurrent processing of multiple inference requests by the inference engine, the performance data including time-series performance data of the key token corresponding to the inference request; and a data analysis module, configured to analyze the performance data of the multiple inference requests to obtain observation indicators corresponding to each processing stage of the inference request during the inference process.

[0020] Thirdly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the implementations in the first aspect.

[0021] In the above-described solution provided in this specification, during the concurrent processing of multiple inference requests by the inference engine, the performance data of each inference request during the inference process is collected in fine detail, including the time-series performance data of key tokens. Based on this, cross-request correlation and collaborative analysis is performed on the performance data of multiple inference requests to generate observation indicators that can reflect the concurrent interaction relationship between requests. This enables the observation of the concurrent behavior of the large model inference engine requests, facilitating the rapid location of the root cause of the problem during subsequent problem diagnosis. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a method for observing the request and issuance behavior of a large model inference engine, as described in the embodiments of this specification.

[0024] Figure 2 This is a schematic diagram of the visualization of one observation indicator in the embodiments of this specification;

[0025] Figure 3 This is a schematic diagram of the visualization of one observation indicator in the embodiments of this specification;

[0026] Figure 4 This is a schematic diagram of the visualization of one observation indicator in the embodiments of this specification;

[0027] Figure 5 This is a schematic diagram of the visualization of one observation indicator in the embodiments of this specification;

[0028] Figure 6 This is a schematic diagram of the visualization of one observation indicator in the embodiments of this specification;

[0029] Figure 7 This is a schematic diagram of the visualization of one observation indicator in the embodiments of this specification;

[0030] Figure 8 This is a schematic diagram of the device used to observe the request and issuance behavior of a large model inference engine in the embodiments of this specification. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0032] To facilitate understanding of the solutions in this specification, the following is a brief explanation of the concepts of some terms used in this specification.

[0033] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, enabling them to generalize. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0034] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0035] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025), and will not be repeated here.

[0036] Large models have a massive number of parameters, reaching hundreds or even trillions, leading to extremely high inference overhead. To accommodate such a large number of parameters, high-end GPUs with large amounts of video memory, or multiple high-end GPUs, are required, resulting in very high equipment purchase and maintenance costs. Furthermore, due to LLM's unique autoregressive generation mechanism and key-value pair caching mechanism, GPU computational efficiency is relatively low during LLM inference. To address this, the industry has proposed using large model inference engines (hereinafter referred to as inference engines) to optimize the model inference process. This allows large models to handle more inference requests initiated by service-side applications with the same computing resources, thus reducing the average cost per request.

[0037] An inference engine is a core software component or system specifically designed for loading and running large-scale model inference computations. It is responsible for feeding user inference requests (such as prompts) into a trained model, generating output results (such as text responses, images, etc.) through forward propagation computation, and returning them to the user. Specifically, the upper-layer application initiates an inference request to the inference engine. The large-scale model inference engine schedules the large model to process the inference request. For example, it uses the prompts in the inference request as input to the large model and then uses the output of the large model as the response data or processing result of the inference request, feeding it back to the upper-layer application. When handling multiple concurrent inference requests, the inference engine typically involves complex resource scheduling and sharing mechanisms. For example, dynamic batch processing, continuous batch processing, KV cache sharing, and asynchronous execution scheduling techniques can be used to improve the inference speed and throughput of LLM models. This specification does not restrict the inference engine used; for example, it can be an SGLang, vLLM, TRT, or other inference engines.

[0038] Concurrency, in an inference engine, refers to the system state where multiple inference requests share the same set of hardware resources (such as XPU compute units, video memory, host memory, KV cache blocks, etc.) and execute in an interleaved manner across time and space. These inference requests can generate complex resource contention relationships, potentially leading to mutual interference, performance disruption, or even request aborts.

[0039] Concurrency analysis is the analysis of the mutual influence relationships when multiple inference requests are executed simultaneously in an inference engine.

[0040] Large-scale model inference typically consists of two phases: Prefill and Decode. The Prefill phase is the initial computation stage for handling inference requests. In this phase, the model parses and preprocesses the entire inference request at once, generating a corresponding key-value cache. This stage is computationally intensive and usually requires significant GPU memory resources. The generation of the first token typically occurs during the Prefill phase. The Decode phase follows the inference process, responsible for generating subsequent tokens one by one. During this phase, the model iteratively predicts the next token based on the key-value cache generated in the Prefill phase, using an autoregressive generation model. The Prefill phase is typically resource-intensive and highly exclusive, which can easily lead to interruptions in the Decode phase.

[0041] As mentioned earlier, model inference engines need to handle a large number of concurrent inference requests. During this process, multiple inference requests share critical resources such as GPU memory, computing units, and key-value caches. Due to the complex scheduling dependencies and resource contention relationships among inference requests (such as memory swapping and dynamic batch processing conflicts), long-tail latency, reduced throughput, and even request abortion are common problems. Current mainstream performance analysis systems can only obtain macro-level indicators at the infrastructure level (such as GPU utilization and network latency), and cannot delve into the internal runtime state of the inference engine. It is difficult to observe the key timing relationships in the concurrent execution of multiple requests (such as the scheduling order of requests in the prefill and decode stages, key-value cache contention events, and resource preemption behavior), making it difficult to pinpoint the root cause of performance problems. For example, it may be known that a user's inference request response is slow, but it may not know which part of the inference request's execution is slow or what caused the slowdown.

[0042] Based on this, this specification proposes a method for observing the request concurrency behavior of a large model inference engine. The request concurrency behavior of a large model inference engine refers to its resource scheduling and inference behaviors when processing multiple concurrent inference requests. This method can obtain the performance data produced by the inference engine during the inference process for each inference request, and then analyze the performance data of multiple inference requests to obtain the observation indicators corresponding to each processing stage of different inference requests, thereby achieving systematic observation of the concurrent behavior of the inference engine.

[0043] The following is combined with Figure 1 This specification describes the methods provided for observing the request and issuance behavior of large model inference engines.

[0044] Figure 1This is a flowchart illustrating a method for observing the request and issuance behavior of a large model inference engine, as described in an embodiment of this specification. This method can be applied to any device, platform, or cluster of devices with computing and processing capabilities. It includes steps 101-102 as shown below.

[0045] In step 101, during the concurrent processing of multiple inference requests by the inference engine, the performance data generated by the inference engine during the inference process is obtained for each inference request.

[0046] In this step, data can be collected to obtain performance data generated by the inference engine during the concurrent processing of multiple inference requests. This performance data includes token-level performance data, such as the time-series performance data of the key token corresponding to the inference request, or counting information related to token generation, and may also include performance data at the inference request level.

[0047] In one optional implementation, the timing performance data of the key token includes at least one of the following: the key token's in iteration time, the key token's out iteration time, the key token's execution time, the key token's scheduling delay, the key token's total execution time, and the key token's preprocessing time and queuing time.

[0048] In one alternative implementation, the performance data further includes at least one of the following: the time when the inference request enters the inference engine, the time when the inference engine completes processing the inference request, the number of concurrent inference requests in the execution batch, and the number of concurrent tokens in the execution batch.

[0049] For example, the key token may include any of the following: the first token generated by the inference engine based on the inference request (usually generated in the prefill stage), or the second token generated by the inference engine based on the inference request under the prefill-decode separation architecture (i.e., PD separation architecture) (generated in the decode stage).

[0050] The PD separation architecture refers to a system architecture where the Prefill and Decode phases of large model inference are deployed on different computing nodes. The Prefill node is responsible for processing the prompts corresponding to the input inference requests, generating a key-value cache and the first token. The Decode node is responsible for generating output tokens based on the key-value caches, with the second token being the first token generated in the Decode phase. In this PD separation architecture, the key-value cache generated by the Prefill needs to be transferred from the Prefill node to the Decode node, i.e., key-value cache transfer.

[0051] In other embodiments, a key token can be specified according to actual needs. For example, the 10th generated token can be specified as the key token, or each generated token can be specified as the key token.

[0052] The performance data mentioned above will be explained below.

[0053] The time when an inference request enters the inference engine: the timestamp when the inference request arrives at the inference engine (such as an API gateway or scheduler).

[0054] The time when the inference engine completes the inference request: the timestamp when the entire inference request is completed (all output tokens are generated and returned to the user or upper-layer application).

[0055] The timing of an inference request entering the inference engine and the timing of the inference engine completing the processing of the inference request can be used in the subsequent analysis phase of observation metrics to calculate end-to-end latency, determine whether an inference request has timed out or is abnormal, and identify "long-tail requests" by comparing them with other inference requests.

[0056] The critical token's iteration point: This refers to the moment when the scheduler includes the critical token in the current computation batch, that is, the moment when the token is selected for execution and begins preparation for computation. This moment marks the token entering the inference engine's computation pipeline and beginning to wait for GPU resource allocation.

[0057] Understandably, during the inference engine's operation, the system performs iterative processing in a loop: in each iteration, the scheduler selects multiple inference requests from those en route to form an execution batch based on available computing resources and scheduling policies, and sends them to the GPU for one model computation. For example, during the decode phase, this one model computation typically advances each unfinished inference request within the batch by at least one generation step, thereby generating one or more output tokens; after the iteration is completed, the next iteration begins, and the above scheduling and execution process is repeated.

[0058] The exit iteration moment of a key token: This refers to the point in time when the key token completes the computation of the current iteration and returns the result from the GPU. In other words, it's the moment when the computation result for this token has been generated and is ready to enter the next processing stage. This moment marks the completion of the current iteration's processing of this token.

[0059] The entry and exit times of key tokens can be used to accurately measure the computation latency of each key token during the observation metric analysis phase, and to analyze issues such as GPU utilization fluctuations and scheduling jitter.

[0060] Execution time of key tokens: This refers to the actual computation time of key tokens on the GPU, that is, the time interval from when the key token enters the iteration (is sent to the GPU for computation) to when the computation is completed and the result is returned. For the first token, this mainly reflects the computation time of the Prefill phase; for tokens in the Decode phase, this mainly reflects the computation time of generating a single token. The execution time of key tokens reflects operator efficiency and kernel performance.

[0061] Key Token scheduling latency: refers to the time interval from when a key token is recognized as processable by the system to when it is selected by the scheduler into the execution batch, i.e. the waiting time in the scheduling queue, which can reflect the scheduler efficiency and resource contention (such as queuing for GPUs).

[0062] Total time spent on key tokens: This refers to the entire time span from when the inference request corresponding to the key token enters the system to when the token is generated. In other words, it's the total time from when the request enters the engine to when the key token is generated in an iteration. This is also the token generation interval perceived by the user.

[0063] Number of concurrent inference requests in a batch: This refers to the number of independent inference requests selected by the scheduler for parallel processing in the current iteration. It can be used to evaluate batch processing efficiency and analyze the problem of a slow request dragging down the entire batch.

[0064] The number of tokens concurrently executed in a batch: refers to the total number of tokens selected by the scheduler for parallel processing in the current iteration, including all input tokens for all requests and generated output tokens, reflecting the parallel computing load of the GPU.

[0065] The preprocessing time for a critical token refers to the time interval from when a request enters the system to when the critical token is ready to enter the scheduling queue. This includes necessary parsing and preparation operations, reflecting the initial overhead of the system processing the request. For the first token, the preprocessing time may include the time spent on prompt parsing, tokenization, and request enqueueing. For the second token in a PD-separated architecture, the preprocessing time may include the network latency of transferring the KV cache from the Prefill node to the Decode node, the time for the Decode node to receive the request and enqueue it, and the process of building request metadata.

[0066] Queuing time for critical tokens: This refers to the time a critical token waits in the scheduling queue to be selected for execution, i.e., the waiting time in the scheduling queue. Queuing time reflects the degree of competition for system resources; a long queuing time usually indicates resource scarcity at the current stage. For the first token, this includes the waiting time in the Prefill scheduling queue; for the second token in a PD-separated architecture, it can include the waiting time in the Decode scheduling queue, reflecting the cross-node coordination overhead under the PD architecture.

[0067] In large-model generative inference scenarios, the end-to-end latency of a request is closely related to the process of generating output tokens. For the same inference request, performance anomalies may manifest as an increase in the Time to First Token (TTFT) and / or an increase in the Time Per Output Token (TPOT). Under high concurrency conditions, multiple inference requests share resources such as computing units, GPU memory, and cache, which may lead to resource contention and mutual interference during the token generation phase, resulting in latency jitter or decreased throughput. Therefore, collecting performance data at the token granularity level can provide a basis for subsequent observation and analysis of request concurrency behavior and resource contention.

[0068] This embodiment does not limit the specific method for obtaining performance data. For example, a Python probe can be used to achieve non-intrusive data collection during inference, or an intrusive data collection can be achieved through functions.

[0069] Considering the significant performance overhead of non-intrusive data collection, an alternative implementation involves setting multiple tracking points within the inference engine code for intrusive data collection. These tracking points correspond to multiple processing stages in the inference process of each inference request. As the inference engine concurrently processes multiple inference requests, performance data generated by the engine during inference is obtained for each request through these tracking points. These tracking points are configured within the internal execution logic of the inference engine, enabling the acquisition of runtime context information and thus achieving token-level observation of the inference process. Furthermore, compared to general performance analysis tools that only show operator-level execution time or hardware utilization but lack understanding of the lifecycle semantics of large model inference (such as Prefill / Decode stage division, KV cache reuse, dynamic batch processing, etc.) and cannot perform correlation analysis of performance interference between different requests, deep tracking within the inference engine can also help to gain a deeper understanding of the inference semantics of LLM.

[0070] In this embodiment, the performance data collected by the internal embedding points of the inference engine is uniformly collected by the OpenTelemetryCollector and transmitted to the backend storage system via OTLP (OpenTelemetry Protocol).

[0071] In one embodiment, to reduce the overhead of data transmission and processing, a lightweight acquisition strategy can be adopted. Upon completion of an inference request, performance data can be attached as an attribute to the inference request's trace record unit (i.e., Trace node, also known as a Span), and the trace record unit is reported to the storage system. A Trace refers to the end-to-end execution path of a single inference request in a distributed tracing system, composed of Trace nodes. In this embodiment, each inference request corresponds to one Trace, which records the complete lifecycle of the request from entering the inference engine to returning a result. By attaching the Token-level performance data corresponding to each inference request as an attribute to the Traced node associated with that inference request, and reporting it asynchronously in batches only upon completion of the inference request, the observation overhead can be kept to the per mille level.

[0072] In step 102, the performance data of multiple inference requests are analyzed to obtain the observation indicators corresponding to each processing stage of different inference requests during the inference process.

[0073] This step can involve periodically analyzing performance data from multiple inference requests at preset time intervals (e.g., daily, weekly) to obtain observation metrics corresponding to each processing stage of the inference process for different inference requests. Alternatively, it can be in response to a user query command, determining the performance data to be analyzed based on the time range specified in the user query command (e.g., 9:00 AM to 5:00 PM) and the inference engine identifier, and then analyzing the performance data to obtain observation metrics corresponding to each processing stage of the inference request. Each inference engine has an inference engine identifier.

[0074] This embodiment does not limit the deployment architecture of the inference engine. For example, an inference engine can be distributed across multiple computing nodes (such as hardware devices). Alternatively, a PD (Prefill / Decode) separation architecture can be used, deploying the Prefill and Decode stages of the inference engine on two separate computing nodes. The observation method provided in this embodiment can support various deployment architectures such as single-machine / multi-machine and PD separation. In this specification, an instance can refer to a computing node running an inference engine, which may contain 8 or more GPUs. In complex cases, multiple computing nodes can be combined to form an inference engine to meet the memory requirements of large models, and multiple instances can jointly form a service cluster to provide services to the outside world. It is understood that the interaction between multiple requests occurs within the same instance. Instances on different computing nodes do not share resources and have no direct interference. However, requests within the same instance competing for hardware resources such as GPUs can lead to execution delays or interruptions.

[0075] In this step, when analyzing the performance data of multiple inference requests, time-series alignment and resource context association can be used to generate observation metrics that characterize the scheduling dependencies, resource contention intensity, and performance interference levels among requests. For example, observation metrics could include the time consumed by each inference request at different processing stages, the time consumed by each inference request at different sub-stages, the number of concurrent requests within a specified time window, contention events among multiple requests for shared resources (including GPU memory, KV cache blocks, and computing units), the degree of performance interference between requests caused by dynamic batch processing or cache replacement, the concurrent interaction topology obtained based on the correlation analysis of multi-request performance data, the generation time of key tokens, the average queuing time, abnormal tokens exceeding the preset generation time, or the number of times the Prefill stage preempts the Decode stage execution, etc.

[0076] In one optional implementation, each processing stage may include a pre-filling stage and a decoding stage. The pre-filling stage and decoding stage may each include multiple sub-stages. For example, the pre-filling stage may include the following sub-stages: inference request preprocessing, queuing, and generation of the first token; the decoding stage may include the following sub-stages: queuing and subsequent token generation. Alternatively, in a PD-separated architecture, the pre-filling stage may include the following sub-stages: inference request preprocessing, queuing, generation of the first token, and sending of the key-value cache; the decoding stage may include the following sub-stages: receiving the key-value cache and subsequent token generation.

[0077] In this embodiment, the observed metrics can be presented visually, and the specific visualization method is not limited. For example, a directed graph can be constructed based on the time series data of the "entry iteration time" and "exit iteration time" of each request. When the exit iteration time of inference request A is greater than the entry iteration time of inference request B and the two have overlapping resources, a dependency relationship A→B is established to intuitively show how one request blocks other requests.

[0078] In one embodiment, a multi-track waterfall plot can be generated based on observed metrics. For example... Figure 2 As shown, each track in the multi-track waterfall plot represents a different process within the inference engine. Each track displays observation metrics corresponding to at least one processing stage of several inference requests, arranged chronologically. Figure 2 The observation metrics shown are the processing time for each inference request across different processing stages (Prefill and Decode stages). Figure 2 This diagram illustrates a PD-separated architecture where the Prefill and Decode phases are handled by different compute nodes. Each inference request's Prefill and Decode phases each occupy one process. The horizontal axis represents the timing sequence, and each processing phase consists of one or more colored blocks. The length of each block represents the execution time of the corresponding processing phase. Different colors are used to distinguish different processing phases; for example, blue represents the Prefill phase, and orange represents the Decode phase. Figure 3 As shown, when a processing stage includes sub-stages, different sub-stages within the same processing stage can be further displayed using color blocks of different colors, with the length of the color block representing the execution time of the corresponding sub-stage.

[0079] In one embodiment, in response to receiving a selection command for any orbit, a detailed graph of the observation indicators for the processing stage corresponding to the orbit can be displayed. The processing stage includes multiple sub-stages, and the detailed graph includes an identifier for each sub-stage, the corresponding color block color, and the execution duration. For example... Figure 4As shown, clicking on a colored block in the Prefill stage displays a pop-up window showing a detailed graph of the corresponding observation metrics for that stage. This embodiment does not limit the specific content of the detailed graph. For example, the Token ID represents the identity of the business calling the inference engine. This identifier is used for business billing, resource quota management, and access control. Each business has a unique Token ID, and all inference requests initiated by that business are associated with this identifier. The Request ID represents a unique identifier for a single user request. When a user submits a request to the inference engine, the system generates a globally unique Request ID. This identifier persists throughout the entire lifecycle of the request processing, enabling users to track the processing status of a specific request, obtain diagnostic information, or submit feedback through this ID. The Trace ID represents a trace identifier for the internal processing flow of the inference engine. Unlike the Request ID, one Request ID may correspond to multiple Trace IDs. This is because: in a PD separation architecture, the Prefill and Decode stages may generate different Trace IDs; when a request is interrupted and rescheduled due to insufficient resources, a new Trace ID may be created; and in a multi-instance deployment scenario, different instances may generate independent Trace IDs for the same request. One token ID can be associated with multiple RequestIDs (representing multiple requests initiated by the same business party), while one Request ID can correspond to multiple Trace IDs (representing the multi-stage processing of the request within the inference engine). The diagram also shows key performance data points, including the generation time of the first token and the execution time of each sub-stage in the Prefill phase.

[0080] In yet another example, multiple tracks corresponding to the same inference request in a multi-track waterfall graph are displayed together using connection markers to indicate that multiple tracks belong to the same inference request. For example... Figure 5 As shown, two tracks can be connected by a connecting line. The inference request continues to execute the decode stage after the prefill stage. In other embodiments, multiple tracks can also be characterized by using methods such as box selection, symbol annotation, or highlighting.

[0081] In yet another example, abnormal inference requests can also be located by searching for time or traceid, such as... Figure 6The example shown is Request 2. The timeline shows that the execution time of the decode phase (represented by the yellow block) for Request 2 is unusually long. Further observation reveals that within the lifecycle of Request 2, there is Request 1. The bright blue block corresponding to Request 1's production of the first token (prefill phase) (from approximately 17:38:29 to 17:38:35) takes a full 6 seconds, overlapping with the decode phase of Request 2 on the timeline. Therefore, it can be analyzed that the delay in the decode phase of Request 2 is due to Request 1's prefill phase monopolizing GPU resources, preventing Request 2's decode phase from being scheduled, resulting in a prolonged interruption in Request 2's execution.

[0082] To demonstrate the high-concurrency scenarios in the inference engine, this manual also provides Figure 7 The multi-track waterfall chart shown provides a panoramic view of the scheduling and concurrency relationships of multiple inference requests within the inference engine, revealing resource contention and performance interference between requests.

[0083] Figure 8 This is a schematic diagram of a device used to observe the request and issuance behavior of a large model inference engine, as described in the embodiments of this specification. This device can be applied to any device, platform, or cluster of devices with computing and processing capabilities. The device includes:

[0084] The data acquisition module 81 is used to acquire the performance data produced by the inference engine during the inference process for each inference request during the concurrent processing of multiple inference requests by the inference engine. The performance data includes the time-series performance data of the key token corresponding to the inference request.

[0085] The data analysis module 82 is used to analyze the performance data of multiple inference requests and obtain the observation indicators corresponding to each processing stage of the inference process.

[0086] In one optional implementation, the timing performance data of the key token includes at least one of the following: the key token's in iteration time, the key token's out iteration time, the key token's execution time, the key token's scheduling delay, the key token's total execution time, and the key token's preprocessing time and queuing time.

[0087] In one optional implementation, the performance data further includes at least one of the following: the time when the inference request enters the inference engine, the time when the inference engine completes processing the inference request, the number of concurrent inference requests in the execution batch, and the number of concurrent tokens in the execution batch.

[0088] In one alternative implementation, the key token includes any one of the following: the first token generated by the inference engine based on the inference request, or the second token generated by the inference engine based on the inference request under the pre-filled decode separation architecture.

[0089] In one alternative implementation, each processing stage includes a pre-filling stage and a decoding stage; the pre-filling stage includes the following sub-stages: inference request preprocessing, queuing, and generation of the first token; the decoding stage includes the following sub-stages: queuing and subsequent token generation.

[0090] In one optional implementation, during the concurrent processing of multiple inference requests by the inference engine, the performance data generated by the inference engine during the inference process is obtained for each inference request, including: setting multiple instrumentation points in the code of the inference engine, the multiple instrumentation points corresponding to multiple processing stages in the inference process of the inference request; and during the concurrent processing of multiple inference requests by the inference engine, the performance data generated by the inference engine during the inference process is obtained for each inference request through the multiple instrumentation points.

[0091] In one optional implementation, after obtaining the performance data generated by the inference engine during the inference process through multiple tracking points for each inference request, the method further includes: when the inference request is completed, attaching the performance data as an attribute to the tracking record unit of the inference request, and reporting the tracking record unit to the storage system.

[0092] In one optional implementation, the performance data of multiple inference requests are analyzed to obtain the observation indicators corresponding to each processing stage of the inference request during the inference process. This includes: in response to receiving a user query instruction, determining the performance data to be analyzed based on the time range specified by the user query instruction and the inference engine identifier, and analyzing the performance data to be analyzed to obtain the observation indicators corresponding to each processing stage of the inference request during the inference process.

[0093] In one optional implementation, after analyzing the performance data of multiple inference requests to obtain the observation metrics corresponding to each processing stage of the inference process, the method further includes: generating a multi-track waterfall chart based on the observation metrics, wherein each track in the multi-track waterfall chart is used to display different processes in the inference engine, and the observation metrics corresponding to at least one processing stage of several inference requests are arranged in chronological order on each track.

[0094] In one alternative implementation, each processing stage consists of one or more color blocks, the length of which represents the execution time of the corresponding processing stage, and the color blocks of different processing stages are distinguished by different colors.

[0095] In one alternative implementation, multiple tracks corresponding to the same inference request in the multi-track waterfall graph are displayed together using connection identifiers to indicate that multiple tracks belong to the same inference request.

[0096] In an optional implementation, the method further includes: in response to receiving a selection instruction for any orbit, displaying a detailed graph of the observation indicators of the processing stage corresponding to the orbit, wherein the processing stage includes multiple sub-stages, and the detailed graph includes an identifier for each sub-stage, a corresponding color block color, and an execution duration.

[0097] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method described in any of the above embodiments.

[0098] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0099] This specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0100] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0103] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0104] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for observing request and issuance behavior of a large model inference engine, the method comprising: During the concurrent processing of multiple inference requests by the inference engine, for each inference request, the performance data generated by the inference engine during the inference process is obtained. The performance data includes the time-series performance data of the keyword token corresponding to the inference request. The time-series performance data of the key token includes the ingress time of the key token and the outgress time of the key token. The performance data of multiple inference requests are analyzed to obtain the observation indicators corresponding to each processing stage of the inference process for different inference requests. The analysis process includes time alignment and context association. The time alignment includes determining the time overlap of multiple inference requests based on the in iteration time and out iteration time of the key token. The context association includes establishing the dependency relationship between multiple inference requests when resource overlap is determined. The observation metrics are used to characterize the scheduling dependencies, resource contention intensity, or performance interference degree among different inference requests; the observation metrics are used to present them in a visual manner.

2. The method according to claim 1, wherein, The timing performance data of the key token also includes at least one of the following: the execution time of the key token, the scheduling latency of the key token, the total execution time of the key token, and the preprocessing time and queuing time of the key token.

3. The method according to claim 1, wherein, The performance data also includes at least one of the following: the time when the inference request enters the inference engine, the time when the inference engine completes processing the inference request, the number of concurrent inference requests in the execution batch, and the number of concurrent tokens in the execution batch.

4. The method according to claim 1, wherein, The key token includes any one of the following: the first token generated by the inference engine based on the inference request, or the second token generated by the inference engine based on the inference request under the pre-filled decode separation architecture.

5. The method according to claim 1, wherein each processing stage includes a pre-filling stage and a decoding stage; the pre-filling stage includes the following sub-stages: inference request preprocessing, queuing, and generation of the first Token; the decoding stage includes the following sub-stages: queuing and subsequent Token generation.

6. The method according to claim 1, wherein, During the concurrent processing of multiple inference requests by the inference engine, the process of obtaining performance data generated by the inference engine during the inference process for each inference request includes: Multiple tracking points are set in the code of the inference engine, and these multiple tracking points correspond to multiple processing stages in the inference process of the inference request; During the concurrent processing of multiple inference requests by the inference engine, the performance data generated by the inference engine during the inference process is obtained for each inference request through the multiple data points.

7. The method according to claim 6, wherein, After obtaining the performance data generated by the inference engine during the inference process through the multiple data points for each inference request, the method further includes: When the inference request processing is completed, the performance data is attached as an attribute to the tracking record unit of the inference request, and the tracking record unit is reported to the storage system.

8. The method according to claim 1, wherein, The analysis of performance data from multiple inference requests yields observation metrics corresponding to each processing stage of the inference process, including: In response to receiving a user query command, the system determines the performance data to be analyzed based on the time range specified in the user query command and the inference engine identifier, and analyzes the performance data to be analyzed to obtain the observation indicators corresponding to each processing stage of the inference request during the inference process.

9. The method according to claim 1, wherein, After analyzing the performance data of multiple inference requests to obtain the observation metrics corresponding to each processing stage of the inference process, the method further includes: A multi-track waterfall plot is generated based on the observed metrics. Each track in the multi-track waterfall plot is used to display different processes in the inference engine. The observed metrics corresponding to at least one processing stage of several inference requests are arranged in chronological order on each track.

10. The method according to claim 9, wherein, Each processing stage consists of one or more color blocks, the length of which represents the execution time of the corresponding processing stage, and the color blocks of different processing stages are distinguished by different colors.

11. The method according to claim 9, wherein, In the multi-track waterfall diagram, multiple tracks corresponding to the same inference request are displayed together using connection identifiers to indicate that the multiple tracks belong to the same inference request.

12. The method according to claim 9, wherein, The method further includes: In response to receiving a selection command for any of the orbits, a detailed graph of the observation indicators of the processing stage corresponding to the orbit is displayed. The processing stage includes multiple sub-stages, and the detailed graph includes the identifier of each sub-stage, the corresponding color block color, and the execution duration.

13. An apparatus for observing the request and issuance behavior of a large model inference engine, the apparatus comprising: The data acquisition module is used to acquire the performance data generated by the inference engine during the inference process for each inference request in the concurrent processing of multiple inference requests by the inference engine. The performance data includes the time-series performance data of the key token corresponding to the inference request. The time-series performance data of the key token includes the ingress time and the outgress time of the key token. The data analysis module is used to analyze the performance data of multiple inference requests to obtain the observation indicators corresponding to each processing stage of the inference request during the inference process. The analysis process includes time alignment and context association. The time alignment includes determining the time overlap of multiple inference requests based on the in iteration time and out iteration time of the key token. The context association includes establishing the dependency relationship between multiple inference requests when resource overlap is determined. The observation metrics are used to characterize the scheduling dependency, resource contention intensity, or performance interference degree among different inference requests; the observation metrics are used to present them in a visual manner.

14. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-12.

Images