Request tracking data display method, device, storage medium and program product

Abstract

Embodiments of the present application provide a method, device, storage medium and program product for displaying request tracking data, the method comprising: creating a plurality of first-level table entries corresponding to thread identifiers of a plurality of execution segments; creating second-level table entries under the plurality of first-level table entries, and determining display positions of execution segments corresponding to a plurality of reasoning requests under a same thread identifier in the second-level table entries, so that execution time periods of the execution segments corresponding to the same thread identifier do not overlap in a same second-level table entry, and consecutive adjacent execution segments of a same reasoning request are allocated to the same second-level table entry, thereby realizing display of concurrent execution between multiple requests and concurrent execution within a single reasoning request in a same second-level table. A directed connection relationship between respective corresponding predecessor execution segments of the plurality of execution segments is established in the second-level table entries, thereby not only displaying execution flow corresponding to a same reasoning request, but also intuitively distinguishing concurrent execution between multiple reasoning requests.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, device, storage medium, and program product for displaying request tracking data. Background Technology

[0002] With the widespread application of large language models in fields such as natural language understanding and multimodal interaction, the scale of model parameters has increased rapidly, and the complexity of inference services has risen accordingly.

[0003] To support the demands of high-throughput, low-latency online services, current inference systems commonly employ techniques such as dynamic batching and continuous batching, achieving highly concurrent execution in multi-processor (e.g., GPU) and multi-node environments. This complex runtime behavior makes system performance tuning extremely challenging, urgently requiring precise, fine-grained performance tracing and visualization analysis methods. Summary of the Invention

[0004] This application provides a method, device, storage medium, and program product for displaying request tracking data, which can display both the execution flow reflecting the concurrent execution of a single inference request and the concurrent execution of multiple inference requests in a two-level table structure.

[0005] In a first aspect, embodiments of this application provide a method for displaying request tracking data, the method comprising:

[0006] Obtain the tracking data generated by the model inference service when processing multiple inference requests. The tracking data includes the thread identifier and execution time period of different execution segments of the multiple inference requests.

[0007] Create multiple first-level table entries corresponding to the thread identifiers of multiple execution segments, wherein the multiple execution segments are the execution segments corresponding to the multiple inference requests;

[0008] According to the secondary entry allocation principle, secondary entries are created under the multiple primary entries, and the display position of the execution segment corresponding to the same thread identifier in the corresponding secondary entry is determined. The content filling direction of the secondary entry corresponds to the set time axis. The secondary entry allocation principle ensures that the execution time periods of the execution segments corresponding to the same thread identifier do not overlap in the same secondary entry and that consecutive adjacent execution segments of the same inference request are allocated in the same secondary entry.

[0009] In the secondary table entries, directed connections are established between the multiple execution segments and their respective predecessor execution segments to form an execution flow corresponding to the same inference request. The predecessor execution segment of an execution segment in the target inference request is the previous execution segment in the target inference request whose execution time is adjacent to that of the execution segment. The target inference request is any one of the multiple inference requests.

[0010] Secondly, embodiments of this application provide a device for displaying request tracking data, the device comprising:

[0011] The acquisition module is used to acquire the tracking data generated by the model inference service when processing multiple inference requests. The tracking data includes the thread identifier and execution time period of different execution segments of the multiple inference requests.

[0012] A creation module is used to create multiple first-level table entries corresponding to the thread identifiers of multiple execution segments, wherein the multiple execution segments are the execution segments corresponding to the multiple inference requests;

[0013] The determination module is used to create secondary entries under the multiple primary entries according to the secondary entry allocation principle, and determine the display position of the execution segment corresponding to the same thread identifier in the corresponding secondary entry. The content filling direction of the secondary entry corresponds to the set time axis. The secondary entry allocation principle ensures that the execution time periods of the execution segments corresponding to the same thread identifier do not overlap in the same secondary entry and that consecutive adjacent execution segments of the same inference request are allocated in the same secondary entry.

[0014] The module is used to establish directed connections between the multiple execution segments and their respective predecessor execution segments in the secondary table entries to form an execution flow corresponding to the same inference request. The predecessor execution segment of an execution segment in the target inference request is the previous execution segment in the target inference request whose execution time is adjacent to that of the target inference request. The target inference request is any one of the multiple inference requests.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores a computer program, and when the computer program is executed by the processor, the processor can at least implement the request tracking data display method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the method for displaying request tracking data as described in the first aspect.

[0017] Fifthly, embodiments of this application provide a computer program product, including: a computer program or instructions, which, when executed by a processor of an electronic device, enable the processor to at least implement the method for displaying request tracking data as described in the first aspect.

[0018] Current model inference services feature concurrent execution within a single request and parallel processing of multiple requests. To help operations and maintenance personnel monitor the operational status of the model inference service in real time and quickly identify performance bottlenecks in concurrent scenarios, this application proposes a request tracking data display scheme. This scheme can intuitively display the execution flow at the granularity of execution fragments within a single inference request within a two-level table structure, reflecting both the concurrent execution within a request and the concurrent execution between multiple inference requests.

[0019] Specifically, firstly, tracking data is collected as the model inference service processes multiple inference requests. This tracking data includes thread identifiers and execution time periods for different execution segments of multiple inference requests. Each execution segment corresponds to a logically complete and continuous block of code or instruction sequence during the execution of an inference request; generally, one execution segment logically represents one execution stage of the inference request. Next, based on the thread identifiers of the execution segments corresponding to the multiple inference requests in the tracking data, a first-level entry is created for each thread in the inference service. Then, according to the second-level entry allocation principle, second-level entries are created under the multiple first-level entries, and the display position of the execution segment corresponding to the same thread identifier in the corresponding second-level entry is determined. This ensures that the execution time periods of the execution segments corresponding to the same thread identifier do not overlap in the same second-level entry, and that consecutive adjacent execution segments of the same inference request are allocated to the same second-level entry. The content filling direction in the second-level entries corresponds to the set time axis. Finally, directed connections are established between multiple execution segments and their corresponding predecessor execution segments in the secondary table entries to form the execution flow corresponding to the same inference request. Here, the predecessor execution segment of an execution segment in an inference request refers to the previous execution segment in the inference request whose execution time is adjacent to that of the execution segment.

[0020] In the above scheme, the execution process of the inference task corresponding to each of the multiple inference requests is divided into multiple execution segments. Tracking data is collected at the granularity of the execution segments. Compared with collecting data at the granularity of functions, this can greatly reduce the amount of tracking data that needs to be collected, so that the model execution process over a longer period of time can be displayed in the view. Furthermore, based on the tracking data, when displaying the execution segments corresponding to multiple inference requests, a first-level table entry is first created for each thread in the inference service according to the thread identifier of each execution segment corresponding to multiple inference requests in the tracking data. Then, according to the second-level table entry allocation principle, second-level table entries are created under multiple first-level table entries respectively, and the display positions of the execution segments corresponding to multiple inference requests under the same thread identifier are determined in the second-level table entries. This ensures that the execution segments corresponding to multiple inference requests under the same thread can be displayed simultaneously in the second-level table entries under the first-level table entries of each thread, and that the execution time periods of the execution segments under the same thread do not overlap in the same second-level table entries. It also ensures that consecutive adjacent execution segments of the same inference request are assigned to the same second-level table entries. In this way, a single second-level table structure can display both the concurrent execution between multiple inference requests under the same thread and the concurrent execution within multiple inference requests, while also avoiding stacked display.

[0021] In addition, by establishing directed connections between the predecessor execution segments corresponding to multiple execution segments in the secondary table entries, the execution flow corresponding to the same inference request can be formed. This not only clearly and intuitively shows the execution flow between various execution segments within the same inference request and the cross-thread transfer process of the request, but also more intuitively distinguishes multiple inference requests, since there are no directed connections between the execution segments corresponding to different inference requests. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a method for displaying request tracking data provided in an embodiment of this application;

[0024] Figure 2 A schematic diagram of a two-level table structure provided in an embodiment of this application;

[0025] Figure 3 A flowchart for determining the predecessor execution fragments corresponding to each execution fragment of a target inference request, provided in an embodiment of this application;

[0026] Figure 4 This application provides an schematic diagram illustrating the determination of a predecessor execution segment in an embodiment of this application.

[0027] Figure 5 A flowchart illustrating a method for visually laying out tracking data using a two-level table structure, provided in an embodiment of this application;

[0028] Figure 6 This application provides a schematic diagram of a visual layout for an embodiment of the present application.

[0029] Figure 7 A flowchart illustrating another method for displaying request tracking data provided in this application embodiment;

[0030] Figure 8 This is an application diagram illustrating a method for displaying request tracking data provided in an embodiment of this application;

[0031] Figure 9 A schematic diagram of the structure of a request tracking data display device provided in an embodiment of this application;

[0032] Figure 10 To and Figure 9 The illustrated embodiment provides a schematic diagram of the electronic device corresponding to the request tracking data display device. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0035] The following describes some embodiments of this application in detail with reference to the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0036] First, the terms or concepts involved in the embodiments of this application will be explained:

[0037] Large language model inference frameworks refer to software systems specifically designed for deploying and running large language models. They primarily improve inference speed and achieve high-throughput, low-latency generative AI services through the coordinated optimization of model algorithms and computational resource scheduling. Common large language model inference frameworks include Very Large Language Model Serving Framework (vLLM), Structured Generation Language Framework (SGLang), Text Generation Inference (TGI), and TensorRT Large Language Model Inference Framework (TensorRT-LLM).

[0038] A process is a running instance of a program in an operating system and is the basic unit of resource allocation and scheduling. It has its own independent memory address space and system resources, can contain multiple threads, and processes are isolated from each other; a crash in one process typically does not affect other processes. The operating system centrally manages the lifecycle of a process and uses a thread group identifier (ID) as its unique identifier, ensuring precise management and scheduling of processes.

[0039] A thread is an execution flow within a process and is the basic unit of scheduling by the Central Processing Unit (CPU). Multiple threads within the same process can share the process's memory space and system resources. Each thread has its own independent program counter, stack, and register state, and the operating system uses a thread identifier as a unique identifier to distinguish each thread. In practical applications, multiple threads can be created to execute tasks concurrently, thereby improving program efficiency and responsiveness.

[0040] Trace: This involves tracking the entire execution process of each inference request handled by the model inference service, recording its execution path across different components, the time consumed at each stage, and the resource usage status. By tracing, the logical sequence of inference requests can be reconstructed, providing crucial evidence for performance analysis, fault location, and optimization of the inference framework. It is a core method for observing the operational behavior of complex model inference frameworks.

[0041] Prefill-Decode Disaggregation (PD Disaggregation) refers to a technical architecture that separates the prefill and decoding stages during large language model inference, executing them in parallel across different computing resources. Typically, high-performance computing devices are used for computationally intensive prefill tasks, while high-memory or low-power devices are used for decoding tasks to improve resource utilization and reduce latency and cost. In a large language model inference framework, tasks can be distributed through router nodes, and multiple threads in the prefill nodes and decode nodes can be launched to execute prefill tasks and decoding tasks, respectively.

[0042] Tensor Parallel (TP) is a model parallelization strategy that splits a single tensor computation (such as matrix multiplication) across multiple devices for parallel execution. In large language model inference frameworks, multiple TP processes or threads can be created to collaboratively handle the computation of an inference request, thereby improving the computational efficiency of model inference.

[0043] Nested spans: When tracing model inference, a logical operation unit (Span) contains one or more sub-operation units (sub-Spans), and the execution time of the sub-Span is entirely within the time interval of the parent Span, forming a tree-like execution trajectory.

[0044] Traditional request trace data visualization solutions primarily collect trace data corresponding to each function during the processing of inference requests by the model inference service and display this trace data in the form of a timeline view or flame graph. While this solution can provide data visualization at the function level, the amount of trace data required is enormous, resulting in a limited time range for the displayed trace data. It cannot show the complete request inference process, and when multiple inference requests are combined into a batch, it is not intuitive to distinguish the request inference process corresponding to different inference requests.

[0045] Furthermore, current large language model inference services are characterized by high concurrency within a single request and parallel processing of multiple requests. For example, in the large language model inference process, parallel attention computation is used in the pre-filling stage, and multiple tokens are generated in parallel in the decoding stage. Therefore, when displaying request tracking data for large language model inference services, it is necessary to show both the parallel execution of multiple execution segments within a single inference request and the concurrent execution of multiple inference requests. However, traditional solutions cannot display both the parallel execution of multiple execution segments within a single inference request and the concurrent execution of multiple inference requests within a single view.

[0046] To address the aforementioned technical issues, this application proposes a scheme for displaying request trace data. This scheme can clearly and intuitively display both the concurrent execution and execution flow within a single inference request, as well as the concurrent execution between multiple inference requests, within a single two-level table structure. Specifically, in this scheme, the model inference process of a single inference request is broken down into multiple execution segments. Trace data is collected at the execution segment level and displayed using a two-level table structure. Within this two-level table structure, multiple threads may be used during the execution of a single inference request. Each first-level table entry represents one of these threads, and each first-level entry is associated with multiple second-level table entries, corresponding to the concurrency level within the same thread (i.e., the execution segments within one or more inference requests executed by the same thread). By identifying the predecessor execution segments for each execution segment corresponding to each inference request, directed connections are established between the execution segments within the same inference request within the second-level table entries, generating the execution flow corresponding to the same inference request. Based on this, in the two-level table structure, the execution process information of a reasoning request can be clearly seen from the first-level table entries corresponding to the multiple threads involved in a reasoning request, the second-level table entries under each thread used to display execution fragments, and the directed connections between execution fragments. Since there are no directed connections between the execution fragments corresponding to different reasoning requests, the execution process views of different reasoning requests can be clearly distinguished in the two-level table structure.

[0047] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0048] The method for displaying request tracking data provided in this application embodiment can be executed by an electronic device, which can be a terminal device such as a PC, laptop, or smartphone, or a server. The server can be a physical server containing an independent host, a virtual server, a cloud server, or a server cluster.

[0049] Figure 1A flowchart illustrating a method for displaying request tracking data provided in this application embodiment is shown below. Figure 1 As shown, it may include the following steps:

[0050] 101. Obtain the trace data generated by the model inference service when processing multiple inference requests. The trace data includes the thread identifier and execution time period of different execution segments of multiple inference requests.

[0051] 102. Create multiple first-level table entries corresponding to the thread identifiers of multiple execution segments, where multiple execution segments are execution segments corresponding to multiple inference requests.

[0052] 103. Based on the principle of secondary table entry allocation, create secondary table entries under multiple primary table entries respectively, and determine the display position of the execution segment corresponding to the same thread identifier in the corresponding secondary table entry.

[0053] 104. Establish directed connections between multiple execution segments and their corresponding predecessor execution segments in the secondary table entries to form the execution flow corresponding to the same inference request. The predecessor execution segment of an execution segment in the target inference request is the previous execution segment in the target inference request whose execution time is adjacent to that of the execution segment. The target inference request is any one of the multiple inference requests.

[0054] The request tracking data display scheme provided in this application embodiment can be applied to track and display the execution process of inference tasks corresponding to various model inference services in multi-request concurrency scenarios and high-concurrency scenarios within a single request. This helps operation and maintenance personnel to clearly understand the execution status, performance bottlenecks, and interrelationships between various concurrent operations of each inference request, thereby better optimizing the performance of model inference services.

[0055] In practice, the first step is to obtain the trace data generated by the model inference service when processing multiple inference requests. In real-world applications, a model inference service may receive and process multiple inference requests from clients simultaneously or sequentially. Therefore, to conduct a more comprehensive performance analysis of the current model inference service, the execution process of the model inference service handling multiple inference requests can be traced separately to obtain the trace data generated by the model inference service when processing multiple inference requests.

[0056] Before tracking multiple inference requests, the execution process of the model task corresponding to each inference request can be divided into multiple execution segments. Execution data corresponding to different execution segments in each inference request can then be collected at the granularity of the execution segments. An execution segment refers to a logical execution stage defined in the model inference process. The complete execution flow can be decomposed into multiple continuous or concurrent execution segments. Generally, one execution stage of an inference request often corresponds to one execution segment.

[0057] For example, in a PD (Programming Domain) separation scenario, for a large language model inference service, after receiving an inference request from a client, the router node first selects a pair of PD nodes (Prefill node and Decode node) to perform the inference computation corresponding to the request, and distributes the client-triggered inference request to the selected pair of PD nodes. Next, the Prefill node and Decode node convert the language description text in the inference request into a token sequence. Then, the Prefill node generates the first target token based on the token sequence and sends the corresponding KV cache data to the Decode node. Next, the Decode node uses the KV cache data corresponding to the first target token to autoregressively generate the next target token. Then, the Decode node continues to iterate and generate subsequent target tokens until an end marker is generated or the maximum length is reached, finally returning the complete result to the client.

[0058] Therefore, the execution process of this model inference task can be broken down into multiple execution stages, such as request dispatch, encoding execution, prefill calculation, key-value cache transfer, and decode calculation, each representing a logical execution segment. It's important to note that these execution segments are coarser-grained compared to function tracing. Each execution segment represents a logically complete and continuous block of code or instruction sequence during execution. This significantly reduces the amount of tracing data required compared to collecting data at the function tracing granularity, and allows for a longer view of the model inference task's execution process. Furthermore, in practical applications, the granularity can be set according to actual operational needs to divide the execution process into multiple execution segments.

[0059] It should be noted that the solution provided in this application embodiment can be applied to both PD separation scenarios and non-PD separation scenarios. The non-PD separation scenario can be considered a special case of the PD separation scenario: it is just that there is no need to distinguish the node type.

[0060] The tracking data refers to the timing and execution context information collected during the execution of inference tasks by the model inference service, used to record the execution segments corresponding to each inference request. Specifically, the tracking data includes the thread identifier and execution time period of each execution segment of each inference request. In addition, the tracking data may also include event information corresponding to each execution segment, the node identifier to which each execution segment of each inference request belongs, the host identifier to which each execution segment of each inference request belongs, the request identifier to which each execution segment of each inference request belongs, the segment name corresponding to each execution segment of each inference request, the segment identifier of the predecessor execution segment corresponding to each execution segment of each inference request, and attribute information corresponding to each execution segment (e.g., the batch size of this scheduling can be added for the Decode segment).

[0061] The event information specifically includes the event name, the event occurrence time, and the execution segment to which the event belongs. The node identifier refers to the identification information corresponding to the logical computation node running the execution segment during the inference task execution process. Logical computation nodes can be computation nodes in the inference framework, such as Prefill nodes, Decode nodes, and router nodes. The host identifier refers to the identification information corresponding to the machine device in the distributed system running the execution segment during the inference task execution process. These machine devices can be servers, virtual machines, container instances, etc., in the distributed system. The request identifier is used to identify which inference request the execution segment belongs to. The segment name is used to intuitively describe the specific function or operation content of different execution segments in the inference request, enabling technical personnel to quickly understand the role of the execution segment in the entire inference process. The segment identifier is used to represent the identity of each independent execution segment after the inference request is split.

[0062] Furthermore, the collected tracking data can be in formats such as execution span data (opentelemetry span data), log data, or a custom data structure. Additionally, the methods for tracking each execution segment in multiple inference requests and acquiring the tracking data can be implemented using existing related technologies, which will not be elaborated upon here.

[0063] Furthermore, after obtaining the tracking data generated when the model inference service processes multiple inference requests, the predecessor execution fragments corresponding to each inference request can be determined separately. When determining the predecessor execution fragments, multiple execution fragments in the tracking data can first be grouped according to the request identifier corresponding to the execution fragment to determine the execution fragments corresponding to each inference request, and then the predecessor execution fragments corresponding to each inference request can be determined sequentially. Since the process of determining the predecessor execution fragments of each execution fragment in each inference request is roughly the same, this explanation uses the target inference request as an example.

[0064] For a target inference request, if the collected tracking data includes the segment identifiers of the predecessor execution segments of each execution segment corresponding to the target inference request, then the predecessor execution segments of each execution segment corresponding to the target inference request can be determined directly based on the segment identifiers of the predecessor execution segments of each execution segment in the tracking data. Specifically, the predecessor execution segment of an execution segment in the target inference request is the preceding execution segment in the target inference request whose execution time is adjacent to that of the previous execution segment.

[0065] If the collected tracking data does not include the segment identifiers of the predecessor execution segments corresponding to each execution segment of the target inference request, then the predecessor execution segments corresponding to each execution segment of the target inference request are determined based on the preset predecessor execution segment identification rules. The preset predecessor execution segment identification rules include intra-node predecessor execution segment identification rules and inter-node predecessor execution segment identification rules.

[0066] For example, when the model inference service adopts an inference architecture that separates the prefilling and decoding phases, the intra-node preceding execution fragment identification rule can be used to determine the preceding execution fragment of the first execution fragment in the Prefill node and the preceding execution fragment of the first execution fragment in the Decode node corresponding to the target inference request. The inter-node preceding execution fragment identification rule can be used to sequentially determine the preceding execution fragments of non-first execution fragments in the Prefill node, non-first execution fragments in the Decode node, and non-first execution fragments in the router node corresponding to the target inference request. When the model inference service does not adopt an inference architecture that separates the prefilling and decoding phases, it is equivalent to using only one node to handle both the prefilling and decoding phases. In this case, the intra-node preceding execution fragment identification rule can be directly used to determine the preceding execution fragments of each execution fragment corresponding to the target inference preceding execution. For example, for the target inference request, the execution fragments within the target inference request are sorted according to their start execution time. Based on the sorting result, the preceding execution fragment in each execution fragment is determined as the preceding execution fragment of each execution fragment.

[0067] In other words, regardless of the inference architecture used by the model inference service, the predecessor execution fragments corresponding to each execution fragment of the target inference predecessor can be determined based on the preset predecessor execution fragment identification rules. The detailed determination process is described below.

[0068] After preprocessing the tracking data, specifically identifying the predecessor execution fragments corresponding to each execution fragment in the multiple inference requests, a two-level table structure is used to visually arrange the execution fragments corresponding to the multiple inference requests. To facilitate understanding of the specific implementation process of visually arranging the execution fragments corresponding to multiple inference requests, the following section will first combine... Figure 2 The general design framework for the two-level table structure used to display each execution segment is described in detail.

[0069] like Figure 2 As shown, in this two-level table structure, each thread in the inference service is designated as a first-level table entry, and each first-level table entry name uses its corresponding thread label, host identifier, and thread identifier. The thread label is a string identifier used to describe the task content performed by that thread during the model inference task.

[0070] Each thread's primary entry can include multiple secondary entries. The number of secondary entries reflects the thread's maximum concurrency: the maximum number of inference requests and their corresponding inference tasks that can be processed simultaneously. Each secondary entry can be assigned a sequence number, which serves as the secondary entry name.

[0071] In reality, the number of second-level entries under a first-level entry needs to be determined based on the allocation principles of the execution fragments stored in the second-level entries, and is not a known value. For example, if the execution time periods of two inference requests executed by the same thread do not overlap, then according to the allocation principles, the execution fragments of these two inference requests can be displayed in the same second-level entry. In this case, the number of second-level entries under that thread may be less than the number of inference requests it executes.

[0072] Each second-level table entry can contain multiple execution segments. Furthermore, each second-level table entry can sequentially display multiple execution segments from the corresponding thread, arranged according to their execution time periods. The left edge of each execution segment is aligned with the start time of the specified execution timeline, and the right edge is aligned with the end time of the specified execution timeline, thus illustrating the time distribution of each execution segment. In other words, the second-level table structure corresponds to a timeline, and the execution time periods of each execution segment within each second-level table entry are placed based on this timeline.

[0073] Furthermore, when displaying multiple execution segments under the same thread in a secondary table entry, the following allocation principles are followed to determine the display position of the execution segments: the execution time periods of execution segments under the same thread do not overlap in the same secondary table entry, and consecutive adjacent execution segments of the same inference request are allocated in the same secondary table entry. In addition, directed connection relationships between execution segments are set in the secondary table entry to represent the execution flow corresponding to each inference request.

[0074] In an optional embodiment, special markers can be used to associate and display event information corresponding to each execution segment in each secondary entry. The special marker overlaps with the execution segment in which the event occurred, and the position of the marker is aligned with the event occurrence time on the set time axis.

[0075] In practical applications, after identifying the predecessor execution segments corresponding to each inference request in the tracking data, the execution segments corresponding to multiple inference requests can be visualized and laid out according to the designed two-level table structure.

[0076] Specifically, when creating a visual layout, you can first create a first-level table entry for each thread in the inference service based on the thread identifiers of the execution segments corresponding to multiple inference requests in the tracking data. Since the thread identifier of each execution segment is known, you create as many first-level table entries as there are threads.

[0077] Then, based on thread identifiers, multiple execution segments are grouped to determine the multiple execution segments under each thread. Then, the position of each execution segment under the first-level table entry corresponding to its respective thread is determined. That is, for multiple execution segments belonging to the same thread, the location of each execution segment under the first-level table entry corresponding to that thread is determined, which involves the creation of second-level table entries and the layout of these execution segments within the second-level table entries.

[0078] This involves creating secondary entries under multiple primary entries based on secondary entry allocation principles, and determining the display position of the execution segment corresponding to the same thread identifier within the corresponding secondary entry. The content filling direction of the secondary entries corresponds to the set timeline, and this allocation principle ensures that the execution time periods of execution segments corresponding to the same thread identifier do not overlap within the same secondary entry, and that consecutive adjacent execution segments of the same inference request are allocated to the same secondary entry.

[0079] In other words, the principle for allocating second-level entries for multiple execution segments under the same thread is as follows: For multiple execution segments corresponding to the same inference request, based on the principles of time conflict and placing consecutive adjacent execution segments in the same second-level entry as much as possible, determine how many second-level entries to create in the first-level entry corresponding to the thread, and which execution segments should be displayed in each second-level entry. Furthermore, the execution segments in each second-level entry are arranged sequentially according to their execution time periods. Among these, consecutive adjacent execution segments of the same inference request, provided their execution time periods do not conflict (i.e., do not overlap), are placed in the same second-level entry, with minimal or no overlap across second-level entries, allowing the execution flow of the inference request to be more clearly observed.

[0080] As described above, a first-level table entry is created for each thread in the inference execution task. This allows for a clear visualization of the concurrent execution of multiple threads within a single request. Furthermore, multiple second-level table entries are created under each thread's corresponding first-level table entry to display the execution fragments corresponding to multiple inference requests under that thread in their respective second-level table entries. This allows for the display of execution fragments corresponding to multiple inference requests under the same thread within a single second-level table structure.

[0081] Finally, directed connections can be established between multiple execution segments and their corresponding predecessor execution segments in the secondary table entries to form the execution flow corresponding to the same inference request. The process of establishing the corresponding execution flow for each inference request is roughly the same; the following uses the target inference request as an example to illustrate the specific implementation of establishing the execution flow corresponding to the target inference request.

[0082] For a target inference request, since the predecessor execution fragments for each corresponding execution fragment have been identified, directed connectors can be used to directly connect these execution fragments with their corresponding predecessor execution fragments, establishing pairwise directed connections to form the execution flow corresponding to the target inference request. These execution fragments are represented by directed connectors, with the arrows pointing from a predecessor execution fragment to the current execution fragment, indicating the execution order. For example, if execution fragment A corresponds to predecessor execution fragment B, then a directed connector is used, starting from execution fragment B and pointing to execution fragment A.

[0083] In summary, in this embodiment of the application, the execution process of the inference task corresponding to each inference request is divided into multiple execution segments, and tracking data is collected at the granularity of the execution segments. Compared with collecting data at the granularity of functions, this can greatly reduce the amount of tracking data that needs to be collected, thereby allowing a longer model execution process to be displayed in the view. Furthermore, based on the tracking data, when displaying the execution segments corresponding to multiple inference requests, a first-level table entry is first created for each thread in the inference service according to the thread identifier of each execution segment corresponding to multiple inference requests in the tracking data. Then, according to the second-level table entry allocation principle, second-level table entries are created under multiple first-level table entries respectively, and the display positions of the execution segments corresponding to multiple inference requests under the same thread identifier are determined in the second-level table entries. This ensures that the execution segments corresponding to multiple inference requests under the same thread can be displayed simultaneously in the second-level table entries under the first-level table entries of each thread, and that the execution time periods of the execution segments under the same thread do not overlap in the same second-level table entries. It also ensures that consecutive adjacent execution segments of the same inference request are assigned to the same second-level table entries. In this way, a single second-level table structure can display both the concurrent execution of multiple inference requests under the same thread and the concurrent execution of each inference request, while also avoiding stacked display. In addition, by sequentially determining the predecessor execution fragments corresponding to each execution fragment of each inference request, and establishing directed connection relationships between the predecessor execution fragments corresponding to each of the multiple execution fragments in the secondary table entries, an execution flow corresponding to the same inference request can be formed. This not only clearly and intuitively displays the execution flow between the various execution fragments within the same inference request, but also more intuitively distinguishes the concurrent execution of multiple inference requests.

[0084] In addition, in an optional embodiment, the tracking data also includes event information corresponding to each execution segment. Then, in the secondary table entries, the event information can be displayed in association with the corresponding execution segment based on the occurrence time of the event information corresponding to each execution segment. This makes it easier for subsequent operation and maintenance personnel to accurately optimize the model inference service based on the event information.

[0085] The following section details the process for determining the predecessor execution fragments corresponding to each execution fragment in the above embodiments.

[0086] Figure 3 This is a flowchart illustrating how to determine the predecessor execution fragments corresponding to each execution fragment of a target inference request, as provided in an embodiment of this application. Figure 3 As shown, the model inference service adopts an inference architecture that separates the pre-filling stage and the decoding stage, and the tracking data also includes the node identifier to which the execution segment belongs. Specifically, the method includes the following steps:

[0087] 301. For each execution segment corresponding to the target inference request, group them according to the node identifier to obtain the first group corresponding to the pre-filled node, the second group corresponding to the decoding node, and the third group corresponding to the routing node.

[0088] 302. Sort the execution segments in the first group, the second group, and the third group according to the order of their start execution time.

[0089] 303. Determine that the predecessor execution fragments corresponding to the first execution fragment in the first group and the second group are both the first execution fragments in the third group.

[0090] 304. Based on the sorting results of the non-first execution segments in the first group, the second group, and the third group, and their corresponding thread identifiers, determine the predecessor execution segments of the non-first execution segments in the first group, the second group, and the third group, respectively.

[0091] The model inference service employs a PD-separated inference architecture, where a routing node distributes tasks, a pre-filling node executes the pre-filling phase of the model inference task, and a decoding node executes the decoding phase. When determining the predecessor execution fragments for each execution fragment corresponding to the target inference request, the predecessor execution fragment of the first execution fragment in a group of execution fragments contained within each node is first determined according to the inter-node predecessor execution fragment identification rules. Then, the predecessor execution fragments of non-first execution fragments in a group of execution fragments contained within each node are determined according to the intra-node predecessor execution fragment identification rules.

[0092] In practice, the execution segments corresponding to the target inference request can first be grouped according to their respective node identifiers, resulting in a first group corresponding to pre-filled nodes, a second group corresponding to decoding nodes, and a third group corresponding to routing nodes. The execution segments within each of these groups are then sorted according to their starting execution time. Next, the rules for identifying predecessor execution segments between execution nodes are as follows: the predecessor execution segment corresponding to the first execution segment in the first group is determined as the first execution segment in the third group, and the predecessor execution segment corresponding to the first execution segment in the second group is also determined as the first execution segment in the third group. Notably, the first execution segment of a group of execution segments contained in a routing node typically does not have a predecessor execution segment; therefore, its first execution segment is considered the first execution segment in the execution flow corresponding to the target inference request.

[0093] Next, the predecessor execution segment identification rule within the execution node is implemented: based on the sorting results of the non-first execution segments in the first group, the second group, and the third group, and the corresponding thread identifiers, the predecessor execution segments of the non-first execution segments in the first group, the second group, and the third group are determined respectively.

[0094] Specifically, when determining the predecessor execution fragment of a non-first execution fragment within each group, the predecessor execution fragment can be searched first within the same thread based on the sorting result of the execution fragment within the corresponding group and the thread identifier corresponding to the execution fragment, and then searched in other threads. This is because the execution of inference requests on a thread is generally sequential; one thread finishes processing before another thread begins processing.

[0095] Since execution segments within the same thread are usually executed sequentially according to their execution time, it is extremely rare for execution segments within the same thread to overlap in time. Therefore, by searching for the predecessor execution segment within the same thread, we can avoid errors in identifying the predecessor execution segment due to time confusion, thereby ensuring that the execution flow corresponding to the generated target inference request conforms to the actual running logic.

[0096] The process of determining the predecessor execution fragment of a non-first execution fragment within each group is roughly the same. The following example, using the target execution fragment in the target group, illustrates the process of determining the predecessor execution fragment of a non-first execution fragment. The target group can be any one of the first, second, or third groups, and the target execution fragment is any non-first execution fragment within that target group.

[0097] For a target execution segment within a target group, we can first search for a predecessor execution segment within the same thread. If no predecessor execution segment is found within the same thread, we then search for a predecessor execution segment across the entire target group. Specifically, we can first determine if there exists at least one execution segment in the target group that shares the same thread identifier as the target execution segment. If there is at least one execution segment in the target group that shares the same thread identifier as the target execution segment, then the predecessor execution segment of the target execution segment is determined to be the execution segment whose order is closest to the target execution segment among the at least one execution segments. If there is no execution segment in the target group that shares the same thread identifier as the target execution segment, then the predecessor execution segment of the target execution segment is determined to be the execution segment whose order is closest to the target execution segment in the target group.

[0098] For example, suppose the target group consists of n threads, and the target execution segment i belongs to thread j. For target execution segment i, first check if thread j contains other execution segments. If thread j contains other execution segments, then the execution segment k in thread j whose execution time is earlier and closest to target execution segment i is the predecessor execution segment of target execution segment i. Otherwise, search among other threads for the execution segment whose execution time is closest to target execution segment i. If execution segment q is found to have an execution time closest to target execution segment i, then this execution segment q is the predecessor execution segment of target execution segment i.

[0099] In an alternative embodiment, in order to more accurately identify the predecessor execution fragments that are not the first execution fragments within the target group, the following method can be used to determine them.

[0100] For the target group, firstly, the global predecessor variable and the thread predecessor variable table are initialized to empty. The global predecessor variable stores the most recently scanned execution fragment within the target group. The thread predecessor variable table stores the thread identifier and the most recently scanned execution fragment under that thread identifier. Then, the following process is repeated until all execution segments in the target group have been scanned: Retrieve the currently scanned execution segment from the target group; Determine if a thread identifier corresponding to the currently scanned execution segment exists in the thread predecessor variable table; If a thread identifier corresponding to the currently scanned execution segment does not exist in the thread predecessor variable table, and the global predecessor variable is empty, then determine that the predecessor execution segment of the currently scanned execution segment is empty; If a thread identifier corresponding to the currently scanned execution segment does not exist in the thread predecessor variable table, and the global predecessor variable is not empty, then determine that the predecessor execution segment of the currently scanned execution segment is the value of the global predecessor variable; If a thread identifier corresponding to the currently scanned execution segment exists in the thread predecessor variable table, then determine that the predecessor execution segment of the currently scanned execution segment is the execution segment corresponding to the thread identifier in the thread predecessor variable table; Update the execution segment corresponding to the thread identifier of the currently scanned execution segment in the thread predecessor variable table to the currently scanned execution segment, and update the value of the global predecessor variable to the currently scanned execution segment.

[0101] To facilitate understanding of the above implementation, the following will be combined with, for example... Figure 4 Let's illustrate the process of determining the predecessor execution slice. Assume the group corresponding to the Prefill node contains 5 execution slices: P1, P2, P3, P4, and P5. P1, P2, and P4 belong to the same thread A1, P3 belongs to thread A2, and P5 belongs to the same thread A3. After sorting these 5 execution slices, the sequence is P1, P2, P3, P4, P5. Before performing the scan, the global predecessor variable `global_prev` and the thread predecessor variable table `thread_prev` are initialized to empty. `global_prev` records the most recently processed execution slice from a global perspective throughout the scan, while `thread_prev` records the most recently processed execution slice for each thread.

[0102] In other words, initially, `global_prev` is null, and `thread_prev` is an empty table. The index of the `thread_prev` table can be the thread identifier `slice.pid` corresponding to the execution segment, or a combination of the thread identifier `slice.pid` and the host identifier `slice.host_id`. In this example, only the thread identifier is used as the index. Furthermore, each entry in the `thread_prev` table corresponding to the thread identifier `slice.pid` stores the slice that the thread has just scanned. The next slice of this thread will use this stored slice as its predecessor execution segment. Additionally, the total number of threads is initially unknown; if a new thread is encountered during subsequent scans, an entry is added to the `thread_prev` table.

[0103] The following steps involve scanning each execution segment within the group. Specifically, execution segment P1 is extracted from the group, i.e., the current slice = P1. The thread identifier (slice.pid) of execution segment P1 is A1. Since thread identifier A1 is not found in `thread_prev` and `global_prev` is empty, the predecessor execution segment of execution segment P1 is set to `none`. Then, `thread_prev` and `global_prev` are updated. Specifically, the thread identifier corresponding to the scanned slice is recorded in the `thread_prev` table, ensuring that the updated `thread_prev` records the most recently scanned execution segment under thread identifier A1 as P1. Additionally, execution segment P1 is updated in `global_prev`.

[0104] Next, the execution segment P2 is retrieved from the group, i.e., the current slice = P2. The thread identifier (slice.pid) of execution segment P2 is A1. The entry corresponding to thread identifier A1 is found in thread_prev: the query result is P1. Therefore, the predecessor execution segment of execution segment P2 is determined to be P1, and the entry for thread A1 in thread_prev is updated: execution segment P2 is written to this entry, and the previously written entry for P1 is deleted; simultaneously, the value of global_prev is updated to P2.

[0105] Then, extract execution segment P3 from the group, i.e., the current slice = P3. The thread identifier (slice.pid) of execution segment P3 is A2. Since thread identifier A2 is not found in `thread_prev`, and the current value of `global_prev` is P2, the predecessor execution segment of execution segment P3 is P2. Then, update `thread_prev` and `global_prev`, adding the thread identifier corresponding to the recently scanned slice as a new entry to `thread_prev`. That is, a new entry is added to the updated `thread_prev`, with the index A2 and the value P3. Additionally, the value of `global_prev` is updated to P3. Following the same method, extract P4 and P5 from the group, and determine the predecessor execution segments for each of P4 and P5 respectively. These steps will not be elaborated further.

[0106] The above method can accurately identify the predecessor execution fragments that are not the first execution fragment in the first group, the second group, and the third group, thereby ensuring the display effect of each execution fragment in the secondary table generated by the display method.

[0107] The following provides a detailed explanation of the process of visualizing the layout of tracking data using a two-level table structure in the above embodiments.

[0108] Figure 5 This is a flowchart illustrating a method for visually laying out tracking data using a two-level table structure, as provided in an embodiment of this application. Figure 5 As shown, the specific method includes the following steps:

[0109] 501. Generate multiple execution fragment groups corresponding to multiple inference requests. Sort the multiple inference requests according to the order of the start execution time of the first execution fragment in the multiple execution fragment groups to obtain a request sequence. Sort the execution fragments in the execution fragment group corresponding to each inference request according to the order of the start execution time.

[0110] 502. Traverse the execution segments in the target execution segment group one by one, and check if there is a first-level table entry in the already created first-level table entry that corresponds to the thread identifier of the target execution segment currently being traversed. If it exists, proceed to steps 503-506; otherwise, proceed to step 507. The target execution segment group is the execution segment group corresponding to the currently traversed request when traversing the request sequence in sequence.

[0111] 503. If a first-level table entry already created exists that corresponds to the thread identifier of the currently traversed target execution segment, then determine the most recently stored first-level table entry for the execution segment under the first-level table entry. The target execution segment is grouped into any one of multiple execution segment groups.

[0112] 504. Determine whether the execution time period of the target execution segment overlaps with the execution time period of the execution segments already stored in the first and second level entries.

[0113] 505. If the execution time period of the target execution segment does not overlap with the execution time period of the execution segments already stored in the first and second level entries, then the target execution segment is stored in the first and second level entries according to the execution time period of the target execution segment.

[0114] 506. If the execution time period of the target execution fragment overlaps with the execution time period of an execution fragment already stored in the first or second-level table entry, then the target execution fragment is stored in the second-level table entry under the first-level table entry according to its execution time period. Wherein, the execution time period of the execution fragment already stored in the second-level table entry does not overlap with the execution time period of the target execution fragment, and the second-level table entry is either already created or newly created under the first-level table entry.

[0115] 507. If there is no first-level table entry corresponding to the thread identifier of the target execution segment in the already created first-level table entries, then create a second-level table entry corresponding to the thread identifier of the target execution segment, and create a third-level table entry under the second-level table entry. Store the target execution segment into the third-level table entry according to the execution time period of the target execution segment.

[0116] During the layout process, firstly, multiple execution segments in the trace data are grouped according to inference requests, resulting in execution segment groups corresponding to each inference request. Then, the execution segments within each execution segment group corresponding to a specific inference request are sorted according to their chronological order of start execution time, generating multiple execution segment groups for multiple inference requests. In other words, each execution segment group contains all the execution segments within a single inference request, ensuring that the execution segments within each inference request are ordered.

[0117] Furthermore, based on the order of the start execution times of the first execution segment in each of the multiple execution segment groups, the multiple inference requests are sorted to obtain a request sequence. Then, the request sequence is traversed sequentially, and the execution segment group corresponding to the currently traversed request is called the target execution segment group.

[0118] Next, each execution segment in the target execution segment group is stored in a secondary table. The process of storing each execution segment in each execution segment group is roughly the same; the following example uses the target execution segment in the target execution segment group as an illustration. The target execution segment is the currently traversed execution segment in the target execution segment group.

[0119] The process iterates through the execution segments in the target execution segment group, checking if any of the existing first-level entries in the table correspond to the thread identifier of the currently traversed target execution segment. In other words, when laying out execution segments, the principle that only one thread can create one first-level entry is followed. Therefore, when determining the layout information of the currently traversed target execution segment in the second-level table structure, it first checks if any of the existing first-level entries correspond to the thread identifier of the currently traversed target execution segment. If so, no new first-level entry is created; instead, the display position of the currently traversed target execution segment in the second-level table under the first-level entry corresponding to the thread identifier of the currently traversed target execution segment is determined.

[0120] When determining the display position of the target execution segment currently being traversed in the second-level entry under the first-level entry, the allocation principle of the second-level entry needs to be followed so that the execution time periods of the execution segments corresponding to the same thread identifier do not overlap in the same second-level entry, and that consecutive adjacent execution segments of the same inference request are allocated in the same second-level entry.

[0121] Specifically, if a first-level table entry already created exists that corresponds to the thread identifier of the currently traversed target execution segment, then the most recently stored first-level table entry for the execution segment is determined under the first-level table entry. It is then determined whether the execution time period of the target execution segment overlaps with the execution time period of any execution segment already stored in the first-level table entry. If the execution time period of the target execution segment does not overlap with the execution time period of any execution segment already stored in the first-level table entry, the target execution segment is stored in the first-level table entry based on its execution time period. If the execution time period of the target execution segment overlaps with the execution time period of any execution segment already stored in the first-level table entry, the target execution segment is stored in the second-level table entry under the first-level table entry based on its execution time period. Here, the execution time period of any execution segment already stored in the second-level table entry does not overlap with the execution time period of the target execution segment, and the second-level table entry is either already created or newly created under the first-level table entry.

[0122] As described above, if a first-level table entry already created exists that corresponds to the thread identifier of the currently traversed target execution segment, the first-level table entry corresponding to the most recently stored execution segment under that first-level table entry is determined. In the first-level table entry, a time conflict detection is performed on the target execution segment to determine whether the execution time period of the target execution segment overlaps with that of the execution segment already stored in the first-level table entry. If they do not overlap, the target execution segment is directly stored in the first-level table entry. If they overlap, a new second-level table entry is created or the target execution segment is stored in an already created second-level table entry.

[0123] Simply put, for each new inference request, we try to store its corresponding execution segments in an existing second-level table entry. When a time conflict occurs, we create a new second-level table entry.

[0124] In an optional embodiment, to more accurately determine the first and second-level entries of the most recently stored execution fragment, before traversing the execution fragments in the target execution fragment group one by one, a global view variable table describing the occupancy of time intervals for all second-level entries under all first-level entries, and a request view state table describing the occupancy of second-level entries for the multiple inference requests under different first-level entries, can be initialized. Based on whether the thread identifier of the target execution fragment exists in the global view variable table, it is determined whether there is a first-level entry in the already created first-level entries corresponding to the thread identifier of the currently traversed target execution fragment. If the thread identifier of the target execution fragment exists in the request view state table, then the second-level entry in the request view state table corresponding to the thread identifier of the target execution fragment is determined to be the first and second-level entry of the most recently stored execution fragment under the first-level entry.

[0125] Additionally, when the target execution fragment has been stored in the secondary table, the execution time period of the target execution fragment can be updated in the record corresponding to the thread identifier of the target execution fragment and the secondary table entry where it is stored in the global view variable table, and the secondary table entry where the target execution fragment is stored can be updated in the record corresponding to the thread identifier of the target execution fragment in the request view status table.

[0126] Furthermore, in practical applications, there are situations where existing first-level entries do not contain a corresponding first-level entry for the thread identifier of the target execution segment. Specifically, if an existing first-level entry does not contain a corresponding first-level entry for the thread identifier of the target execution segment, a second-level entry corresponding to the thread identifier of the target execution segment is created. Under this second-level entry, a third-level entry is created, and the target execution segment is stored in the third-level entry according to its execution time period.

[0127] In other words, if there is no first-level entry in the currently created first-level table that corresponds to the thread identifier of the target execution segment being traversed, meaning that a first-level table entry corresponding to the thread to which the target execution segment belongs has not yet been created, then a new first-level table entry can be created directly. Furthermore, a second-level table entry can be created in the newly created first-level table entry, and the target execution segment can be stored in the created second-level table entry.

[0128] In summary, this embodiment of the application locates the first-level table entry by the thread identifier corresponding to the currently traversed target execution segment, and then locates the display position of the target execution segment in the second table entry based on the second-level table entry allocation rules. This allows for the precise simultaneous display of execution segments corresponding to multiple inference requests under the same thread in the second-level table entry under the first-level table entry for each thread. Furthermore, the execution time periods of each execution segment under the same thread do not overlap in the same second-level table entry, and consecutive adjacent execution segments of the same inference request are allocated to the same second-level table entry. Thus, a second-level table structure can display both the concurrent execution of multiple inference requests under the same thread and the concurrent execution of each inference request within each inference request, while also avoiding stacked display.

[0129] In another optional embodiment, after generating multiple execution fragment groups corresponding to multiple inference requests, the multiple inference requests can be sorted according to the order of the start execution times of the first execution fragment in each execution fragment group to obtain a request sequence. Then, the request sequence is traversed sequentially, and an execution fragment group corresponding to one inference request is extracted from the request sequence each time. The target execution fragment group is the execution fragment group corresponding to the currently traversed request when traversing the request sequence. Following the above method, the display position of each execution fragment in the execution fragment group corresponding to each inference request extracted from the request sequence in the secondary table can be determined sequentially, and stored in the corresponding display position.

[0130] This means that the execution fragments in the execution fragment group corresponding to each inference request are displayed sequentially according to the execution order of multiple inference requests. This can realistically restore the execution sequence, ensure that the display order of multiple inference requests is consistent with the actual startup time, and can display multiple execution fragments within the same inference request together to prevent the execution fragments of different inference requests from being mixed together, so as to more intuitively show the execution status of multiple inference requests.

[0131] To facilitate understanding of the above implementation, the following will be combined with, for example... Figure 6 This section explains the process of determining the display position of each execution fragment corresponding to multiple inference requests in the secondary table. Assume the model inference service concurrently processes inference requests from three clients: Inference Request 1, Inference Request 2, and Inference Request 3. Furthermore, when processing the inference task corresponding to each inference request, seven threads are invoked to collaboratively complete the inference task.

[0132] First, based on the request identifier corresponding to each execution fragment, multiple execution fragments are grouped to generate execution fragment group a for inference request 1, execution fragment group b for inference request 2, and execution fragment group c for inference request 3, respectively. Furthermore, within each execution fragment group corresponding to an inference request, the execution fragments are sorted according to their starting execution time.

[0133] Then, based on the order of the start times of the first execution segment in each of the three execution segment groups, the three inference requests are sorted to obtain the request sequence. That is, the request sequence is inference request 1, inference request 3, and inference request 2. Next, the execution segment groups corresponding to each inference request in the request sequence are traversed sequentially.

[0134] Before traversing the request sequence, we can first initialize a global view variable table `graph` to describe the occupancy of time intervals for all second-level entries under all first-level entries, and a request view status table `req_graph_status` to describe the occupancy of second-level entries for the three inference requests under different first-level entries. Specifically, `graph` describes the position of each execution fragment in the second-level table and its corresponding identification information (including thread identifier `pid` and host identifier `host_id`, or only the thread identifier, depending on the design). `graph` can be denoted as `graph: Map<Tuple(host_id, pid), Map<line, SliceContainer>>`, where `Tuple(host_id, pid)` is the index of the first-level entry, representing the thread name, `line` represents the index of the second-level entry, and `SliceContainer` stores the time interval corresponding to the currently placed execution fragment in the second-level entry. `req_graph_status` describes the position of the most recently placed second-level entry for each thread involved in an inference request. req_graph_status can be denoted as req_graph_status: Map<Tuple(host_id, pid), line>. In this embodiment, the name of the first-level table entry can be represented by the thread identifier, or by a combination of the thread identifier and the host identifier, or it can be considered that the thread identifier in a broad sense can include the host identifier. In this application, the thread identifier and the host identifier are not strictly distinguished.

[0135] Specifically, we can first extract the execution fragment group 'a' corresponding to inference request 1 from the request sequence. For execution fragment group 'a', assume that execution fragment group 'a' includes execution fragments P1, P2, P3, P4, and P5. And P1, P2, and P4 belong to the same thread A1, P3 belongs to thread A2, and P5 belongs to the same thread A3.

[0136] Extract execution segment P1 from the execution segment group a, i.e., the current slice = P1. The thread identifier (slice.pid) of execution segment P1 is A1, and the host identifier (slice.host_id) is B1. Use the combination of the thread identifier and host identifier corresponding to execution segment P1 as the index key, i.e., the current key = Tuple(slice.host_id, slice.pid) = Tuple(B1, A1).

[0137] Subsequently, if the key is not found in the graph, a first-level entry 1 corresponding to thread A1 is created, and a second-level entry 1 is created under first-level entry 1. The current line is designated as second-level entry 1. Execution segment P1 is stored in second-level entry 1 according to its execution time period x1. The occupancy information of execution segment P1 in second-level entry 1 is updated to the corresponding entry graph[key][line] in the graph table, that is, the occupancy information of the time interval of execution segment P1 in second-level entry 1 under first-level entry 1 is updated to this graph table. Simultaneously, the information of second-level entry 1 occupied by execution segment P1 can also be updated to the entry req_graph_status[key] in req_graph_status, that is, the second-level entry 1 stored under first-level entry 1 of execution segment P1 is updated to this req_graph_status table.

[0138] Next, extract execution segment P2 from execution segment group a, i.e., the current slice = P2. Here, the thread identifier slice.pid of execution segment P2 is A1, and the host identifier slice.host_id is B1. The combination of the thread identifier and the host identifier corresponding to execution segment P2 is used as the index key, i.e., the current key = Tuple(B1, A1).

[0139] Then, the key is found in both the graph and req_graph_status. The current line is determined as line = req_graph_status [Tuple(B1, A1)], which is the second-level entry 1 mentioned above. It is checked that the execution time period x1 corresponding to the execution segment P1 already placed in the graph does not overlap with the execution time period x2 of the current slice = P2. Therefore, the execution segment P2 is stored in the second-level entry 1. The occupancy information of the execution segment P2 in the second-level entry 1 is updated in the graph, that is, the occupancy information of the time interval of the execution segment P2 in the second-level entry 1 under the first-level entry 1 is updated in the graph table. At the same time, the information of the second-level entry 1 occupied by the execution segment P2 can also be updated in the req_graph_status, that is, the second-level entry 1 stored under the first-level entry 1 of the execution segment P2 is updated in the req_graph_status table.

[0140] Then, extract execution segment P3 from execution segment group a, i.e., the current slice = P3. Here, the thread identifier slice.pid of execution segment P3 is A2, and the host identifier slice.host_id is B1. The combination of the thread identifier and the host identifier corresponding to execution segment P3 is used as the index key, i.e., the current key = Tuple(B1, A2).

[0141] If the key is not found in the graph, create a first-level entry 2 corresponding to thread A2, and create a second-level entry 2 under first-level entry 2. Set the current line as `req_graph_status [Tuple(B1, A2)]`, which is second-level entry 2. Store execution segment P3 into second-level entry 2 according to its execution time period x3. Update the graph table with the occupancy information of execution segment P3 in second-level entry 2, that is, update the time interval occupancy information of execution segment P3 in second-level entry 2 under first-level entry 2. Simultaneously, update the information of second-level entry 2 occupied by execution segment P3 in `req_graph_status`, that is, update the second-level entry 2 stored under first-level entry 2 of execution segment P3 in `req_graph_status`.

[0142] Retrieve execution segment P4 from the sequence, i.e., the current slice = P4. Here, the thread identifier slice.pid of execution segment P4 is A1, and the host identifier slice.host_id is B1. Use the combination of the thread identifier and host identifier corresponding to execution segment P2 as the index key, i.e., the current key = Tuple(B1, A1).

[0143] Then, the key is found in the graph and in req_graph_status. The current line is determined as line = req_graph_status [Tuple(B1, A1)], which is the second-level entry 1. It is checked that the execution time period x2 corresponding to the execution segment P4 already placed in the graph overlaps with the execution time period x4 of the current slice = P4. Then, all second-level entries in graph[key] (i.e., first-level entry 1) are traversed in a loop to find a second-level entry that does not overlap with the execution time period of the execution segment P4. Since there is only one second-level entry 1 under the current first-level entry 1, and the condition of non-overlapping execution time periods is not met, a second-level entry 3 can be created under the first-level entry 1. The execution segment P4 is stored in the second-level entry 3 according to the execution time period x4 of the execution segment P4. Then, the occupancy information of execution segment P4 in second-level table entry 3 is updated to the entry `graph[key][line]` in the `graph` table. This means updating the time interval occupancy information of execution segment P4 in second-level table entry 3 under first-level table entry 1 to this `graph` table. Simultaneously, the information of second-level table entry 1 occupied by execution segment P4 can also be updated to the entry `req_graph_status[key]` in `req_graph_status`. P5 can be extracted from the sequence using the above method, and the preceding execution segment corresponding to P5 can be determined; this will not be elaborated further here.

[0144] Similarly, following the above execution process, the display positions of each execution segment in execution group segment b corresponding to inference request 2 and each execution segment in execution group segment c corresponding to inference request 3 in the secondary table can be determined sequentially, which will not be elaborated here.

[0145] As described above, this method allows the current execution segment i under thread d to be preferentially stored in the second-level table entry where the previous execution segment i-1 under thread d was stored, achieving the effect of placing consecutive segments of the same thread d in the same second-level table entry. Furthermore, only when there is a conflict in this second-level table entry will other second-level table entries under the corresponding first-level table entry for thread d be traversed. If no other second-level table entry meets the non-conflict requirement, a new second-level table entry is created under the current first-level table entry to store this execution segment i.

[0146] In practical applications, performance bottleneck analysis can also be performed on the already created second-level table structure to identify the execution segments where performance bottlenecks occur, and function-level tracing can be performed on these execution segments. Specifically, combined with... Figure 7 An example is provided.

[0147] Figure 7 A flowchart illustrating another method for displaying request tracking data provided in this application embodiment is shown below. Figure 7 As shown, based on the above embodiments, the method may further include the following steps:

[0148] 701. Identify the execution segments that meet the performance bottleneck conditions from the already created secondary table structure. The secondary table structure includes multiple primary table entries and secondary table entries created under each primary table entry.

[0149] 702. Based on the execution time period of the execution segment that meets the performance bottleneck condition, collect function call information for the inference requests corresponding to the execution segment that meets the performance bottleneck condition.

[0150] 703. The first-level table structure corresponding to the generated and collected function call information.

[0151] 704. Display the first-level table structure in association with the second-level table structure.

[0152] Specifically, the execution segments that meet the performance bottleneck conditions are first identified from the already created secondary table structure. This secondary table structure includes multiple primary table entries created above and secondary table entries created under each primary table entry. Then, based on the execution time period of the execution segments that meet the performance bottleneck conditions, function call information is collected for the inference requests corresponding to the execution segments that meet the performance bottleneck conditions. The specific collection process can refer to existing function call information collection techniques and will not be elaborated here. Optionally, the performance bottleneck condition can be set as the length of the execution time period of the execution segment being greater than a set threshold.

[0153] Next, a first-level table structure corresponding to the collected function call information is generated. This first-level table structure can be generated using traditional request tracing data display methods (such as torch profiler).

[0154] Finally, the first-level table structure can be displayed in association with the second-level table structure. This association can be displayed side-by-side or in other ways, without specific limitations, as long as both table structures are visible in a visualization interface aligned with the timeline.

[0155] In this embodiment, the generation and display process of the above-mentioned first-level table structure can be specifically implemented as follows: generating a flame graph structure corresponding to the collected function call information (generating spans for each level of function, and generating a flame graph by stacking spans). The flame graph is then displayed in the second-level table structure, with the flame graph occupying a separate first-level table entry. The flame graph is associated with the corresponding request through the thread information and timeline contained in the first-level table entry name in the second-level table structure.

[0156] In summary, this embodiment identifies execution segments that meet performance bottleneck conditions from an existing secondary table structure. Based on the execution time periods of these segments, it collects function call information for the inference requests corresponding to these segments, generating a primary table structure corresponding to the collected function call information. This primary table structure is then displayed in association with the secondary table structure. This significantly simplifies performance analysis of the model inference service, allowing users to view the complete execution status of multiple inference requests and precisely display the specific execution status of segments with performance bottleneck conditions. Furthermore, the function-level tracing data collection is limited to the performance bottleneck execution segments, resulting in a limited data collection volume.

[0157] The following is combined with Figure 8 The generated secondary tables and specific application scenarios are explained in detail regarding the implementation process of this solution. In practical applications, the execution process of the large language model's inference service handling multiple inference requests is tracked and displayed. Specifically, the model's inference service adopts an inference architecture that separates the prefilling and decoding stages. One Prefill node handles the inference tasks corresponding to the prefilling stage, and one Decode node handles the inference tasks corresponding to the decoding stage. Each Prefill and Decode node has two concurrent TPs (Processing Threads), meaning two TP threads are started simultaneously to execute their respective inference tasks.

[0158] Specifically, once the model inference service begins processing the inference request, it starts tracking the request and collecting tracking data generated during the process. This tracking data includes thread identifiers and execution time periods for multiple execution segments. These segments include: launch, two tokenize segments, two wait-for-handshake-signal segments, two prefill segments, two key-value cache transfer segments, two key-value cache prefill segments, two wait-for-prefill segments, and eight decode segments. Launch indicates that the router node first selects a pair of prefill and decode nodes (Prefill and Decode nodes) to perform the inference computation corresponding to the request and distributes the client-triggered inference request to the corresponding execution stage of the selected pair of prefill and decode nodes. Tokenize indicates that the Prefill node converts the language description text in the inference request into a token sequence, and the Decode node converts the language description text in the inference request into a token sequence. `wait prealloc` indicates the execution process where the TP thread in the prefill node waits for the decode node to complete allocating video memory resources and receive the handshake signal. `Prefill` indicates the TP thread in the prefill node generates the first target token based on the token sequence. `kv cache transfer` indicates the prefill node sends the kv cache to the decode node. `kv cache prealloc` indicates the decode node allocates video memory resources. `wait prefill` indicates the decode node waits for the prefill node to complete prefilling and send the kv cache. `Decode` indicates the decode node automatically generates the next target token.

[0159] Then, the predecessor execution segments corresponding to each execution segment of the inference request are determined. Multiple first-level entries are created corresponding to the thread identifiers of the multiple execution segments. According to the second-level entry allocation principle, second-level entries are created under each of the multiple first-level entries, and the display position of the execution segment corresponding to the same thread identifier in the corresponding second-level entry is determined. The content filling direction of the second-level entries corresponds to the set time axis. The second-level entry allocation principle ensures that the execution time periods of the execution segments corresponding to the same thread identifier do not overlap in the same second-level entry, and that consecutive adjacent execution segments of the same inference request are allocated to the same second-level entry. Directed connections are established between multiple execution segments and their corresponding predecessor execution segments in the second-level entries to form the execution flow corresponding to the inference request, such as... Figure 8 As shown. The specific process of creating first-level and second-level entries, as well as the process of determining the display position of multiple execution fragments in the second-level entries, are described in the above embodiments.

[0160] The following will describe in detail one or more embodiments of a request tracking data display apparatus according to this application. Those skilled in the art will understand that these apparatuses can all be configured using commercially available hardware components through the steps taught in this solution.

[0161] Figure 9 A schematic diagram of the structure of a request tracking data display device provided in this application embodiment is shown below. Figure 9 As shown, the device includes: an acquisition module 11, a creation module 12, a determination module 13, and an establishment module 14.

[0162] The acquisition module 11 is used to acquire the tracking data generated by the model inference service when processing multiple inference requests. The tracking data includes the thread identifier and execution time period of different execution segments of the multiple inference requests.

[0163] The creation module 12 is used to create multiple first-level table entries corresponding to the thread identifiers of multiple execution segments, wherein the multiple execution segments are the execution segments corresponding to the multiple inference requests.

[0164] The determination module 13 is used to create secondary entries under the multiple primary entries according to the secondary entry allocation principle, and determine the display position of the execution segment corresponding to the same thread identifier in the corresponding secondary entry. The content filling direction of the secondary entry corresponds to the set time axis. The secondary entry allocation principle ensures that the execution time periods of the execution segments corresponding to the same thread identifier do not overlap in the same secondary entry and that consecutive adjacent execution segments of the same inference request are allocated in the same secondary entry.

[0165] The module 14 is used to establish directed connections between the multiple execution segments and their respective predecessor execution segments in the secondary table entries to form an execution flow corresponding to the same inference request. The predecessor execution segment of an execution segment in the target inference request is the previous execution segment in the target inference request whose execution time is adjacent to that of the target inference request. The target inference request is any one of the multiple inference requests.

[0166] Optionally, the tracking data also includes event information corresponding to each execution segment; the establishment module 14 is further configured to: display the event information in the secondary table entry in association with the corresponding execution segment according to the occurrence time of the event information.

[0167] Optionally, the model inference service adopts an inference architecture that separates the pre-filling stage and the decoding stage, and the tracking data also includes the node identifier to which the execution segment belongs; wherein, the device further includes a preprocessing module, which is specifically used to: group each execution segment corresponding to the target inference request according to its node identifier to obtain a first group corresponding to the pre-filling node, a second group corresponding to the decoding node, and a third group corresponding to the routing node; sort the execution segments in the first group, the second group, and the third group according to the order of their start execution time; determine that the predecessor execution segment corresponding to the first execution segment in the first group and the second group is the first execution segment in the third group; and determine the predecessor execution segments of the non-first execution segments in the first group, the second group, and the third group according to the sorting results of the non-first execution segments in the first group, the second group, and the third group and their corresponding thread identifiers.

[0168] Optionally, the preprocessing module is specifically configured to: for a target execution segment in a target group, if there is at least one execution segment in the target group that corresponds to the same thread identifier as the target execution segment, then determine the predecessor execution segment of the target execution segment as the execution segment in the at least one execution segment that is closest in order to the target execution segment, wherein the target group is any one of the first group, the second group, and the third group, and the target execution segment is any non-first execution segment in the target group; if there is no execution segment in the target group that corresponds to the same thread identifier as the target execution segment, then determine the predecessor execution segment of the target execution segment as the execution segment in the target group that is closest in order to the target execution segment.

[0169] Optionally, the preprocessing module is specifically used to: initialize the global predecessor variable to empty and the thread predecessor variable table to empty for the target group, wherein the thread predecessor variable table is used to store the thread identifier and the most recently scanned execution segment under the thread identifier; and repeatedly execute the following process until all execution segments in the target group have been scanned:

[0170] Retrieve the currently scanned execution segment from the target group, the currently scanned execution segment including the target execution segment; determine whether there is a thread identifier corresponding to the currently scanned execution segment in the thread predecessor variable table; if there is no thread identifier corresponding to the currently scanned execution segment in the thread predecessor variable table, and the global predecessor variable is empty, then determine that the predecessor execution segment of the currently scanned execution segment is empty; if there is no thread identifier corresponding to the currently scanned execution segment in the thread predecessor variable table, and the global predecessor variable is not empty, then determine that the predecessor execution segment of the currently scanned execution segment is the value of the global predecessor variable; if there is a thread identifier corresponding to the currently scanned execution segment in the thread predecessor variable table, then determine that the predecessor execution segment of the currently scanned execution segment is the execution segment corresponding to the thread identifier in the thread predecessor variable table; update the execution segment corresponding to the thread identifier of the currently scanned execution segment in the thread predecessor variable table to the currently scanned execution segment, and update the value of the global predecessor variable to the currently scanned execution segment.

[0171] Optionally, the creation module 12 is specifically used to: generate multiple execution fragment groups corresponding to the multiple inference requests, and sort the execution fragments in the multiple inference request execution fragment groups according to the order of their start execution time; traverse the execution fragments in the target execution fragment group one by one, and if there is a first-level table entry in the already created first-level table entry that corresponds to the thread identifier of the currently traversed target execution fragment, then determine the first-level table entry of the most recently stored execution fragment under the first-level table entry, wherein the target execution fragment group is any one of the multiple execution fragment groups; if the execution time period of the target execution fragment is consistent with the first-level table entry in the first-level table entry... If the execution time periods of the already stored execution segments do not overlap, then the target execution segment is stored in the first secondary table entry according to its execution time period. If the execution time period of the target execution segment overlaps with the execution time period of the execution segments already stored in the first secondary table entry, then the target execution segment is stored in the second secondary table entry under the first primary table entry according to its execution time period. The execution time periods of the execution segments already stored in the second secondary table entry do not overlap with the execution time period of the target execution segment. The second secondary table entry is either already created or newly created under the first primary table entry.

[0172] Optionally, the creation module 12 is specifically used to: if there is no first-level table entry corresponding to the thread identifier of the target execution segment in the already created first-level table entries, then create a second-level table entry corresponding to the thread identifier of the target execution segment, and create a third-level table entry under the second-level table entry, and store the target execution segment into the third-level table entry according to the execution time period of the target execution segment.

[0173] Optionally, after generating the multiple execution fragment groups corresponding to the multiple inference requests, the creation module 12 is further configured to: sort the multiple inference requests according to the order of the start execution times of the first execution fragment in the multiple execution fragment groups to obtain a request sequence; wherein, the target execution fragment group is the execution fragment group corresponding to the request currently traversed when traversing the request sequence in sequence.

[0174] Optionally, before traversing the execution segments in the target execution segment group one by one, the creation module 12 is further configured to: initialize a global view variable table for describing the occupancy of time intervals of all second-level entries under multiple created first-level entries, and initialize a request view status table for describing the occupancy of second-level entries under different first-level entries of the multiple inference requests; determine whether there is a first-level entry in the created first-level entries corresponding to the thread identifier of the currently traversed target execution segment, based on whether the thread identifier of the target execution segment exists in the global view variable table; if the thread identifier of the target execution segment exists in the request view status table, determine that the second-level entry in the request view status table corresponding to the thread identifier of the target execution segment is the first second-level entry of the most recently stored execution segment under the first-level entry.

[0175] Optionally, the establishment module 14 is further configured to: in response to the storage of the target execution segment, update the execution time period of the target execution segment to the record in the global view variable table corresponding to the thread identifier of the target execution segment and the secondary table entry to which it is stored, and update the secondary table entry to which the target execution segment is stored to the record in the request view status table corresponding to the thread identifier of the target execution segment.

[0176] Optionally, the establishment module 14 is further configured to: identify execution segments that meet performance bottleneck conditions from the already created secondary table structure, wherein the secondary table structure includes the plurality of primary table entries and secondary table entries created under each primary table entry; collect function call information for the inference requests corresponding to the execution segments that meet performance bottleneck conditions based on the execution time period of the execution segments that meet performance bottleneck conditions; generate a primary table structure corresponding to the collected function call information; and display the primary table structure in association with the secondary table structure.

[0177] Figure 9 The device shown can perform the steps described in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0178] In one possible design, the above Figure 9 The structure of the device for displaying the requested tracking data can be implemented as an electronic device, such as... Figure 10 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores a computer program, which, when executed by the processor 22, enables the processor 22 to at least implement the request tracking data display method provided in the foregoing embodiments.

[0179] The aforementioned memory 21 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0180] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0181] Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable request-tracking data display device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable request-tracking data display device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0182] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0185] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for displaying request tracking data, characterized in that, include: Obtain the tracking data generated by the model inference service when processing multiple inference requests. The tracking data includes the thread identifier and execution time period of different execution segments of the multiple inference requests. Create multiple first-level table entries corresponding to the thread identifiers of multiple execution segments, wherein the multiple execution segments are the execution segments corresponding to the multiple inference requests; According to the secondary entry allocation principle, secondary entries are created under the multiple primary entries, and the display position of the execution segment corresponding to the same thread identifier in the corresponding secondary entry is determined. The content filling direction of the secondary entry corresponds to the set time axis. The secondary entry allocation principle ensures that the execution time periods of the execution segments corresponding to the same thread identifier do not overlap in the same secondary entry and that consecutive adjacent execution segments of the same inference request are allocated in the same secondary entry. In the secondary table entries, directed connections are established between the multiple execution segments and their respective predecessor execution segments to form an execution flow corresponding to the same inference request. The predecessor execution segment of an execution segment in the target inference request is the previous execution segment in the target inference request whose execution time is adjacent to that of the execution segment. The target inference request is any one of the multiple inference requests.

2. The method according to claim 1, characterized in that, The tracking data also includes event information corresponding to each execution segment; the method further includes: In the secondary table entries, the event information is displayed in association with the corresponding execution segment based on the occurrence time of the event information.

3. The method according to claim 1, characterized in that, The model inference service adopts an inference architecture that separates the pre-filling stage and the decoding stage, and the tracking data also includes the node identifier to which the execution segment belongs; the method further includes: Each execution segment corresponding to the target inference request is grouped according to its node identifier to obtain a first group corresponding to the pre-filled node, a second group corresponding to the decoding node, and a third group corresponding to the routing node; The execution segments within the first group, the second group, and the third group are sorted according to the order of their start execution times. It is determined that the predecessor execution fragments corresponding to the first execution fragments in the first group and the second group are both the first execution fragments in the third group; Based on the sorting results of the non-first execution segments in the first group, the second group, and the third group, and their corresponding thread identifiers, the predecessor execution segments of the non-first execution segments in the first group, the second group, and the third group are determined respectively.

4. The method according to claim 3, characterized in that, The step of determining the predecessor execution segments of the non-first execution segments in the first group, the second group, and the third group based on the sorting results of the non-first execution segments in the first group, the second group, and the third group, and the corresponding thread identifiers, includes: For a target execution segment in a target group, if there is at least one execution segment in the target group that corresponds to the same thread identifier as the target execution segment, then the predecessor execution segment of the target execution segment is determined to be the execution segment that is closest to the target execution segment in the order of the at least one execution segment. The target group is any one of the first group, the second group, and the third group, and the target execution segment is any non-first execution segment in the target group. If there is no execution segment in the target group that has the same thread identifier as the target execution segment, then the predecessor execution segment of the target execution segment is determined to be the execution segment in the target group that is closest to the target execution segment in the order.

5. The method according to claim 4, characterized in that, The step of determining the predecessor execution segments of the non-first execution segments in the first group, the second group, and the third group based on the sorting results of the non-first execution segments in the first group, the second group, and the third group, and the corresponding thread identifiers, includes: For the target group, the global predecessor variable is initialized to empty and the thread predecessor variable table is initialized to empty. The global predecessor variable is used to store the most recently scanned execution fragments in the target group, and the thread predecessor variable table is used to store the thread identifier and the most recently scanned execution fragments under the thread identifier. The following process is repeated until all execution segments in the target group have been scanned: Extract the currently scanned execution segment from the target group, wherein the currently scanned execution segment includes the target execution segment; Determine whether the thread identifier corresponding to the currently scanned execution segment exists in the thread predecessor variable table; If the thread identifier corresponding to the currently scanned execution segment does not exist in the thread predecessor variable table, and the global predecessor variable is empty, then it is determined that the predecessor execution segment of the currently scanned execution segment is empty; If the thread identifier corresponding to the currently scanned execution segment does not exist in the thread predecessor variable table, and the global predecessor variable is not empty, then the predecessor execution segment of the currently scanned execution segment is determined to be the value of the global predecessor variable. If the thread identifier corresponding to the currently scanned execution segment exists in the thread predecessor variable table, then the predecessor execution segment of the currently scanned execution segment is determined to be the execution segment corresponding to the thread identifier in the thread predecessor variable table; Update the execution segment corresponding to the thread identifier of the currently scanned execution segment in the thread predecessor variable table to the currently scanned execution segment, and update the value of the global predecessor variable to the currently scanned execution segment.

6. The method according to claim 1, characterized in that, The process of creating multiple first-level entries corresponding to thread identifiers of multiple execution fragments, and creating second-level entries under each of the multiple first-level entries according to the second-level entry allocation principle, and determining the display position of the execution fragment corresponding to the same thread identifier in the corresponding second-level entry, includes: Generate multiple execution fragment groups corresponding to the multiple inference requests, and sort the execution fragments in the multiple inference request execution fragment groups according to the order of their start execution time; The execution segments in the target execution segment group are traversed one by one. If there is a first-level table entry in the already created first-level table entry that corresponds to the thread identifier of the target execution segment currently being traversed, then the first-level table entry of the most recently stored execution segment is determined under the first-level table entry. The target execution segment group is any one of the multiple execution segment groups. If the execution time period of the target execution segment does not overlap with the execution time period of the execution segment already stored in the first secondary table entry, then the target execution segment is stored in the first secondary table entry according to the execution time period of the target execution segment; If the execution time period of the target execution segment overlaps with the execution time period of the execution segment already stored in the first secondary table entry, then the target execution segment is stored in the second secondary table entry under the first primary table entry according to the execution time period of the target execution segment. The execution time period of the execution segment already stored in the second secondary table entry does not overlap with the execution time period of the target execution segment, and the second secondary table entry is either already created or newly created under the first primary table entry.

7. The method according to claim 6, characterized in that, The method further includes: If there is no first-level table entry corresponding to the thread identifier of the target execution segment in the already created first-level table entries, then a second-level table entry corresponding to the thread identifier of the target execution segment is created, and a third-level table entry is created under the second-level table entry. The target execution segment is stored in the third-level table entry according to the execution time period of the target execution segment.

8. The method according to claim 6, characterized in that, After generating the multiple execution fragment groups corresponding to the multiple inference requests, the method further includes: The multiple inference requests are sorted according to the order of the start execution times of the first execution segment in the multiple execution segment groups to obtain a request sequence; The target execution fragment group is the execution fragment group corresponding to the currently traversed request when the request sequence is traversed sequentially.

9. The method according to claim 7, characterized in that, Before traversing the execution segments in the target execution segment group one by one, the method further includes: Initialize a global view variable table to describe the occupancy of time intervals for all second-level entries under multiple first-level entries that have been created, and initialize a request view status table to describe the occupancy of second-level entries for the multiple inference requests under different first-level entries. Based on whether the thread identifier of the target execution segment exists in the global view variable table, determine whether there is a first-level table entry in the already created first-level table entries that corresponds to the thread identifier of the currently traversed target execution segment; The step of determining the first and second level entries of the execution fragment most recently stored under the first level entry includes: If the thread identifier of the target execution fragment exists in the request view status table, then the second-level entry in the request view status table corresponding to the thread identifier of the target execution fragment is determined to be the first second-level entry of the execution fragment most recently stored under the first-level entry.

10. The method according to claim 9, characterized in that, The method further includes: In response to the storage of the target execution segment, the execution time period of the target execution segment is updated in the record corresponding to the thread identifier of the target execution segment and the second-level table entry in the global view variable table, and the second-level table entry in which the target execution segment is stored is updated in the record corresponding to the thread identifier of the target execution segment in the request view status table.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: From the already created secondary table structure, identify the execution segments that meet the performance bottleneck conditions. The secondary table structure includes the multiple primary table entries and the secondary table entries created under each primary table entry. Based on the execution time period of the execution segment that meets the performance bottleneck condition, function call information is collected for the inference request corresponding to the execution segment that meets the performance bottleneck condition. The first-level table structure corresponding to the generated and collected function call information; The first-level table structure is displayed in association with the second-level table structure.

12. An electronic device, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform a method for displaying request trace data as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor of an electronic device, causes the processor to perform the method for displaying request tracking data as described in any one of claims 1 to 11.

14. A computer program product, characterized in that, include: A computer program or instruction that, when executed by a processor of an electronic device, causes the processor to perform the method of displaying requested trace data as described in any one of claims 1 to 11.

Images