Method, device and equipment for acquiring reasoning performance data of large model reasoning cluster

Abstract

The invention discloses a method, a device and equipment for obtaining reasoning performance data of a large model reasoning cluster, and relates to the technical field of artificial intelligence. The method comprises the following steps: in the presence of a reasoning request, aiming at each computing node in a target large model reasoning cluster, calculating the reasoning performance data of the target large model reasoning cluster; determining a reasoning request initiation position, a reasoning request reply completion position, a first token return position and a reasoning total time length; and obtaining reasoning performance data according to the request initiation number of the obtained reasoning request initiation position, the throughput of the reasoning request completion reply position in unit time, the processing duration of the first token and the processing duration of each token. Therefore, the reasoning performance information of all the computing nodes of the cluster can be obtained by counting the request initiating number, the throughput capacity and the time delay of each computing node, the problem that the cluster reasoning performance data is difficult to accurately obtain is solved, and the technical effect of efficiently and accurately monitoring and managing the large model reasoning cluster performance is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and equipment for acquiring reasoning performance data of a large model reasoning cluster. Background Art

[0002] With the development of artificial intelligence technology, the scale of AI (Artificial Intelligence) models is increasing, the scale of model parameters and network structure are constantly upgrading, and the challenges faced by large model development, training and reasoning deployment are becoming increasingly severe. A single computing node often cannot meet the computing power requirements of large-scale model real-time reasoning, so it is necessary to deploy reasoning clusters to provide sufficient parallel processing capabilities. Monitoring the performance indicators of AI reasoning clusters during the reasoning phase is crucial to ensuring the quality and efficiency of services.

[0003] In related technologies, the GPU (Graphics Processing Unit) boards launched by major chip manufacturers mainly come with board information collection tools similar to NVIDIA's nvidia-smi, which mainly collect the underlying information of the single-node board hardware chip as a monitoring indicator of inference performance, such as video memory utilization, temperature, GPU utilization, etc. However, the cluster inference performance data obtained by this method is not very accurate, and is not the core indicator information for inference cluster inference performance monitoring, which needs to be improved urgently. Summary of the invention

[0004] The present invention provides a method, device and equipment for obtaining reasoning performance data of a large model reasoning cluster, so as to at least solve the problem that cluster reasoning performance data is difficult to obtain accurately in the related technology, and achieve the technical effect of efficiently and accurately monitoring and managing the performance of a large model reasoning cluster.

[0005] The present invention provides a method for acquiring inference performance data of a large model inference cluster, comprising: Determine whether there is an inference request; If there is an inference request, determine the inference request initiation location, the inference request completion reply location, the first token return location, and the total inference duration for each computing node in the target large model inference cluster; Get the number of requests initiated at the inference request initiation location, the throughput per unit time at the inference request completion reply location, the processing time of the first token, and the processing time of each token. And get the inference performance data of each computing node based on the number of requests initiated at the inference request initiation location, the throughput per unit time at the inference request completion reply location, the processing time of the first token, and the processing time of each token.

[0006] The present invention also provides a device for acquiring inference performance data of a large model inference cluster, comprising: A judgment module, used to judge whether there is an inference request; The determination module is used to determine the inference request initiation location, the inference request completion reply location, the first token return location and the total inference time for each computing node in the target large model inference cluster when there is an inference request; The acquisition module is used to obtain the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token, and obtain the inference performance data of each computing node based on the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token.

[0007] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for acquiring inference performance data of a large model inference cluster when executing the computer program.

[0008] The present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned methods for obtaining inference performance data of a large model inference cluster are implemented.

[0009] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for acquiring inference performance data of a large model inference cluster.

[0010] Through the present invention, when there is an inference request, for each computing node in the target large model inference cluster, the inference request initiation location, the inference request completion reply location, the location where the first token is returned, and the total inference time are determined; according to the number of requests initiated at the inference request initiation location, the throughput of the inference request completion reply location per unit time, and the processing time of each token, the inference performance data of each computing node is obtained. Therefore, the problem that the cluster inference performance data is difficult to obtain accurately can be solved, and the technical effect of efficiently and accurately monitoring and managing the performance of the large model inference cluster can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A flowchart of a method for acquiring inference performance data of a large model inference cluster provided by an embodiment of the present invention; Figure 2 A schematic diagram of a preset multi-level summary structure provided in an embodiment of the present invention; Figure 3 A schematic diagram of a visualization display of delay provided by an embodiment of the present invention; Figure 4 A schematic diagram of throughput visualization provided by an embodiment of the present invention; Figure 5 A schematic diagram of a visual display of the number of requests provided by an embodiment of the present invention; Figure 6 A schematic diagram of a GPU utilization visualization provided by an embodiment of the present invention; Figure 7 A schematic diagram of the process architecture of a method for acquiring inference performance data of a large model inference cluster provided by an embodiment of the present invention; Figure 8 A block diagram of an inference performance data acquisition device for a large model inference cluster provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0013] Figure numerals: 10 - device for acquiring inference performance data of a large model inference cluster, 100 - judgment module, 200 - determination module, 300 - acquisition module; 901 - memory, 902 - processor, 903 - communication interface. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0017] An embodiment of the present invention provides a method for acquiring reasoning performance data of a large model reasoning cluster, and the method is described in detail in conjunction with the execution flow of the method for acquiring reasoning performance data of a large model reasoning cluster.

[0018] Figure 1 It is a flowchart of a method for acquiring inference performance data of a large model inference cluster according to an embodiment of the present invention.

[0019] For example, Figure 1 As shown, the method for obtaining the inference performance data of the large model inference cluster includes the following steps: In step S101, it is determined whether there is an inference request.

[0020] It is understandable that an inference request refers to a computing task performed on a large model, which generally includes input data and expected output results. The embodiment of the present invention involves obtaining inference performance data of a large model inference cluster, and therefore, needs to be performed when there is an inference request.

[0021] In step S102, if there is an inference request, the location where the inference request is initiated, the location where the inference request is completed and replied, the location where the first token is returned, and the total inference time are determined for each computing node in the target large model inference cluster.

[0022] Among them, the inference request initiation location refers to the code location where the user or system issues the inference request. This location can be a specific function call or a key point in the program execution process, which marks the beginning of the inference process. The inference request completion reply location refers to the code location where the inference result is returned. This location is usually pre-set by the developer when writing the program to ensure that the inference result can be correctly received and used. It may be the return value of a function, or it may be an assignment operation of a variable, or any other code point that can receive data. In the location where the first token is returned, the token usually refers to the smallest unit of the model output (divided according to actual conditions, not specifically limited here). The location where the first token is returned refers to the location where the first output unit is returned. The total inference time refers to the total time consumed from the initiation of the inference request to the return of all the final inference results.

[0023] That is to say, when there is an inference request, the embodiment of the present invention can determine the inference request initiation location, the inference request completion reply location, the first token return location and the total inference time of each computing node in the target large model inference cluster by deeply searching the code.

[0024] In step S103, the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token and the processing time of each token are obtained, and the inference performance data of each computing node is obtained based on the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token and the processing time of each token.

[0025] It is understandable that throughput is an indicator of the processing capacity of the inference cluster, which refers to the amount of work that can be completed in a unit of time, for example, the number of requests processed in a unit of time. The processing time of the first token and the processing time of each token are the embodiment of latency, reflecting the speed at which the inference cluster processes a single task.

[0026] Specifically, after determining the inference request initiation location, the inference request completion reply location, the first token return location, and the total inference time of each computing node in the target large model inference cluster, the embodiment of the present invention can deploy the Prometheus Node Exporter (a lightweight data collection agent, Prometheus is an open source monitoring and alarm toolkit) tool on each computing node in the target large model inference cluster to collect some specific indicator data, that is, the number of requests initiated at the inference request initiation location, the throughput per unit time at the inference request completion reply location, the processing time of the first token, and the processing time of each token. The Prometheus client library is embedded in the application code of the inference engine (such as vllm, fastchat, etc.) to record indicator data related to the inference request. Through the comprehensive analysis of these specific indicator data, the inference performance data of each computing node can be obtained, and the inference performance data can reflect the ability of the computing node to process inference tasks, which includes but is not limited to the throughput, latency (i.e., processing time) and number of requests of each computing node. Key performance indicators.

[0027] It is understandable that large-scale AI clusters may be distributed in different geographical locations or in the same computer room (e.g. Figure 2 As shown in the figure, Mn represents the nth child management node (optional); Nn represents the nth reasoning group; n represents a positive number; the processor refers to the GPU, and the n of the GPU depends on the actual situation, and the minimum n is 1). By adopting a distributed monitoring agent and deploying a lightweight data collection agent (such as Prometheus NodeExporter) on each computing node, accurate data can be efficiently collected from decentralized sources to ensure real-time and low-latency data collection. In addition, in order to unify the data format and facilitate subsequent processing, standardized protocols (such as Prometheus HTTP API (Hyper Text Transfer Protocol Application Programming Interface)) can also be used to process the data to ensure data consistency and compatibility.

[0028] For ease of understanding, the following describes in detail how to obtain the number of inference request initiation locations, the throughput per unit time at inference request completion reply locations, and the processing time for each token.

[0029] As a possible implementation method, in some embodiments, the number of requests initiated at the inference request initiation location, the throughput of the inference request completion reply location per unit time, and the processing time of each token are obtained, including: based on the request number recording code added at the inference request initiation location, the number of requests initiated at the inference request initiation location is obtained; based on the throughput recording code added at the inference request completion reply location, the number of input tokens and the number of output tokens are recorded, and the throughput of the inference request completion reply location per unit time is calculated based on the number of input tokens and the number of output tokens; based on the location where the first token is returned, the processing time of the first token is recorded, and the processing time of each token is calculated using the total inference time.

[0030] Understandably, large-scale AI clusters have heterogeneous hardware (i.e. hardware of different types and vendors) and run multiple services. How to efficiently collect accurate inference performance data (such as throughput, latency, number of requests, etc.) from these dispersed sources is an important issue.

[0031] Based on this, the embodiment of the present invention can make code-level modifications on the deployed inference engine, that is, add code (request number recording code) at the inference request initiation position to record the initiation of each request, so that the number of requests initiated within a certain period of time can be counted; add code (throughput recording code) at the inference request completion reply position to record the number of input and output tokens, so as to calculate the number of tokens processed per unit time, that is, throughput; record the delay at the location where the first token is returned, which usually refers to the time from the initiation of the request to the receipt of the first token (that is, the processing time of the first token), which can be used to evaluate the response speed of the inference engine. By recording the total inference time and dividing it by the total number of tokens processed, the average delay (processing time) of each token processing can be obtained. In addition, other necessary data can be recorded and passed to Prometheus for storage and recording.

[0032] Therefore, by searching and modifying the code, we can efficiently obtain the core indicator information for monitoring the inference performance, namely, the number of requests initiated at the inference request initiation location, the throughput per unit time, and the processing time of each token. These indicator information helps operation and maintenance personnel or developers to better understand the system performance and make targeted optimizations and adjustments.

[0033] As a possible implementation method, in other embodiments, the inference performance data of each computing node is obtained according to the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token, including: based on a preset multi-level summary structure, determining at least one computing node corresponding to each level in the target large model inference cluster; sequentially obtaining the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token of at least one computing node corresponding to each level within a preset time, to obtain the inference performance data of at least one computing node corresponding to each level; merging the inference performance data of at least one computing node corresponding to each level to obtain the inference performance data of each computing node.

[0034] It is understandable that as the scale of the inference cluster grows, the amount of data will also increase, which puts higher requirements on the real-time and scalability of the monitoring system.

[0035] Based on this, the embodiment of the present invention can construct a monitoring platform based on multi-level aggregation (i.e., a preset multi-level aggregation structure) for ultra-large-scale reasoning clusters, and clarify at least one computing node corresponding to each level. Subsequently, data is collected for the computing nodes corresponding to each level in turn, including the number of requests initiated at the inference request initiation location of each computing node within a preset time range, the throughput per unit time of the reasoning request completion reply location, the processing time of the first token, and the processing time of each token, etc., so as to obtain the reasoning performance data of at least one computing node corresponding to each level. By using the data summarized by the previous level to pass it to the next level to complete data stream transmission and integration, the reasoning performance data of each computing node can be obtained.

[0036] Therefore, by utilizing the preset multi-level aggregation structure to pass the information summarized at the previous level to the next level to complete data flow transmission and integration, it is easy to cope with large amounts of data aggregation needs and realize the aggregation and analysis of inference performance data of computing nodes at different levels.

[0037] Furthermore, in some embodiments, before determining at least one computing node corresponding to each level in the target large model reasoning cluster based on a preset multi-level aggregation structure, it also includes: judging whether the scale of the target large model reasoning cluster is greater than a preset condition; if the scale of the target large model reasoning cluster is greater than the preset condition, dividing the target large model reasoning cluster into a preset multi-level aggregation structure.

[0038] Specifically, before aggregating the inference performance data of each computing node, the monitoring system can also check whether the scale of the inference cluster exceeds the preset conditions, that is, whether the scale of the target large model inference cluster is too large. If it exceeds, then this huge inference cluster can be divided into multiple levels of summary structure. In each level, at least one computing node will be determined to facilitate subsequent processing and inference tasks. Conversely, smaller inference clusters do not need to be divided into multiple levels, and can be combined with distributed monitoring to cope with large amounts of data aggregation needs.

[0039] Therefore, a monitoring platform based on multi-level aggregation (i.e., a preset multi-level aggregation structure) is built for ultra-large-scale clusters, and the cluster is divided into a multi-level aggregation structure, which can more effectively monitor and manage the inference performance of the cluster.

[0040] In addition, in some embodiments, after obtaining the reasoning performance data of each computing node according to the number of requests initiated at the reasoning request initiation location, the throughput of the reasoning request completion reply location per unit time, the processing time of the first token and the processing time of each token, it also includes: sending the reasoning performance data of each computing node to Consul Server according to a first preset period, and using a preset time series database to hierarchically store the reasoning performance data of each computing node; configuring the corresponding visualization panel according to monitoring requirements to display the reasoning performance data of different dimensions.

[0041] That is to say, in an embodiment of the present invention, the acquired inference performance data can be periodically sent to Consul Server via the HTTP protocol for centralized storage and processing. Consul Server is a distributed, highly available, horizontally scalable service discovery tool that can efficiently store and query large-scale time series data.

[0042] Since large model inference clusters generate a huge amount of performance data, especially under high load conditions, data such as throughput, latency, and number of requests are generated at a very high frequency. How to efficiently store this massive data and ensure data integrity and accessibility is a key issue. Traditional relational databases may perform poorly when storing and querying large-scale time series data.

[0043] Based on this, the embodiment of the present invention can also use a preset time series database to store the reasoning performance data of each computing node. The database can be optimized specifically for time series data, so as to efficiently process a large amount of data with timestamps.

[0044] In addition, the embodiment of the present invention can also use Grafana as a data visualization tool and integrate it with ConsulServer. By creating multiple dashboards on the Grafana platform, monitoring data of various dimensions are displayed, thereby providing users with an intuitive and easy-to-understand monitoring information display interface.

[0045] Therefore, through periodic data updates, time series database storage, and configuration of visualization panels, real-time monitoring and management of large model inference cluster performance can be achieved, improving the transparency and operability of the system.

[0046] Next, we will introduce in detail how to use the preset time series database to hierarchically store the inference performance data of each computing node.

[0047] As a possible implementation method, in some embodiments, a preset time series database is used to hierarchically store the reasoning performance data of each computing node, including: based on a preset frequency division condition, dividing the reasoning performance data of all computing nodes into a first reasoning performance data group and a second reasoning performance data group, wherein the usage frequency of the first reasoning performance data group is higher than the usage frequency of the second reasoning performance data group; storing the first reasoning performance data group in a first performance storage medium area of ​​the preset time series database, and storing the second reasoning performance data group in a second performance storage medium area of ​​the preset time series database, wherein the performance of the first performance storage medium area is higher than the performance of the second performance storage medium area.

[0048] Among them, the preset frequency division condition refers to a method of dividing data according to the frequency of data access or use. High-frequency use means that the data needs to be read and responded to quickly, while low-frequency data has relatively low requirements for response speed. The first performance storage medium area refers to a storage area with high-performance characteristics, which usually has a faster read and write speed and is used to store data with high access frequency to ensure efficient data processing. Compared with the first performance storage medium area, the second performance storage medium area has lower storage medium performance and is usually used to store data with low access frequency and not particularly strict performance requirements.

[0049] Specifically, based on the preset frequency division condition (i.e., the usage frequency of the inference performance data), the data can be divided into two groups (i.e., the first inference performance data group and the second inference performance data group), and they are stored in different storage media areas to achieve hierarchical storage. That is, the first inference performance data group is stored in a storage media area with better performance, i.e., the first performance storage media area (such as a solid-state hard drive), because of its higher usage frequency, for quick query and analysis; while the second inference performance data group is stored in a storage media area with relatively lower performance, i.e., the second performance storage media area (such as a tape library or a large-capacity mechanical hard drive), because of its lower usage frequency, for long-term archiving and historical data analysis.

[0050] Therefore, by dividing the inference performance data into two groups according to the frequency of use, and storing the data with higher frequency of use in the storage medium with lower performance, and storing the data with lower frequency of use in the storage medium with lower performance, it can ensure that the data accessed frequently can be responded to quickly while effectively reducing the storage cost.

[0051] Furthermore, in some embodiments, when displaying reasoning performance data of different dimensions, it also includes: based on the reasoning performance data of each computing node, determining whether there is reasoning performance data that meets the preset alarm conditions; if there is reasoning performance data that meets the preset alarm conditions, an alarm reminder is issued based on the reasoning performance data that meets the preset alarm conditions.

[0052] That is to say, for large model inference clusters, it is necessary to promptly discover abnormal changes in inference performance indicators in order to respond quickly. Based on this, the embodiment of the present invention sets an alarm rule in Prometheus, that is, by judging whether the inference performance data of each computing node meets the preset alarm conditions, to check whether there is an abnormality in the inference performance data of each computing node. If there is inference performance data that meets the preset alarm conditions, it can indicate that there is a problem with the performance of the computing node, such as slow processing speed or errors. At this time, Prometheus can trigger an alarm to remind relevant personnel or systems to take action to solve the abnormal problem.

[0053] Therefore, by identifying whether the inference performance data is abnormal based on preset alarm conditions, abnormal changes in the inference performance indicators can be discovered in time for rapid response.

[0054] How to set reasonable alarm thresholds and how to promptly notify relevant personnel when indicators are abnormal is an important technical issue. Because performance indicators may change slightly due to normal business fluctuations, it is crucial to correctly distinguish between normal fluctuations and true abnormalities.

[0055] The following describes in detail how an embodiment of the present invention determines whether there is reasoning performance data that meets a preset alarm condition based on the reasoning performance data of each computing node.

[0056] As a possible implementation method, in some embodiments, based on the inference performance data of each computing node, it is determined whether there is inference performance data that meets the preset alarm condition, including: calculating at least one statistical feature of the inference performance data of each computing node, and determining whether each statistical feature is not within the corresponding preset interval; when at least one statistical feature is not within the corresponding preset interval, it is determined that there is inference performance data that meets the preset alarm condition.

[0057] Specifically, dynamic alarm thresholds are set in the monitoring system, and by statistically analyzing historical data, that is, calculating at least one statistical feature (such as mean, standard deviation, etc.) of the inference performance data of each computing node, a reasonable alarm threshold range is set for inference performance indicators such as throughput, latency, and number of requests. For example, for the indicator of the number of requests initiated at the inference request initiation location (number of requests), the upper limit of the corresponding preset interval can be set to the mean plus 3 times the standard deviation, and the lower limit can be set to the mean minus 3 times the standard deviation. When the indicator of the number of requests initiated at the inference request initiation location (number of requests) is not within the preset interval, it can be determined that there is inference performance data that meets the preset alarm conditions, that is, there is an abnormality.

[0058] Therefore, by setting a reasonable alarm threshold range, it is possible to accurately identify whether the inference performance data is abnormal, achieve real-time monitoring of cluster performance, and provide data support for operation and maintenance personnel to quickly gain insight into abnormal situations.

[0059] Furthermore, in some embodiments, after an alarm reminder is issued based on the inference performance data of preset alarm conditions, it also includes: generating alarm location information based on the inference performance data that meets the preset alarm conditions, so that operation and maintenance personnel can locate the abnormal point based on the alarm location information.

[0060] That is to say, by analyzing and processing the inference performance data that meets the preset alarm conditions, accurate alarm location information can be generated and automatically printed out. The information contains detailed abnormal indicator data, thereby helping operation and maintenance personnel to quickly locate abnormal points and perform targeted performance optimization and adjustments.

[0061] Therefore, based on the generated alarm location information, operation and maintenance personnel can quickly locate abnormal points, thereby improving the efficiency and accuracy of problem solving.

[0062] Optionally, in some embodiments, the monitoring requirement is at least one of throughput monitoring, latency monitoring, and request quantity monitoring.

[0063] That is to say, in an embodiment of the present invention, monitoring requirements may include three types: throughput monitoring, latency monitoring, and request quantity monitoring. Throughput monitoring refers to the rate at which the inference group processes tasks, that is, how much work the system can complete within a certain period of time, which corresponds to the throughput per unit time at the location where the inference request is completed and replied in the inference performance data. Latency monitoring is to observe and record the time required for the inference cluster to respond to requests, including delays in various aspects such as network transmission and data processing. The shorter the latency, the faster the response speed, which corresponds to the processing time of the first tokens and the processing time of each token in the inference performance data. Request quantity monitoring refers to recording the number of requests received by the inference cluster within a specific period of time, which can reflect the load situation and help understand the frequency of use and busyness of the inference cluster, which corresponds to the number of requests initiated at the location where the inference request is initiated in the inference performance data.

[0064] By monitoring the throughput, latency, and number of requests of the inference cluster, you can fully understand the performance of the inference cluster, promptly identify and resolve performance bottlenecks, and optimize system configuration, thereby improving overall performance and user experience.

[0065] It is understandable that it is challenging to present complex reasoning performance data such as throughput, latency, and number of requests to relevant personnel in an intuitive and easy-to-understand manner. It is necessary to consider how to choose the appropriate visualization chart type (such as line charts, bar charts, pie charts, etc.) to facilitate users to analyze data.

[0066] The following describes in detail how the embodiment of the present invention configures a corresponding visualization panel according to monitoring requirements to display reasoning performance data of different dimensions.

[0067] As a possible implementation method, in some embodiments, a corresponding visualization panel is configured according to the monitoring requirements to display reasoning performance data of different dimensions, including: when the monitoring requirement is throughput monitoring, using a preset line chart to display the throughput changes of the target large model inference cluster and the throughput changes of each computing node in different time intervals; when the monitoring requirement is latency monitoring, using a preset line chart or a preset bar chart to display the average latency of different types of reasoning requests and the latency of at least one preset percentile; when the monitoring requirement is request quantity monitoring, using a preset line chart or a preset pie chart to display the correspondence between the number of inference requests initiated and the time.

[0068] Specifically, using modern data visualization tools (such as Grafana), which support multiple data sources, including the above-mentioned preset time series database, etc., different visualization panels can be configured according to different monitoring requirements. For example, for throughput monitoring, the embodiment of the present invention draws a line graph, such as Figure 3As shown in the figure (the vertical axis is the number of tokens, and the horizontal axis is the acquisition time), this figure can show the throughput changes of the inference cluster as a whole and each computing node within it in different time periods. Through this figure, you can very intuitively understand the overall processing capacity of the cluster and how the load is balanced between each computing node. For latency monitoring, the embodiment of the present invention draws a line graph (or a bar graph), such as Figure 4 As shown (the vertical axis is time (in s or ms), the horizontal axis is the acquisition time, Figure 4 (a) is a schematic diagram of E2E (End-to-End Delay). Figure 4 (b) Schematic diagram of the delay for each token. Figure 4 (b) is a schematic diagram of the first token delay), which can show the average delay of different types of requests and the P50 delay (i.e., the delay when 50% of the request response time does not exceed the average value), the P90 delay (i.e., the delay when 90% of the request response time does not exceed the average value), the P95 delay (i.e., the delay when 95% of the request response time does not exceed the average value), the P99 delay (i.e., the delay when 99% of the request response time does not exceed the average value), etc. For request quantity monitoring, the embodiment of the present invention draws a line graph (it can also be a circular graph), such as Figure 5 As shown in the figure (the vertical axis is the number of requests (due to the concurrent design, the number of requests is slightly different here, and the collection point of the inference performance data can be adjusted as needed), the horizontal axis is the collection time), this figure can show the change trend of the number of inference requests initiated over time, and the number of requests in different business modules and different time periods can be viewed through the filtering function of Grafana. In addition, the visualization panel of the embodiment of the present invention can also display the GPU utilization, such as Figure 6 As shown (the vertical axis is the percentage of GPU utilization, and the horizontal axis is the acquisition time).

[0069] For example, through Grafana's interactive function, users can filter data by clicking on chart elements, such as viewing the throughput data of a computing node within a specific time period; they can also view the details of the data through the zoom function, such as zooming in to view the latency fluctuations during high-load periods.

[0070] Therefore, by configuring the corresponding visualization panel, the performance data of the large model inference cluster can be effectively monitored and intuitively displayed, thereby realizing real-time monitoring, analysis and optimization of cluster performance and improving the stability and efficiency of the system.

[0071] Furthermore, in some embodiments, after using a preset time series database to hierarchically store the reasoning performance data of each computing node, it also includes: setting a remote storage configuration; backing up the reasoning performance data of each computing node to the remote storage configuration according to a second preset period.

[0072] It is understandable that in order to prevent data loss and improve system reliability, remote storage can be configured on the basis of Consul Server to regularly back up inference performance data to other server storage. After configuring remote storage, system administrators can flexibly set backup policies, such as backup frequency, backup data retention time, etc., to ensure data persistence and security.

[0073] Therefore, remote storage configuration can be used to achieve off-site data backup, thereby ensuring that data is not lost when a local failure or disaster occurs, thereby improving data security and reliability.

[0074] To facilitate those skilled in the art to further understand the method for obtaining inference performance data of a large model inference cluster proposed in an embodiment of the present invention, it is further elaborated below in conjunction with a specific embodiment.

[0075] Specifically, if Figure 7 As shown ( Figure 7 Nn in the figure represents the nth node), the method for obtaining the inference performance data of the large model inference cluster may further include the following steps: Step 1: Install and deploy large model inference images (images that are generally not used according to different GPU hardware), that is, package a pre-configured software environment into an image file in the computer system (the image file can be loaded and run in different computing environments without reconfiguring the environment, thereby achieving rapid deployment), such as Prometheus images (i.e. monitoring and alarm toolkits), Grafana images (i.e. data visualization tools), Consul images (i.e. data storage tools) and other offline images, and increase the collection of inference performance data in the inference engine.

[0076] Step 2, the server startup script is designed for the management node and is executed only on the management node. Before starting, you need to carefully configure the IP address of the management node and other key parameters. Once the configuration is complete, you can start the master service (master task), Prometheus service, Consul service, and Grafana service on the management node. Among them, the main responsibility of the master service is to start and manage the inference service on the node. It is worth noting that if the management node is not equipped with GPU board resources, the inference service is not necessary and can be not started; the Prometheus service is used to collect the inference performance data of this node. However, if the management node does not have GPU resources, the collection task of the Prometheus service will not be executed; the Consul service is used to summarize the inference performance data of the entire cluster to provide support for performance monitoring and management of the entire system; the Grafana service is used to visualize the inference performance data summarized by Consul for easy monitoring and analysis.

[0077] Step 3, the client startup script is designed for computing nodes and is executed only on computing nodes. First, you need to correctly configure the IP address of the Node node, the IP address of the management node, and some other necessary parameters. Once the configuration is complete, you can start the node service, Prometheus service, and Grafana service on the Node node. Among them, the node service is used to start the inference service of the Node node. This inference service will collect inference performance data in real time as the inference request proceeds according to the previously added code; the role of the Prometheus service is to collect the inference performance data of this node, and will send this data to Consul at regular intervals; the Grafana service is used to visualize the inference performance data of this node, so that users can intuitively see various indicators of inference performance.

[0078] Step 4: Prometheus on each node will automatically collect the inference performance data (such as throughput, latency, number of requests, etc.) collected from the model service at regular intervals according to the configuration settings, and automatically store it on the local server in time series.

[0079] Step 5: When Prometheus of each node collects inference performance data, it can automatically determine whether an alarm needs to be issued based on the set threshold (for example, the delay exceeds 10 seconds).

[0080] Step 6: The Prometheus data of each node can be registered with the Consul service, and the required data can be uploaded to the Consul Server at regular intervals according to the configuration settings.

[0081] In step 7, the Grafana service on the management node collects the inference performance data stored by Consul Server for visualization. It can display the inference performance data of the entire inference cluster or the inference performance data of a single specified node.

[0082] Therefore, the embodiments of the present invention have the following beneficial effects: (1) By deploying image files, the required software environment can be quickly deployed on different GPU hardware, reducing the time and error probability of manual configuration.

[0083] (2) Performance optimization: Through real-time monitoring of throughput, latency, and number of requests, the operation and maintenance team can promptly identify performance bottlenecks in the inference cluster and improve the throughput of the inference cluster and reduce average latency by optimizing model parameters and adjusting computing resource allocation, thereby greatly improving the overall performance of the inference cluster.

[0084] (3) Rapid fault location and resolution: In the event of a sudden service failure, the operation and maintenance team can quickly find that the GPU of a certain node has failed through the visual monitoring system, causing the throughput of the node to drop sharply and the latency to increase significantly. By replacing the failed node in time, the inference cluster can resume normal service in a short time, avoiding long-term business interruption.

[0085] (4) Cost optimization: Based on the inference performance data, the resources of the inference cluster were reasonably adjusted and optimized, and some long-term idle nodes were shut down, saving a lot of hardware procurement and operating costs. At the same time, by optimizing resource allocation, the resource utilization rate was improved, allowing the inference cluster to process more business requests with the same hardware resources.

[0086] According to the method for obtaining the inference performance data of the large model inference cluster proposed in an embodiment of the present invention, in the presence of an inference request, for each computing node in the target large model inference cluster, the inference request initiation location, the inference request completion reply location, the location where the first token is returned, and the total inference time are determined; the inference performance data of each computing node is obtained according to the number of requests initiated at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token. Therefore, the problem that the cluster inference performance data is difficult to obtain accurately can be solved, and the technical effect of efficiently and accurately monitoring and managing the performance of the large model inference cluster can be achieved.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0088] An embodiment of the present invention also provides a device for acquiring inference performance data of a large model inference cluster.

[0089] Figure 8 It is a block diagram of an apparatus for acquiring inference performance data of a large model inference cluster according to an embodiment of the present invention.

[0090] like Figure 8 As shown, the inference performance data acquisition device 10 of the large model inference cluster includes: a judgment module 100, a determination module 200 and an acquisition module 300.

[0091] The determination module 100 is used to determine whether there is an inference request; Determination module 200, for determining the inference request initiation location, the inference request completion reply location, the first token return location and the total inference time for each computing node in the target large model inference cluster when there is an inference request; The acquisition module 300 is used to obtain the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token and the processing time of each token, and obtain the inference performance data of each computing node based on the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token and the processing time of each token.

[0092] Further, in some embodiments, the obtaining module 300 includes: A first obtaining unit, configured to obtain the number of requests initiated at the inference request initiation location based on a request number record code added at the inference request initiation location; A first calculation unit is used to record the number of input tokens and the number of output tokens based on the throughput recording code added to the reasoning request completion reply position, and calculate the throughput of the reasoning request completion reply position in unit time based on the number of input tokens and the number of output tokens; The second calculation unit is used to record the processing time of the first token based on the position returned by the first token, and calculate the processing time of each token using the total inference time.

[0093] Further, in some embodiments, the obtaining module 300 includes: A determination unit, configured to determine at least one computing node corresponding to each level in the target large model inference cluster based on a preset multi-level summary structure; The second obtaining unit is used to sequentially obtain the number of requests initiated at the inference request initiation position of at least one computing node corresponding to each level within a preset time, the throughput of the inference request completion reply position within a unit time, the processing time of the first token and the processing time of each token, and obtain the inference performance data of at least one computing node corresponding to each level; The third obtaining unit is used to obtain the reasoning performance data of each computing node by merging the reasoning performance data of at least one computing node corresponding to each level.

[0094] Furthermore, in some embodiments, before determining at least one computing node corresponding to each level in the target large model reasoning cluster based on the preset multi-level summary structure, the determining unit is further configured to: Determine whether the size of the target large model inference cluster is greater than the preset condition; When the size of the target large model reasoning cluster is larger than a preset condition, the target large model reasoning cluster is divided into a preset multi-level summary structure.

[0095] Further, in some embodiments, after obtaining the inference performance data of each computing node according to the number of inference request initiation locations, the throughput of inference request completion reply locations per unit time, the processing time of the first token, and the processing time of each token, the obtaining module 300 further includes: A storage unit, used to send the reasoning performance data of each computing node to ConsulServer according to a first preset period, and hierarchically store the reasoning performance data of each computing node using a preset time series database; The display unit is used to configure the corresponding visualization panel according to monitoring requirements and display the reasoning performance data of different dimensions.

[0096] Furthermore, in some embodiments, the storage unit is specifically used to: Based on a preset frequency division condition, the reasoning performance data of all computing nodes are divided into a first reasoning performance data group and a second reasoning performance data group, wherein the usage frequency of the first reasoning performance data group is higher than the usage frequency of the second reasoning performance data group; The first reasoning performance data group is stored in a first performance storage medium area of ​​a preset time series database, and the second reasoning performance data group is stored in a second performance storage medium area of ​​a preset time series database, wherein the performance of the first performance storage medium area is higher than the performance of the second performance storage medium area.

[0097] Furthermore, in some embodiments, when displaying the reasoning performance data of different dimensions, the display unit further includes: A judgment subunit, used to judge whether there is reasoning performance data that meets a preset alarm condition based on the reasoning performance data of each computing node; The alarm subunit is used to issue an alarm reminder based on the inference performance data that meets the preset alarm conditions when there is inference performance data that meets the preset alarm conditions.

[0098] Furthermore, in some embodiments, the judging subunit is specifically configured to: Calculate at least one statistical feature of the inference performance data of each computing node, and determine whether each statistical feature is not within a corresponding preset interval; When at least one statistical feature is not within the corresponding preset interval, it is determined that there is inference performance data that meets the preset alarm condition.

[0099] Furthermore, in some embodiments, after the alarm reminder is issued based on the inference performance data of the preset alarm condition, the alarm subunit is further used to: The alarm location information is generated based on the inference performance data that meets the preset alarm conditions, so that the operation and maintenance personnel can locate the abnormal point based on the alarm location information.

[0100] Furthermore, in some embodiments, the monitoring requirement is at least one of throughput monitoring, latency monitoring, and request quantity monitoring.

[0101] Furthermore, in some embodiments, the display unit is specifically used to: When the monitoring requirement is throughput monitoring, use the preset line chart to display the throughput changes of the target large model inference cluster and the throughput changes of each computing node in different time intervals; When the monitoring requirement is latency monitoring, a preset line chart or a preset bar chart is used to display the average latency of different types of inference requests and the latency of at least one preset percentile; When the monitoring requirement is request quantity monitoring, a preset line graph or a preset pie graph is used to display the corresponding relationship between the number of inference request initiations and the time.

[0102] Further, in some embodiments, after the inference performance data of each computing node is hierarchically stored using a preset time series database, the storage unit is further used to: Set up remote storage configuration; The reasoning performance data of each computing node is backed up to the remote storage configuration according to a second preset period.

[0103] For the description of the features in the embodiment corresponding to the device for acquiring inference performance data of the large model inference cluster, please refer to the relevant description of the embodiment corresponding to the method for acquiring inference performance data of the large model inference cluster, which will not be repeated here.

[0104] According to the inference performance data acquisition device of the large model inference cluster proposed in the embodiment of the present invention, by determining the inference request initiation location, the inference request completion reply location, the location where the first token is returned, and the total inference time for each computing node in the target large model inference cluster when there is an inference request; the inference performance data of each computing node is obtained according to the number of requests initiated at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token. Therefore, the problem that the cluster inference performance data is difficult to obtain accurately can be solved, and the technical effect of efficiently and accurately monitoring and managing the performance of the large model inference cluster can be achieved.

[0105] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include: A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .

[0106] When the processor 902 executes the program, the steps in any of the above-mentioned methods for acquiring inference performance data of a large model inference cluster are implemented.

[0107] Furthermore, the electronic device further comprises: The communication interface 903 is used for communication between the memory 901 and the processor 902 .

[0108] The memory 901 is used to store computer programs that can be executed on the processor 902 .

[0109] The memory 901 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0110] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the communication interface 903, the memory 901 and the processor 902 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0111] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0112] The processor 902 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0113] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the method for obtaining inference performance data of a large model inference cluster when running.

[0114] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory, a mobile hard disk, a magnetic disk, or an optical disk.

[0115] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned embodiments of the method for obtaining inference performance data of a large model inference cluster.

[0116] An embodiment of the present invention also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned embodiments of the method for obtaining reasoning performance data of a large model reasoning cluster.

[0117] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] The above is a detailed introduction to the method for acquiring inference performance data of a large model inference cluster provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for acquiring inference performance data of a large model inference cluster, characterized in that: include: Determine whether there is an inference request; If the inference request exists, determine the inference request initiation location, the inference request completion reply location, the first token return location, and the total inference duration for each computing node in the target large model inference cluster; The number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token and the processing time of each token are obtained, and the inference performance data of each computing node is obtained based on the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token and the processing time of each token.

2. The method for acquiring inference performance data of a large model inference cluster according to claim 1, characterized in that: The number of requests initiated to obtain the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token and the processing time of each token include: Based on the request number record code added at the inference request initiation location, obtaining the request initiation quantity of the inference request initiation location; Based on the throughput recording code added at the reasoning request completion reply position, record the number of input tokens and the number of output tokens, and calculate the throughput of the reasoning request completion reply position in unit time based on the number of input tokens and the number of output tokens; The processing time of the first token is recorded based on the position returned by the first token, and the processing time of each token is calculated using the total inference time.

3. The method for acquiring inference performance data of a large model inference cluster according to claim 2, characterized in that: The inference performance data of each computing node is obtained according to the number of requests initiated at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token, including: Based on a preset multi-level summary structure, determining at least one computing node corresponding to each level in the target large model inference cluster; Sequentially obtain the number of requests initiated at the inference request initiation position within a preset time, the throughput of the inference request completion reply position within a unit time, the processing time of the first token, and the processing time of each token of at least one computing node corresponding to each level, and obtain the inference performance data of at least one computing node corresponding to each level; The reasoning performance data of each computing node is obtained by merging the reasoning performance data of at least one computing node corresponding to each level.

4. The method for acquiring inference performance data of a large model inference cluster according to claim 3, characterized in that: Before determining at least one computing node corresponding to each level in the target large model reasoning cluster based on the preset multi-level summary structure, the method further includes: Determine whether the size of the target large model inference cluster is greater than a preset condition; When the size of the target large model reasoning cluster is greater than the preset condition, the target large model reasoning cluster is divided into the preset multi-level summary structure.

5. The method for acquiring inference performance data of a large model inference cluster according to claim 1, characterized in that: After obtaining the inference performance data of each computing node according to the number of initiation requests at the inference request initiation location, the throughput of the inference request completion reply location per unit time, the processing time of the first token, and the processing time of each token, the method further includes: Sending the reasoning performance data of each computing node to the Consul Server according to a first preset period, and storing the reasoning performance data of each computing node in layers using a preset time series database; Configure the corresponding visualization panel according to monitoring requirements to display inference performance data in different dimensions.

6. The method for acquiring inference performance data of a large model inference cluster according to claim 5, characterized in that: The method of using a preset time series database to hierarchically store the reasoning performance data of each computing node includes: Based on a preset frequency division condition, the reasoning performance data of all computing nodes are divided into a first reasoning performance data group and a second reasoning performance data group, wherein the usage frequency of the first reasoning performance data group is higher than the usage frequency of the second reasoning performance data group; The first reasoning performance data group is stored in a first performance storage medium area of ​​the preset time series database, and the second reasoning performance data group is stored in a second performance storage medium area of ​​the preset time series database, wherein the performance of the first performance storage medium area is higher than the performance of the second performance storage medium area.

7. The method for acquiring inference performance data of a large model inference cluster according to claim 5, characterized in that: When presenting the inference performance data of the different dimensions, the following are also included: Based on the reasoning performance data of each computing node, determining whether there is reasoning performance data that meets a preset alarm condition; If there is inference performance data that meets the preset alarm condition, an alarm reminder is issued based on the inference performance data that meets the preset alarm condition.

8. The method for acquiring inference performance data of a large model inference cluster according to claim 7, characterized in that: The determining, based on the inference performance data of each computing node, whether there is inference performance data that meets a preset alarm condition includes: Calculating at least one statistical feature of the inference performance data of each computing node, and determining whether each statistical feature is not within a corresponding preset interval; When the at least one statistical feature is not within the corresponding preset interval, it is determined that there is inference performance data that meets the preset alarm condition.

9. The method for acquiring inference performance data of a large model inference cluster according to claim 7, characterized in that: After the alarm is given based on the inference performance data of the preset alarm condition, the method further includes: Alarm location information is generated based on the inference performance data that meets the preset alarm conditions, so that operation and maintenance personnel can locate abnormal points based on the alarm location information.

10. The method for acquiring inference performance data of a large model inference cluster according to claim 5, characterized in that: The monitoring requirement is at least one of throughput monitoring, latency monitoring, and request quantity monitoring.

11. The method for acquiring inference performance data of a large model inference cluster according to claim 10, characterized in that: The corresponding visualization panel is configured according to the monitoring requirements to display the reasoning performance data of different dimensions, including: When the monitoring requirement is the throughput monitoring, a preset line chart is used to display the throughput change of the target large model inference cluster and the throughput change of each computing node in different time intervals; In the case where the monitoring requirement is the latency monitoring, using the preset line chart or the preset bar chart to display the average latency of different types of inference requests and the latency of at least one preset percentile; In the case where the monitoring requirement is the request quantity monitoring, the preset line graph or the preset pie graph is used to display the corresponding relationship between the number of inference request initiations and the time.

12. The method for acquiring inference performance data of a large model inference cluster according to claim 5, characterized in that: After the preset time series database is used to hierarchically store the reasoning performance data of each computing node, the method further includes: Set up remote storage configuration; The reasoning performance data of each computing node is backed up to the remote storage configuration according to a second preset period.

13. A device for acquiring inference performance data of a large model inference cluster, characterized in that: include: A judgment module, used to judge whether there is an inference request; A determination module, used for determining, for each computing node in the target large model inference cluster, the location where the inference request is initiated, the location where the inference request is completed and replied, the location where the first token is returned, and the total inference time when the inference request exists; An acquisition module is used to obtain the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, and the processing time of each token, and obtain the inference performance data of each computing node based on the number of request initiations at the inference request initiation location, the throughput of the inference request completion reply location per unit time, and the processing time of each token.

14. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the method for acquiring inference performance data of a large model inference cluster as described in any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for acquiring inference performance data of a large model inference cluster as described in any one of claims 1 to 12.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for acquiring inference performance data of a large model inference cluster as described in any one of claims 1 to 12 are implemented.

Images