Reasoning service methods, apparatuses, systems, devices, and media

Abstract

This disclosure provides an inference service method, apparatus, system, device, and medium, relating to the field of artificial intelligence technology. The method includes: in response to receiving a service request for an inference service and corresponding task configuration parameters, obtaining inference service environment information of the inference service framework corresponding to the service request; mapping the inference service environment information to environment parameters based on preset differentiated configuration information; generating a startup command corresponding to the service request based on the task configuration parameters and environment parameters; and starting the inference service corresponding to the service request based on the startup command. This method can meet various configuration methods to adapt to different hardware platforms and model deployment requirements.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a reasoning service method, apparatus, system, device, and medium. Background Technology

[0002] With the development of large-scale language models, large-scale model inference services and evaluation techniques based on Graphics Processing Units (GPUs) have become one of the fundamental technologies in the field of artificial intelligence. Current technologies typically employ dedicated inference frameworks to deploy large-scale model inference services on GPU servers, and then perform performance testing and accuracy evaluation of these inference services. Summary of the Invention

[0003] This disclosure provides an inference service method, apparatus, system, electronic device, and computer-readable storage medium.

[0004] In a first aspect, embodiments of this disclosure propose an inference service method, comprising: in response to receiving a service request for an inference service and task configuration parameters corresponding to the service request, obtaining inference service environment information of an inference service framework corresponding to the service request; mapping the inference service environment information to environment parameters based on preset differentiated configuration information; generating a startup instruction corresponding to the service request according to the task configuration parameters and the environment parameters; and starting the inference service corresponding to the service request based on the startup instruction.

[0005] In some embodiments, the inference service environment information includes at least one of the following: computing device type, acceleration device type, driver version information, runtime version information, and inference framework dependency library information.

[0006] In some embodiments, the differentiated configuration information includes a configuration matrix, which defines the available parameter range, function switch status, and parameter combination constraint relationships of each environment parameter under different hardware platforms, inference frameworks, and operating conditions. Based on the preset differentiated configuration information, the inference service environment information is mapped to the corresponding environment parameters, including: querying the framework path parameters, library version parameters, and resource allocation parameters defined under the hardware platform, inference framework, and operating conditions corresponding to the inference service environment information in the configuration matrix to obtain the environment parameters.

[0007] In some embodiments, generating a startup command corresponding to a service request based on task configuration parameters and environment parameters includes: concatenating the task configuration parameters and environment parameters to obtain startup parameters for the inference service framework; and converting the startup parameters into command-line instructions that conform to the inference service framework call specification in the service request to obtain a startup command corresponding to the service request.

[0008] In some embodiments, the service request includes a performance evaluation request, which includes an evaluation mode identifier; the method further includes: determining a test mode for the inference service based on the evaluation mode identifier; determining test cases based on the test mode; and performing performance evaluation on the inference service based on the test cases.

[0009] In some embodiments, determining test cases based on a test mode includes: when the test mode is a directional combination test mode, obtaining at least one set of preset input data, output data, and concurrent request counts to obtain test cases; and / or, when the test mode is a grid search test mode, constructing a data combination list of input data and output data and a concurrent configuration list of concurrent request counts, and traversing the data combination list and the concurrent configuration list to obtain a test case set including input data, output data, and concurrent request counts.

[0010] In some embodiments, performance evaluation includes throughput evaluation; performance evaluation of the inference service is performed based on test cases, including: when the test mode is a targeted combination test mode, dynamically adjusting the number of concurrent requests based on the preset input and output data in the test cases using a binary search method; determining the target concurrency corresponding to the input and output data based on the throughput performance test results under different numbers of concurrent requests; wherein, the target concurrency is the number of concurrent requests corresponding to the throughput performance test results reaching the preset conditions.

[0011] In some embodiments, performance evaluation of the inference service framework based on test cases further includes: when the test mode is grid search test mode, during the traversal of the data combination list and the concurrency configuration list, performing throughput tests on the current data combination sequentially using each number of concurrent requests in the concurrency configuration list to obtain the throughput test results of the current data combination under different numbers of concurrent requests; if the throughput test results of the current data combination meet the preset stopping condition, terminating the traversal of the remaining number of concurrent requests in the concurrency parameter configuration list, and continuing to test the next data combination in the data combination list.

[0012] In some embodiments, the service request includes an accuracy evaluation request, which is configured with evaluation configuration parameters; the method further includes: determining at least one target evaluation tool from a variety of preset evaluation tools based on the evaluation configuration parameters and environmental parameters; comparing the inference output results generated by the inference service framework with the benchmark results corresponding to the target evaluation tool based on the target evaluation tool, and determining the accuracy evaluation result of the inference service framework under the inference environment information based on the comparison results.

[0013] In some embodiments, the method further includes: during the execution of inference services on the inference service framework, obtaining computing resource information and memory resource information of the computing node where the inference service framework is located; aligning the computing resource information and memory resource information in time to generate a resource usage representation result of the inference service framework under the inference service environment information; and analyzing the inference evaluation result based on the resource usage representation result.

[0014] In some embodiments, obtaining computing resource information and memory resource information of the computing node where the inference service framework is located includes: determining a monitoring mode based on preset monitoring configuration information; when the monitoring mode is a single-machine monitoring mode, collecting in real time the system resources occupied by the inference service process deployed on the computing node where the inference service framework is located to obtain computing resource information and memory resource information; when the monitoring mode is a multi-machine distributed monitoring mode, uniformly monitoring the system resource operation status on multiple computing nodes through a distributed task scheduling and communication mechanism, and extracting computing resource information and memory resource information of the computing node where the inference service framework is located from the monitored data; wherein, the multiple computing nodes include the computing node where the inference service framework is located.

[0015] In some embodiments, the computing resource information includes resource metrics of the graphics processing unit (GPU) and the central processing unit (CPU); wherein, the GPU resource metrics include at least one of GPU utilization information, GPU memory usage status information, GPU current temperature, and GPU operating frequency information; the CPU resource metrics include at least one of CPU resource utilization rate and CPU load level information; and the memory resource information includes at least one of system memory occupancy status and remaining available memory information.

[0016] In some embodiments, the method further includes: during the execution of the inference service, collecting any one of the following: log information of the inference service, process status, system resource usage information, and result data of the inference service; and visually displaying any one of the collected log information, process status, system resource usage information, and result data of the inference service.

[0017] Secondly, this disclosure also provides an inference service apparatus, which includes: an environment detection unit, configured to, in response to receiving a service request for an inference service and task configuration parameters corresponding to the service request, acquire inference service environment information of the inference service framework corresponding to the service request; a parameter decision unit, configured to, based on preset differentiated configuration information, map the inference service environment information to environment parameters; and a startup command generation unit, configured to, based on the task configuration parameters and environment parameters, generate a startup command corresponding to the service request; and start the inference service corresponding to the service request based on the startup command.

[0018] Thirdly, embodiments of this disclosure also provide a reasoning service system, which includes: the reasoning service apparatus as described in the second aspect above.

[0019] Fourthly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the inference service method as described in any implementation of the first aspect.

[0020] Fifthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the inference service method as described in any implementation of the first aspect.

[0021] In this embodiment of the disclosure, firstly, based on the service request of the inference service and the corresponding task configuration parameters, the runtime environment of the inference service framework corresponding to the service request is detected; then, through preset differentiated configuration information, the detected environment information is mapped to environment parameters; next, based on the received task configuration parameters and the mapped environment parameters, a startup command corresponding to the service request is generated; finally, based on the startup command, the automatic startup of the inference service is completed. Attached Figure Description

[0022] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart of an inference service method provided in this disclosure embodiment; Figure 2 This is a schematic diagram illustrating a reasoning service method provided in conjunction with specific hardware according to an embodiment of this disclosure; Figure 3 A structural block diagram of a reasoning service device provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of an inference system provided in an embodiment of the present disclosure; Figure 5 A schematic diagram of a reasoning evaluation module provided in an embodiment of this disclosure; Figure 6 A schematic diagram of a monitoring inference module provided in an embodiment of this disclosure; Figure 7 A schematic diagram of a visualization analysis module provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0024] In evaluating inference services, some embodiments typically employ dedicated inference frameworks, such as Virtual Large Language Model (vLLM) and Structured Generation Language (SGLang), to deploy large-model inference services on GPU servers. Performance and accuracy evaluations of the inference service are then performed using command-line tools or scripts. Common techniques used in inference evaluation include inference service deployment techniques, performance evaluation techniques, accuracy evaluation techniques, and system resource monitoring techniques.

[0025] Among them, inference service deployment technology refers to providing model inference capabilities by specifying the model path, the number of GPU cards for inference, etc., through command line parameters or configuration files for the inference service process.

[0026] Performance evaluation techniques refer to the use of benchmark test scripts to test performance metrics such as throughput and latency of inference services, and are usually executed offline.

[0027] Accuracy evaluation techniques refer to the use of standard datasets or third-party evaluation tools to assess the accuracy of inference results.

[0028] System resource monitoring technology refers to the use of system monitoring tools to collect information such as GPU memory usage and utilization, as well as the usage status of CPU and memory, in order to analyze the consumption of hardware resources by inference tasks.

[0029] In other embodiments, a reasoning and evaluation toolchain can also be built around a single reasoning framework.

[0030] Specifically, this embodiment typically allows direct inference of the service processes provided by the framework via command line; after the inference service is completed, the corresponding benchmark command or evaluation script is executed manually; and performance evaluation and accuracy evaluation can be completed using different tools; furthermore, the usage of system resources such as GPU, CPU, and memory can be viewed using independent system monitoring tools.

[0031] In another embodiment, the inference and evaluation processes can be automated through scripting. This approach typically includes: Specifically, scripts can be written to launch the inference service and execute performance and accuracy evaluation tasks serially or in parallel within the scripts. During script execution, system commands can be used to collect information on GPU and CPU resource usage, and the evaluation results can be manually compiled and analyzed after the script finishes execution.

[0032] The large-model inference service evaluation schemes provided in the above embodiments typically implement inference service deployment, performance testing tools, and resource monitoring tools separately. They lack a unified system architecture for centralized management of different inference frameworks and evaluation tools, resulting in fragmented inference service methods, complex evaluation processes, and difficulty in forming a reusable evaluation system.

[0033] In terms of performance evaluation, a single evaluation strategy is often used, which usually only supports the collection of performance data under fixed concurrency conditions or only focuses on a single indicator such as throughput. It is difficult to flexibly switch evaluation strategies according to different evaluation objectives, and cannot simultaneously meet the stability evaluation requirements based on performance indicator constraints and the performance limit evaluation requirements based on maximizing throughput.

[0034] Furthermore, the evaluation schemes provided in the above embodiments typically rely on manually setting concurrency parameters and performing post-test analysis during the concurrency capability testing process. They do not yet provide a technical means to automatically screen and determine inference concurrency capability under fixed input and output conditions, combined with performance index thresholds or throughput characteristics, resulting in insufficient evaluation efficiency and objectivity.

[0035] Regarding system resource monitoring, most of the evaluation schemes provided in the above embodiments only monitor the overall usage of GPU, CPU and memory resources during the inference process. They do not record the initial idle state before the inference service, nor do they associate resource data with specific evaluation test items. This results in a lack of a unified resource benchmark between different evaluation tasks, making it difficult to conduct detailed comparisons and reproducibility of evaluation results.

[0036] Meanwhile, performance evaluation results, resource usage data, and test logs are usually stored and displayed in a scattered manner, lacking a unified visual analysis interface, making it difficult to achieve overall analysis and interactive management of inference services, evaluation processes, and evaluation results.

[0037] To address at least one problem with the above embodiments, this disclosure proposes a new inference service solution.

[0038] Figure 1 This is a flowchart of an inference service method provided in an embodiment of the present disclosure. The method can be applied to processors, electronic devices, etc., and the present disclosure does not limit it.

[0039] See Figure 1 The process specifically includes the following steps: Step 101: In response to receiving the service request for the inference service and the task configuration parameters corresponding to the service request, obtain the inference service environment information of the inference service framework corresponding to the service request.

[0040] Specifically, task configuration parameters are key settings used before starting the model inference service based on a service request. These settings define at least one of the following: the evaluation object, control inference behavior, specify evaluation rules, and allocate computing resources. They directly determine the evaluation's objectives, process, results, and efficiency. These task configuration parameters are pre-set for executing the inference service task. Specifically, they may include at least one of the following: task and dataset configuration parameters (such as the path to the inference service startup task, evaluation task, or dataset, prompt word templates, benchmark results, etc.), model and inference parameters, execution and resource parameters (such as inference batch size, number of parallel tasks, execution mode, etc.), and evaluation and post-processing parameters (such as evaluation metrics, result post-processing parameters, etc.).

[0041] Inference service environment information refers to the environment information used to run the inference service framework corresponding to the service request. Each inference service task can be executed in a different runtime environment.

[0042] When inference services are required, the service request for inference services and the pre-configured task configuration parameters are first input into the execution entity (such as processor, electronic device, evaluation system, etc.). After receiving the service request and the corresponding task configuration parameters, the execution entity initiates the inference service startup process. Before the inference service starts, it automatically detects the current running environment (i.e., the environment of the inference service framework used to run the service request) to obtain the corresponding environment information.

[0043] Step 102: Based on the preset differentiated configuration information, map the inference service environment information to environment parameters.

[0044] Specifically, environmental information is objective environmental state data obtained through detection, used to characterize the actual operating environment of the current inference service framework, and belongs to raw information; environmental parameters are parameters determined based on environmental information and used to configure the inference service.

[0045] For example, in an inference service task, the environmental information detected during the run can be GPU model xx, number of GPUs 2, total memory size 256GB; the corresponding environmental parameters can be GPU number, batch size, maximum concurrency, device allocation strategy, etc.

[0046] For example, environmental information includes the specific version of the operating system, the specific version of the model framework, etc., and the corresponding environmental parameters can be the precision modes supported by the operating system or model framework, etc.

[0047] Different inference service environment information can correspond to different environment parameters. Differentiated configuration information is a set of pre-set configuration rules with distinct differences, designed to adapt to different operating environments, different inference frameworks, and different hardware platforms. This differentiated configuration information can be in the form of a mapping table, a dictionary, or a differentiated configuration matrix, etc., and this disclosure does not limit its application to these forms.

[0048] This disclosure allows for the pre-setting of different environmental parameters corresponding to various environmental information, and then the conversion of the detected environmental information into specific environmental parameters based on the pre-set differentiated configuration information.

[0049] Step 103: Generate the startup command corresponding to the service request based on the task configuration parameters and environment parameters.

[0050] Specifically, the service request may include a request to start the inference service. As mentioned above, the task configuration parameters are pre-set input parameters based on the purpose, process, result, efficiency, etc., of the service request, and are generalized parameters in the inference service; the environment parameters are differentiated parameters determined for a specific operating environment. Based on these generalized task configuration parameters and differentiated environment parameters, the startup parameters for starting the inference service corresponding to the service request are constructed, thereby obtaining the corresponding startup command based on the startup parameters.

[0051] Step 104: Start the inference service corresponding to the service request based on the startup command.

[0052] Specifically, the inference service corresponding to the service request is started according to the startup command, and then the corresponding inference service is executed.

[0053] This embodiment first triggers the detection of the runtime environment of the inference service framework corresponding to the inference service request based on the service request of the inference service and the corresponding task configuration parameters; then, through preset differentiated configuration information, the detected environment information is mapped to environment parameters; next, based on the received task configuration parameters and the mapped environment parameters, a startup command corresponding to the service request is generated; finally, based on the startup command, the automatic startup of the inference service is completed.

[0054] The embodiments provided in this disclosure realize environment self-awareness, automatic parameter matching and unified management of the startup process of inference service, thereby improving the deployment efficiency and stability of inference service in heterogeneous computing environment.

[0055] In response to the above Figure 1The inference environment information in step 101 may include at least one of hardware environment information and software environment information.

[0056] In some embodiments, the inference service environment information includes at least one of the following: computing device type, acceleration device type, driver version information, runtime version information, and inference framework dependency library information.

[0057] Specifically, computing device type refers to the type of basic hardware device that performs inference services, such as CPU or other processors; acceleration device type refers to the type of hardware acceleration unit used to accelerate model inference, such as GPU, Field-Programmable Gate Array (FPGA); both computing device type and acceleration device type belong to hardware environment information.

[0058] Driver version information refers to the driver version corresponding to the acceleration device or acceleration hardware, which is the software version used to enable the hardware to function properly; runtime version information refers to the runtime environment version that supports model inference; inference framework dependency library information refers to the library files and their versions that the inference service framework depends on for operation. Driver version information, runtime version information, and dependency library information all belong to software environment information.

[0059] This disclosure ensures the accurate startup of the inference service corresponding to the service request by accurately obtaining the hardware and software environment information corresponding to the service request.

[0060] In some embodiments, the differentiated configuration information includes a configuration matrix, which defines the available parameter range, function switch status, and parameter combination constraint relationships of each environment parameter under different hardware platforms, inference frameworks, and operating conditions. Based on the preset differentiated configuration information, the inference service environment information is mapped to the corresponding environment parameters, including: querying the framework path parameters, library version parameters, and resource allocation parameters defined under the hardware platform, inference framework, and operating conditions corresponding to the inference service environment information in the configuration matrix to obtain the environment parameters.

[0061] Specifically, the configuration matrix can be a pre-built multi-dimensional configuration rule table or configuration mapping relationship, used to define the range of available parameters, function switch states, and combination constraints between different parameters for various environmental parameters under different hardware platforms, different inference frameworks, and different operating conditions.

[0062] Among them, the hardware platform can be the underlying hardware for running inference services, including computing devices, acceleration devices, etc.; the inference framework refers to the underlying framework used to carry out model inference; and the running conditions refer to environmental conditions such as driver version, runtime version, system version, and resource usage.

[0063] The range of available parameters refers to the range or set of values ​​that the environment parameters are allowed to take; the function switch status refers to whether certain functions are enabled, such as whether half-precision floating point (FP16) is enabled, whether multithreading is enabled, and whether hardware acceleration is enabled; the parameter combination constraint relationship refers to the rules that parameters must meet, such as a certain type of computing device can only be used with a corresponding version of the dependency library, a certain acceleration device only supports a specific inference framework version, and certain parameters cannot be set at the same time.

[0064] In other words, the configuration matrix pre-stores the environmental parameters corresponding to various environmental information. Furthermore, the specific environmental parameters corresponding to a given environmental information can be obtained by querying this configuration matrix based on the actual environmental information.

[0065] The framework path parameter specifies the installation path, startup path, and core executable file path of the inference service framework. This disclosure can determine the inference service framework, startup parameters, and core executable file corresponding to the given environment information based on the available parameter range, function switch status, and parameter combination constraints in the configuration matrix. This, in turn, determines the framework path parameter corresponding to the given environment information. The purpose is to ensure that the path parameter matches the inference service framework in the current environment and can be correctly identified and invoked by the system.

[0066] The library version parameter specifies the exact version and link path of various library files that the inference service framework depends on for operation, including inference framework dependencies and runtime dependencies. This disclosure can determine various library files based on the available parameter range in the configuration matrix, the function switch status, and the constraints of parameter combinations, thereby obtaining the library version parameter. The purpose is to ensure that the dependent library versions are compatible with the framework, hardware, and runtime environment, and to avoid version incompatibility issues.

[0067] Resource allocation parameters are used to specify the hardware and software resources used by the inference service framework during runtime, including device allocation and resource limits. This disclosure can determine a resource allocation method that matches the environmental information based on the available parameter range in the configuration matrix, the function switch status, and the constraints of parameter combination relationships. This ensures that the resource allocation matches the current resource status of the environment while supporting the efficient and stable operation of the inference service framework, avoiding resource waste or insufficient resources.

[0068] This embodiment of the disclosure pre-stores and maintains a differentiated configuration matrix, and defines the available parameter range, function switch status and parameter combination constraint relationship corresponding to each environmental information in the configuration matrix. By querying the configuration matrix, the framework path parameters, library version parameters and resource allocation parameters corresponding to the environmental information are determined, and the environmental parameters corresponding to the environmental information are obtained, which improves the efficiency and accuracy of environmental parameter determination.

[0069] Furthermore, the differentiated configuration matrix provided in this disclosure also supports updating, expanding, or adding configurations according to actual needs to adapt to new hardware platforms or inference frameworks.

[0070] In some embodiments, generating a startup command corresponding to a service request based on task configuration parameters and environment parameters includes: concatenating the task configuration parameters and environment parameters to obtain startup parameters corresponding to the service request; and converting the startup parameters into command line instructions that conform to the inference service call specification in the service request to obtain a startup command corresponding to the service request.

[0071] Specifically, task configuration parameters and environment parameters together constitute the startup parameters for the inference service in the service request. Furthermore, the operating system can only execute commands, not parameters directly. This disclosure further transforms startup parameters into startup instructions, aiming to convert abstract configuration parameters into instructions that the operating system can understand and execute.

[0072] Specifically, firstly, the prepared task configuration parameters and environment parameters can be concatenated and integrated to form the startup parameters corresponding to the service request. Then, the concatenated startup parameters are converted into an instruction string format that the inference service corresponding to the service request can accept, so that the operating system can execute the startup command by calling the instruction string, thereby ensuring the accurate startup of the inference service in the service request.

[0073] Figure 2 This is a schematic diagram illustrating an inference service method provided in conjunction with specific hardware according to an embodiment of this disclosure.

[0074] like Figure 2 As shown, this evaluation method can be applied to the inference service module, which may include a unified startup unit, an environment detection unit, a parameter decision unit, a differentiated configuration management unit, and a startup command generation unit. The method specifically includes the following steps: Step S201: User transmits parameters.

[0075] This parameter includes the inference service startup request and related configuration parameters.

[0076] Specifically, the unified startup unit serves as the sole external interface for the inference service module. It receives user-input inference service startup requests and related configuration parameters, and triggers the inference service startup process. By setting up a unified startup unit, only a single startup entry point is exposed externally, thereby shielding the differences in the underlying inference framework and hardware platform, and ensuring the consistency of the inference service startup process.

[0077] Step S202: Obtaining environmental information.

[0078] Specifically, before the inference service starts, the runtime environment detection unit automatically detects the current runtime environment and obtains environment information related to the startup of the inference service. This environment information includes at least the hardware type, acceleration device type, driver or runtime environment information, and basic environment information required for the inference framework to run.

[0079] Step S203: Generate the constraint parameters for the decision.

[0080] Here, the limiting parameter refers to the environmental parameters corresponding to the environmental information.

[0081] Specifically, the differentiated configuration management unit stores and maintains a differentiated configuration matrix, which is used to define the range of available parameters, function switch states, and parameter combination constraints under different hardware platforms, inference frameworks, and operating conditions. Furthermore, this differentiated configuration matrix supports expansion or addition of configurations according to actual needs to adapt to new hardware platforms or inference frameworks.

[0082] The parameter decision unit can be used to determine a set of inference service environment parameters suitable for the current operating environment by matching the environmental information obtained by the operating environment detection unit with the differentiated configuration matrix. This parameter decision unit helps avoid inference service startup failures or performance anomalies due to differences in the operating environment, thus achieving automatic adaptation of the inference service configuration.

[0083] Step S204: Generate the startup command.

[0084] Specifically, the startup command generation unit can generate a corresponding inference service startup command based on the parameters passed in by the user and the inference service constraint parameters determined by the parameter decision unit. This inference service startup command is used to start the inference service of the specified inference framework and includes startup options and configuration parameters that match the current runtime environment.

[0085] Furthermore, the startup command can be sent to the service startup execution unit to start the inference service of the specified inference framework.

[0086] In the embodiments provided in this disclosure, during the inference service startup process, the unified startup unit first triggers the runtime environment detection unit to detect the current runtime environment; subsequently, the differentiated configuration management unit provides a differentiated configuration matrix, and the parameter decision unit determines the inference service startup parameters based on the environment information and the differentiated configuration matrix; next, the startup command generation unit generates the inference service startup command; finally, the service startup execution unit executes the inference service startup command, thereby completing the automated startup of the inference service. This embodiment achieves environment self-awareness, automatic parameter matching, and unified management of the startup process during the inference service startup process, improving the deployment efficiency and stability of the inference service in heterogeneous computing environments.

[0087] Furthermore, this disclosure can also evaluate the performance of the inference service after it is started, so as to achieve a quantifiable assessment of inference capabilities under different hardware platforms, inference frameworks, and parameter configurations. A specific implementation method is given below.

[0088] In some embodiments, the service request includes a performance evaluation request, which includes an evaluation mode identifier. The method may further include: determining a test mode for the inference service based on the evaluation mode identifier; determining test cases based on the test mode; and performing performance evaluation on the inference service based on the test cases.

[0089] Specifically, a performance evaluation request includes at least the inference service instance to be evaluated (i.e., the inference service corresponding to the service request) and an evaluation mode identifier.

[0090] The evaluation mode identifier is a pre-defined identifier used to distinguish different performance evaluation requirements. It corresponds one-to-one with the test mode of the inference service, and different evaluation mode identifiers correspond to different test modes. This disclosure allows for the pre-setting of different evaluation mode identifiers based on different requirements.

[0091] Test cases are collections of specific test samples, test steps, and evaluation criteria used to perform performance evaluations on inference services. Different test modes correspond to different test cases, ensuring that the test cases accurately cover the evaluation dimensions and objectives defined by the test mode.

[0092] In some embodiments, the performance evaluation request may also include evaluation constraints. During the performance evaluation process, the evaluation may be conducted based on these evaluation constraints to ensure the accuracy of the evaluation results.

[0093] This disclosure first identifies the test mode of the inference service corresponding to the service request by identifying the evaluation mode, and clarifies the evaluation objectives and dimensions; then, based on the determined test mode, it selects or generates corresponding test cases to clarify the specific steps, test samples and standards of the evaluation; finally, based on the test cases, it performs performance evaluation on the inference service after startup, thereby improving the efficiency and accuracy of performance evaluation.

[0094] The testing modes include directed combination testing mode and grid search testing mode. The specific implementation methods for determining test cases are given below for different testing modes.

[0095] In some embodiments, the step of determining test cases based on the test mode includes: when the test mode is a directed combination test mode, obtaining at least one set of preset input data, output data, and concurrent request count to obtain test cases; and / or, when the test mode is a grid search test mode, constructing a data combination list of input data and output data and a concurrent configuration list of concurrent request count, and traversing the data combination list and the concurrent configuration list to obtain a test case set including input data, output data, and concurrent request count.

[0096] Specifically, the targeted combination testing mode refers to explicitly specifying the input, output, and concurrency parameters for each test item during performance evaluation, in order to assess performance under specific parameter combinations. Input refers to the inference request data (such as sample data and test data required for model inference) input into the inference service framework; output refers to the expected inference result obtained after the input data is inferred by the inference service framework (i.e., the reference result / standard answer used for evaluation); concurrency parameters are environmental parameters used to control the number of inference requests that the inference service framework can process simultaneously during runtime. Concurrency parameters are used to adjust the concurrency processing capability of the inference service framework to adapt to different evaluation needs and hardware resource conditions.

[0097] The grid search testing mode involves constructing a list of input / output combinations and a list of concurrency levels during performance evaluation. By traversing these two parameter spaces, complete combinations of input, output, and concurrency levels are formed, allowing for a systematic evaluation of performance within these combinations. Each set of input data corresponds to a unique expected output data; these are bound together to form an input-output combination. Multiple such combinations are aggregated to form the input / output combination list, providing diverse test data for performance evaluation. The concurrency level list contains a set of multiple different concurrency values, each representing the concurrency parameter mentioned earlier. The concurrency values ​​in the list can be set according to evaluation requirements, hardware resource conditions, and constraints of the configuration matrix (such as the range of available parameters and parameter combination constraints), covering different concurrency scenarios to test the performance of the inference service framework under varying concurrency pressures. Traversing the data combination list and the concurrency configuration list refers to performing a full traversal of the input and output combination list and the concurrency list, binding each input-output combination in the input and output combination list with each concurrency value in the concurrency list, forming multiple complete combinations of input-output-concurrency.

[0098] The selection of test modes and the composition of their test parameters are all defined and managed uniformly by the configuration script.

[0099] In the targeted combination test mode, one or more fixed input, output, and concurrency parameter configurations can be pre-generated to verify the performance indicators under specified parameter combinations. In the grid search test mode, input and output combination lists and concurrency lists can be constructed separately. A complete set of test cases can be generated by traversing the data combination list and the concurrency configuration list. The optimal throughput performance and corresponding concurrency level that different input and output combinations can achieve can be analyzed in the parameter space of the data combination list and the concurrency configuration list.

[0100] Furthermore, this disclosure can automatically explore the optimal or near-optimal operating parameters of the inference service based on preset performance goals or constraints.

[0101] In some embodiments, performance evaluation includes throughput evaluation; performance evaluation of the inference service is performed based on test cases, including: when the test mode is a targeted combination test mode, dynamically adjusting the number of concurrent requests based on the preset input and output data in the test cases using a binary search method; determining the target concurrency corresponding to the input and output data based on the throughput performance test results under different numbers of concurrent requests; wherein, the target concurrency is the number of concurrent requests corresponding to the throughput performance test results reaching the preset conditions.

[0102] This constraint can be a test mode.

[0103] Specifically, in the case of a targeted combination test mode, an automatic search for optimal throughput concurrency can be triggered. Specifically, under fixed input-output combinations, without pre-specifying the concurrency level, the system dynamically adjusts the concurrency level based on adaptive algorithms such as binary search, testing throughput under different concurrency conditions to automatically determine the optimal concurrency configuration for achieving optimal throughput.

[0104] Furthermore, in some embodiments, performance evaluation of the inference service based on test cases further includes: when the test mode is grid search test mode, during the process of traversing the data combination list and the concurrency configuration list, performing throughput tests on the current data combination in turn using each number of concurrent requests in the concurrency configuration list to obtain the throughput test results of the current data combination under different numbers of concurrent requests; when the throughput test results of the current data combination meet the preset stopping condition, terminating the traversal of the remaining number of concurrent requests in the concurrency parameter configuration list, and continuing to test the next data combination in the data combination list.

[0105] Specifically, in the grid search test mode, a throughput-first concurrency limit exploration can be triggered. Specifically, a complete set of test cases can be generated by iterating through the list of input / output data combinations and the list of concurrency configurations, and the test results are analyzed with throughput as the priority target. During the iteration process, when a certain input-output combination has reached its maximum achievable throughput, testing of higher concurrency configurations can be terminated early to improve overall test efficiency, thereby determining the maximum concurrency level corresponding to the highest throughput for that input-output combination.

[0106] In addition, this disclosure can also realize the maximum concurrency exploration based on indicator constraints. For example, under the premise of meeting the threshold of indicators such as Time Per Output Token (TPOT) and extreme latency (99th Percentile Latency, P99), it can explore the maximum concurrency level that can be carried. This disclosure does not limit this.

[0107] This disclosure utilizes any one of the three performance exploration methods described above to model and evaluate the concurrency and throughput performance of inference services from different perspectives, covering typical performance analysis scenarios such as optimal point search, upper bound evaluation under constraints, and traversal performance scanning. Furthermore, these various exploration methods can be flexibly combined, enabling the automatic selection or coordinated use of multiple search strategies under different evaluation objectives and constraints. This ensures evaluation efficiency while comprehensively characterizing the performance boundaries and optimal operating range of the inference service, providing a reliable basis for performance tuning and resource planning.

[0108] Furthermore, this disclosure can also evaluate the quality of the output results of the inference service under different models, configurations and operating environments, focusing on measuring the accuracy and consistency of the model inference results across various tasks.

[0109] In some embodiments, the service request includes an accuracy evaluation request, which is configured with evaluation configuration parameters. The method further includes: determining at least one target evaluation tool from a variety of preset evaluation tools based on the evaluation configuration parameters and environmental parameters; comparing the inference output result generated by the inference service framework with the benchmark result corresponding to the target evaluation tool based on the target evaluation tool; and determining the accuracy evaluation result of the inference service framework under the inference environment information based on the comparison result.

[0110] Specifically, this disclosure can pre-integrate various evaluation tools and frameworks, including self-developed evaluation tools and mainstream open-source evaluation frameworks, and uniformly encapsulate and manage the evaluation processes and result formats of different tools. Each evaluation tool corresponds to a benchmark result.

[0111] Evaluation configuration parameters may include at least one of the following: performance evaluation task type (such as classification task, generation task, etc.), evaluation criteria, comparison threshold, dataset type, etc., to guide the evaluation tool to complete the comparison and indicator calculation, and ensure that the evaluation meets the preset requirements.

[0112] Upon receiving an accuracy evaluation request, one or more suitable evaluation tools can be selected from the preset tools based on the evaluation configuration parameters and environment parameters for starting the inference service, as specified in the request. The target evaluation tool first compares the generated inference output with the benchmark results to obtain preliminary statistical data. This preliminary statistical data may include basic statistical information such as consistency rate and difference rate, intuitively reflecting the overall consistency of the inference results. Then, based on the evaluation configuration parameters (such as the evaluation task type or other configuration parameters), the raw calculation data for the corresponding accuracy indicators, such as variance and covariance, can be obtained. Finally, the accuracy evaluation results of the inference service framework under the inference environment information are determined based on the accuracy indicators.

[0113] The embodiments provided in this disclosure automatically compare inference outputs with benchmark results through a standardized evaluation process, calculating task-related accuracy metrics, thereby providing a unified evaluation basis for model performance evaluation, version comparison, and regression validation. Through flexible combinations of tool capabilities, the embodiments of this disclosure can cover multiple tasks, multiple datasets, and multiple evaluation criteria scenarios, achieving scalability and consistency in accuracy evaluation, and complementing the performance evaluation module to jointly support the comprehensive evaluation and optimization of the inference system.

[0114] Furthermore, this disclosure can also continuously monitor the operating status of underlying system resources during the operation of the inference service.

[0115] In some embodiments, the method may include not only steps 101-104 above, but may also include: during the execution of the inference service, obtaining the computing resource information and memory resource information of the computing node where the inference service framework is located; aligning the computing resource information and memory resource information in time to generate a resource usage representation result of the inference service framework under the inference service environment information, and analyzing the inference evaluation result based on the resource usage representation result.

[0116] Specifically, during the operation of the inference service, the operational status of the underlying system resources is continuously monitored to obtain monitoring information reflecting inference load, resource utilization, and system stability. Furthermore, this monitoring operation can run independently of the inference service and inference evaluation, and can work synchronously during inference service startup, inference evaluation task execution, and continuous inference service operation, thereby providing basic data support for inference performance analysis, anomaly localization, and resource scheduling.

[0117] Furthermore, in some embodiments, obtaining the computing resource information and memory resource information of the computing node where the inference service framework is located includes: determining the monitoring mode according to preset monitoring configuration information; when the monitoring mode is a single-machine monitoring mode, collecting in real time the system resources occupied by the inference service process deployed on the computing node where the inference service framework is located to obtain computing resource information and memory resource information; when the monitoring mode is a multi-machine distributed monitoring mode, uniformly monitoring the system resource operation status on multiple computing nodes through a distributed task scheduling and communication mechanism, and extracting the computing resource information and memory resource information of the computing node where the inference service framework is located from the monitored data; wherein, the multiple computing nodes include the computing node where the inference service framework is located.

[0118] Specifically, this disclosure also supports single-machine monitoring mode and multi-machine distributed monitoring mode. In single-machine monitoring mode, the system resources occupied by the inference service process deployed on the same computing node can be collected in real time; in multi-machine distributed monitoring mode, the system resource operation status on multiple computing nodes can be uniformly monitored and aggregated through distributed task scheduling and communication mechanisms, and then the resource information of the computing node where the inference service framework is located can be determined from the resource information aggregated from multiple computing nodes.

[0119] This disclosure supports both single-machine monitoring mode and multi-machine distributed monitoring mode, thereby meeting different monitoring needs and improving the flexibility of information acquisition.

[0120] Furthermore, this disclosure can also monitor different resource information based on requirements. This resource information may include graphics processing unit (GPU) resource information, central processing unit (CPU) resource information, and memory resource information.

[0121] In some embodiments, the computing resource information includes resource metrics of the graphics processing unit (GPU) and the central processing unit (CPU); wherein, the GPU resource metrics include at least one of GPU utilization information, GPU memory usage status information, GPU current temperature, and GPU operating frequency information; the CPU resource metrics include at least one of CPU resource utilization rate and CPU load level information; and the memory resource information includes at least one of system memory occupancy status and remaining available memory information.

[0122] Specifically, by monitoring the GPU, the operational status information of the graphics processing unit can be obtained during the inference service's operation, reflecting the GPU's workload on inference computing tasks and its operational stability. GPU monitoring can include monitoring GPU utilization, memory resource usage, and temperature and operating frequency changes during operation. This allows for system-level characterization of GPU resource usage during inference without intruding on the inference service's execution logic, providing fundamental data support for inference performance analysis and operational status evaluation.

[0123] By monitoring the CPU, information on its usage status and load level during inference service operation can be obtained, reflecting the computational pressure the system experiences during request processing, task scheduling, and inference-assisted computation phases. CPU monitoring can include acquiring CPU operating status information, monitoring overall CPU usage and task load changes, thereby determining whether the inference service is constrained by processor resources during operation, and providing a basis for analyzing inference performance fluctuations and system bottlenecks.

[0124] By monitoring memory, we can obtain information on the usage and availability of system memory resources during the inference service's operation, reflecting the memory resources' ability to support the inference process and their impact on system stability. Memory monitoring can include tracking system memory occupancy and remaining available memory resources to help identify potential memory pressure or resource shortages during inference, thereby supporting the stable operation of the inference service and providing early warnings of anomalies.

[0125] This disclosure can unify and align the monitoring information of various resources, and use it as a system-level representation of the inference service's operating status for display in a visualization management and analysis panel, or for subsequent inference evaluation result analysis and performance diagnosis.

[0126] This disclosure enables the coordinated monitoring of the status of multiple system resources, thus comprehensively reflecting the resource usage characteristics of the inference service at different operational stages without relying on the implementation details of a specific inference framework, providing a reliable basis for inference performance evaluation, anomaly analysis, and resource planning.

[0127] Furthermore, this disclosure also allows for the visualization and analysis of various information generated during the reasoning service process.

[0128] In some embodiments, the method further includes: during the execution of the inference service, collecting any one of the following: log information of the inference service evaluation, process status, system resource usage information, and inference service evaluation result data; and visually displaying any one of the collected log information, process status, system resource usage information, and inference service evaluation result data.

[0129] Specifically, this disclosure allows for the centralized display and analysis of status information, evaluation results, and log data generated during inference services, inference evaluation, and inference monitoring, providing users with a unified visual viewing portal. The visualization process does not directly participate in the startup control of the inference service or the execution logic of the evaluation task; instead, it presents the various operational states and results in a read-only manner, thereby reducing the coupling between system modules and improving the overall system stability and maintainability.

[0130] The visualization process can include visual management of inference services and visualization of performance and accuracy evaluations.

[0131] Specifically, this disclosure can visualize the overall status of the inference service during its operation, helping users to understand the service's operation, stability, and resource usage in real time. The visualization process does not involve the inference service's startup or scheduling control; rather, it achieves an intuitive presentation and analytical support of the inference service's status through the collection and display of its operational information.

[0132] In this embodiment, log information, process status, and resource usage information generated during the operation of the inference service can be uniformly displayed. This log information includes at least inference service startup logs, running status logs, and exception logs. By displaying the log information in a time-series format, users can intuitively view the complete process of the inference service from startup to continuous operation, and quickly locate the stage at which the problem occurs when the service is abnormal or performance fluctuates. Simultaneously, this disclosure can also display the process status and system resource usage related to the inference service. System resources include at least computing resources and their usage status, wherein the process status includes at least the running status and survival status of the inference service process. Through the visual display of the inference service process and resource usage, users can monitor the resource consumption level and operational stability of the inference service in real time, thereby providing data for service operation status analysis and capacity assessment.

[0133] Visual management of performance and accuracy evaluations allows for the visualization and analysis of process information and evaluation results generated during and after the evaluation process. This process operates independently from the inference evaluation process; the visualization management of performance and accuracy evaluations is only used to organize, analyze, and present evaluation data, and does not directly participate in the execution of the evaluation task.

[0134] Visual management of performance and accuracy evaluation includes real-time display of the testing process and progress, including displaying the execution process and progress information of evaluation tasks and real-time analysis of performance data.

[0135] The progress information displayed in the evaluation task execution process includes at least the current execution stage of the evaluation task, the number of completed tests, and the number of remaining tests to be executed, so that users can intuitively grasp the overall execution status and completion progress of the evaluation task.

[0136] Real-time performance data analysis is used to analyze and display performance metrics data generated during the evaluation process. These performance metrics include at least key performance indicators such as inference throughput and latency. By visualizing these performance metrics under different evaluation parameter conditions, users can intuitively compare performance differences between different test configurations. In one implementation, evaluation visualization management includes a comprehensive analysis of inference performance under different combinations of input parameters, output parameters, and concurrency parameters, and graphically displaying the trend of performance metrics changing with concurrency. Through graphical display, users can identify the concurrency parameters corresponding to the maximum inference throughput under given input and output conditions, thereby assisting in determining the optimal concurrency configuration for the inference service under different load conditions.

[0137] In addition, this disclosure can also provide an interactive reasoning experience interface based on the current reasoning service, through which users can send test requests to the reasoning service and view the reasoning results, thereby intuitively verifying the functional correctness and response effect of the reasoning service.

[0138] Furthermore, this disclosure can also provide unified management of logs and evaluation result files generated by inference services, inference evaluation, and inference monitoring, thereby enabling centralized storage and querying of evaluation results and related log data, and providing a one-click download function so that users can perform offline analysis or archive management of evaluation data.

[0139] This disclosure, by receiving task configuration parameters and combining an automatic detection and differentiated configuration management mechanism for the runtime environment, enables automated startup and adaptive parameter configuration of inference services under different hardware platforms (such as GPUs and other acceleration devices) and different inference frameworks. This avoids the complexity and error-proneness of manual configuration, improving the deployment efficiency and stability of inference services. Furthermore, the embodiments provided in this disclosure allow inference services, inference evaluation, and inference monitoring to run independently, while collaborating during inference service startup, evaluation execution, and operation. This ensures both the flexibility of the system architecture and provides complete data support for performance analysis and problem localization. Moreover, during inference evaluation, this disclosure can systematically evaluate different combinations of input lengths, output lengths, and concurrency parameters. Combined with visual management, the evaluation process and results can be graphically analyzed. This disclosure can intuitively display the changing relationships of performance indicators such as inference throughput and latency under different parameter conditions, assisting users in quickly identifying optimal or limited configurations. Finally, this disclosure also enables continuous monitoring and visualization of inference service operation logs, process status, and system resource usage through service visualization management and inference monitoring, thereby improving the transparency of the inference service operation process and facilitating anomaly localization, resource scheduling, and system stability analysis.

[0140] The embodiments provided in this disclosure reduce users' reliance on the underlying inference framework and hardware details, enabling non-underlying system experts to efficiently complete inference service deployment, performance evaluation, and operational status analysis.

[0141] Based on the same inventive concept as the above-described method embodiments, this disclosure also provides a reasoning service device.

[0142] Figure 3 This is a structural block diagram of a reasoning service device provided in an embodiment of the present disclosure.

[0143] like Figure 3 As shown, the device includes: an environmental detection unit 301, a parameter decision unit 302, a startup command generation unit 303, and a service startup execution unit 304.

[0144] The environment detection unit 301 is used to obtain the inference service environment information of the inference service framework corresponding to the service request in response to receiving the service request of the inference service and the task configuration parameters corresponding to the service request; the parameter decision unit 302 is used to map the inference service environment information into environment parameters based on the preset differentiated configuration information; the startup command generation unit 303 is used to generate the startup command corresponding to the service request according to the task configuration parameters and environment parameters; and the service startup execution unit 304 is used to start the inference service framework based on the startup command.

[0145] In this embodiment, the specific processing of the environment detection unit 301, parameter decision unit 302, startup command generation unit 303, and service startup execution unit 304 in the inference service device, and their resulting technical effects, can be found in reference to [reference needed]. Figure 1 The relevant descriptions of steps 101-104 in the corresponding embodiments will not be repeated here.

[0146] The inference service apparatus provided in this embodiment first triggers the detection of the operating environment of the inference service framework corresponding to the inference service request based on the service request of the inference service and the corresponding task configuration parameters; then, through preset differentiated configuration information, the detected environment information is mapped to environment parameters; next, based on the received task configuration parameters and the mapped environment parameters, a start command corresponding to the service request is generated; finally, based on the start command, the automatic start of the inference service is completed.

[0147] In some embodiments, the device may further include a unified startup unit, which can serve as the device’s only external interface for receiving inference service startup requests and related configuration parameters, and for triggering the inference service startup process.

[0148] This disclosure sets up a unified startup unit to expose only a single startup entry point to the outside world, thereby shielding the differences in the underlying inference framework and hardware platform and ensuring the consistency of the inference service startup process.

[0149] In some embodiments, the inference service environment information includes at least one of the following: computing device type, acceleration device type, driver version information, runtime version information, and inference framework dependency library information.

[0150] In some embodiments, the differentiated configuration information includes a configuration matrix. The device may also include a differentiated configuration management unit for defining the available parameter ranges, function switch states, and parameter combination constraints of various environmental parameters under different hardware platforms, inference frameworks, and operating conditions. Specifically, the parameter decision unit 302 is used to: query the configuration matrix for the framework path parameters, library version parameters, and resource allocation parameters defined under the hardware platform, inference framework, and operating conditions corresponding to the inference service environment information, to obtain the environmental parameters.

[0151] In some embodiments, the startup command generation unit 303 is specifically used to: concatenate task configuration parameters and environment parameters to obtain startup parameters corresponding to the service request; and convert the startup parameters into command line instructions that conform to the inference service framework call specification in the service request to obtain the startup command corresponding to the service request.

[0152] The specific implementation details and technical effects of the inference service device provided in this disclosure are the same as the implementation details and technical effects of the inference evaluation method embodiments described above, and will not be repeated here.

[0153] This embodiment is a device embodiment corresponding to the method embodiment described above. The inference evaluation device provided in this embodiment can be used to achieve unified startup and management of inference services in a heterogeneous computing environment, thereby shielding the differences between different hardware platforms and inference frameworks. The unified startup unit, environment detection unit 301, differentiated configuration management unit, parameter decision unit 302, startup command generation unit 303, and service startup execution unit 304 in this device work collaboratively in a predetermined order to complete the automated startup of the inference service.

[0154] In the apparatus provided in this disclosure, during the inference service startup process, the unified startup unit first triggers the environment detection unit 301 to detect the current operating environment; then, the differentiated configuration management unit provides a differentiated configuration matrix, and the parameter decision unit 302 determines the inference service environment parameters based on the environment information and the differentiated configuration matrix; next, the startup command generation unit 303 generates an inference service startup command based on the environment parameters and the received task configuration parameters; finally, the service startup execution unit 304 executes the inference service startup command.

[0155] The device provided in this disclosure realizes environment self-awareness, automatic parameter matching, and unified management of the startup process for inference services, thereby improving the deployment efficiency and stability of inference services in heterogeneous computing environments. The inference service evaluation device provided in this disclosure is used to start inference services of different inference frameworks according to configuration parameters through a unified startup entry point, and supports multiple configuration methods to adapt to different hardware platforms and model deployment requirements.

[0156] Furthermore, this disclosure also provides a reasoning service system.

[0157] Figure 4 This is a schematic diagram of the structure of a reasoning service system provided in an embodiment of the present disclosure.

[0158] like Figure 4 As shown, the system includes a reasoning service module 401 and a reasoning evaluation module 402.

[0159] The inference service module 401 is used to launch inference services for different inference frameworks through a unified startup entry point based on configuration parameters. It supports multiple configuration methods to adapt to different hardware platforms and model deployment requirements. Its specific details are the same as those in the aforementioned inference service device embodiment and will not be repeated here. The inference evaluation module 402 is used to perform performance evaluation tasks while the inference service is running, including performance evaluation and accuracy evaluation. The performance evaluation supports multiple evaluation strategies. Under fixed input and output conditions, it can filter inference concurrency capabilities based on preset performance index thresholds to determine the maximum concurrency that meets performance constraints; or, in the absence of performance index constraints, it can determine the concurrency capability with the highest throughput based on throughput characteristics, thereby meeting the needs of different inference performance evaluation objectives.

[0160] Figure 5 This is a schematic diagram of a reasoning evaluation module provided in an embodiment of the present disclosure.

[0161] See Figure 5 In some embodiments, the inference evaluation module 402 includes a performance evaluation unit 412 and an accuracy evaluation unit 422.

[0162] See Figure 4 and Figure 5 In some embodiments, the service request received by the inference service module 401 may include a performance evaluation request, which includes an evaluation mode identifier. The inference service module 401 sends the received performance evaluation request to the performance evaluation unit 412 of the inference evaluation module 402.

[0163] In other embodiments, such as Figure 5 As shown, the performance evaluation unit 412 may also include a unified startup entry point. This unified startup entry point serves as the sole external interface of the performance evaluation unit 412, used to receive performance evaluation requests and trigger the performance evaluation process. This unified startup entry point supports various command formats, file types, or third-party tools, such as the `vllm bench` command (a vllm benchmarking command), the `sglang bench_serving` command (an sglang server-side benchmarking command), specifying `vllm_benchmark.py` (a vllm benchmarking script), or the third-party tool `evalscope` (a third-party tool's evaluation scope / toolset), etc. Furthermore, the unified startup entry point in the performance evaluation unit 412 can be the same as the unified startup entry point in the system; this disclosure does not impose any limitations on this.

[0164] See also Figure 5 The performance evaluation unit 412 also supports the selection of test modes, including directional combination test and grid search test.

[0165] Furthermore, in some embodiments, the performance evaluation unit 412 is specifically used to: determine the test mode of the inference service according to the evaluation mode identifier; determine test cases based on the test mode; and perform performance evaluation on the inference service based on the test cases.

[0166] In some embodiments, the performance evaluation unit 412 is specifically used to: when the test mode is a directional combination test mode, obtain at least one set of preset input data, output data and concurrent request number to obtain test cases; and / or, when the test mode is a grid search test mode, construct a data combination list of input data and output data and a concurrent configuration list of concurrent request number, and traverse the data combination list and concurrent configuration list to obtain a test case set including input data, output data and concurrent request number.

[0167] See also Figure 5 The performance evaluation unit 412 also supports a performance exploration mode, which includes automatic search for optimal throughput concurrency, maximum concurrency exploration based on metric constraints, and concurrency limit exploration prioritizing throughput.

[0168] In some embodiments, performance evaluation includes throughput evaluation; the performance evaluation unit 412 is specifically used to: when the test mode is a directed combination test mode, dynamically adjust the number of concurrent requests based on the preset input data and output data in the test cases using a binary search method; determine the target number of concurrent requests corresponding to the input data and output data according to the throughput performance test results under different numbers of concurrent requests; wherein, the target number of concurrent requests is the number of concurrent requests corresponding to the throughput performance test results reaching the preset conditions.

[0169] In some embodiments, the accuracy evaluation unit 422 is specifically used to: when the test mode is grid search test mode, during the process of traversing the data combination list and the concurrency configuration list, sequentially perform throughput tests on the current data combination using each number of concurrent requests in the concurrency configuration list, and obtain the throughput test results of the current data combination under different numbers of concurrent requests; when the throughput test results of the current data combination meet the preset stop condition, terminate the traversal of the remaining number of concurrent requests in the concurrency parameter configuration list, and continue to test the next data combination in the data combination list.

[0170] In some embodiments, the service request includes an accuracy evaluation request. The accuracy evaluation unit 422 is specifically used to: determine at least one target evaluation tool from a variety of preset evaluation tools based on evaluation configuration parameters and environmental parameters; compare the inference output result generated by the inference service framework with the benchmark result corresponding to the target evaluation tool based on the target evaluation tool; and determine the accuracy evaluation result of the inference service framework under the inference environment information based on the comparison result.

[0171] Among them, see Figure 5 The preset evaluation tools can be self-developed GPU tools, open-source tools such as opencompass (an evaluation tool for large-scale evaluation scenarios) or evalscope (an open-source evaluation tool for large-scale evaluation scenarios), or other evaluation tools. This disclosure does not limit them.

[0172] See also Figure 4 In some embodiments, the inference service system may further include an inference monitoring module 403. Specifically, the inference monitoring module 403 is used to: acquire computing resource information and memory resource information of the computing node where the inference service framework resides during the evaluation of the inference service execution; align the computing resource information and memory resource information over time to generate a resource usage representation result of the inference service framework under the runtime environment information; and analyze the inference evaluation results based on the resource usage representation result.

[0173] Figure 6 This is a schematic diagram of an inference monitoring module provided in an embodiment of the present disclosure.

[0174] Specifically, such as Figure 6 As shown, the inference evaluation module 402 can send information such as computing resource information and memory resource information during the inference service evaluation process to the inference monitoring module 403, so that the inference monitoring module 403 can monitor the computing resource information and memory resource information in real time during the inference service evaluation process.

[0175] See also Figure 6 The inference monitoring module 403 supports cross-node and single-node monitoring. Furthermore, the inference monitoring module 403 includes a GPU monitoring unit 413, a CPU monitoring unit 423, and a memory monitoring unit 433. The GPU monitoring unit 413 acquires the operating status information of the graphics processing unit during the inference service's operation to reflect the GPU's load on inference computing tasks and its operational stability. The CPU monitoring unit 423 acquires the usage status and load level information of the central processing unit during the inference service's operation to reflect the computational pressure borne by the system during request processing, task scheduling, and inference-assisted computing phases. The memory monitoring unit 433 acquires the usage and availability of system memory resources during the inference service's operation to reflect the memory resources' support capacity for the inference process and its impact on system stability.

[0176] In some embodiments, the inference monitoring module 403 is specifically used to: determine the monitoring mode according to preset monitoring configuration information; when the monitoring mode is a single-machine monitoring mode, collect in real time the system resources occupied by the inference service process deployed on the computing node where the inference service framework is located, and obtain computing resource information and memory resource information; when the monitoring mode is a multi-machine distributed monitoring mode, uniformly monitor the system resource operation status on multiple computing nodes through a distributed task scheduling and communication mechanism, and extract the computing resource information and memory resource information of the computing node where the inference service framework is located from the monitored data; wherein, the multiple computing nodes include the computing node where the inference service framework is located.

[0177] In some embodiments, the computing resource information includes resource metrics of the graphics processing unit (GPU) and the central processing unit (CPU); wherein, the GPU resource metrics include at least one of GPU utilization information, GPU memory usage status information, GPU current temperature, and GPU operating frequency information; the CPU resource metrics include at least one of CPU resource utilization rate and CPU load level information; and the memory resource information includes at least one of system memory occupancy status and remaining available memory information.

[0178] See also Figure 4 In some embodiments, the inference service system may further include a visualization analysis module 404, which is specifically used to: collect any one of the following during the execution of the inference service on the inference service framework: log information, process status, system resource usage information, and result data of the inference service; and to visualize any one of the collected log information, process status, system resource usage information, and result data of the inference service.

[0179] Figure 7 This is a schematic diagram of a visualization analysis module provided in an embodiment of the present disclosure.

[0180] like Figure 7 As shown, the visualization analysis module 404 includes a service visualization management module 414, an evaluation visualization management module 424, and a LOG management module 434. Here, LOG represents logs.

[0181] Specifically, the service visualization management module 414 is used for visual monitoring of service operation logs and real-time analysis of service processes and resource status; the evaluation visualization management module 424 is used for visualizing the evaluation process and progress, real-time analysis and display of results, and visualization of the experience of the conversational interface component (UI Chat); and the LOG management module 434 is used for unified management of logs and evaluation result files generated by the inference service module, inference evaluation module, and inference monitoring module.

[0182] The specific implementation details and technical effects of the inference service evaluation system provided in this disclosure are the same as the implementation details and technical effects of the inference service evaluation method embodiments described above, and will not be repeated here.

[0183] This embodiment is a system embodiment corresponding to the above method embodiment. The inference service evaluation system provided in this embodiment includes an inference service module 401, an inference evaluation module 402, an inference monitoring module 403, and a visualization analysis module 404. The inference service module 401 is used to start inference services of different inference frameworks according to configuration parameters through a unified startup entry point, and supports multiple configuration methods to adapt to different hardware platforms and model deployment requirements. The inference evaluation module 402 is used to perform performance evaluation tasks while the inference service is running, including performance evaluation and accuracy evaluation. The performance evaluation supports multiple evaluation strategies. Under fixed input and output conditions, it can filter inference concurrency capabilities based on preset performance index thresholds to determine the maximum concurrency that meets performance constraints; or, in the absence of performance index constraints, it can determine the concurrency capability with the highest throughput based on throughput characteristics, thereby meeting the needs of different inference performance evaluation objectives. The inference monitoring module 403 collects the initial idle state of the system and hardware before the inference service starts, and collects the usage of resources such as GPU, CPU, and memory in real time during the inference evaluation process, including information such as video memory usage, CPU utilization, and memory usage. The collected resource data is associated with specific evaluation test items and stored as part of the evaluation data to improve the comparability and reproducibility of the evaluation results. The visualization analysis module 404 provides a unified graphical interface to enable the startup configuration of the inference service, parameter settings for inference evaluation tasks, real-time monitoring of evaluation progress, graphical display of evaluation results, and log downloading, thereby achieving integrated management and analysis of the entire inference evaluation process.

[0184] The system disclosed herein follows a collaborative logic of "deployment-testing-monitoring-analysis" among its modules: the inference service module 401 provides a standardized inference service foundation for the inference evaluation module 402; the inference monitoring module 403 collects hardware status data throughout the entire inference process and uses it as the basis for evaluation and analysis; and the visualization analysis module 404 receives evaluation results and monitoring data in reverse, while also providing a configuration entry point for the inference service module 401, forming a closed-loop function and data linkage throughout the entire process. Through the system disclosed herein, unified orchestration of the inference service and evaluation process, automatic evaluation of inference concurrency capabilities, and correlation analysis between performance indicators and system resource consumption are achieved, significantly improving the efficiency, objectivity, and reproducibility of large-model inference performance evaluation.

[0185] The solution disclosed herein achieves an integrated upgrade from the "tool level" to the "system level." Existing technologies typically focus only on a single aspect of inference service startup, performance evaluation, or resource monitoring, lacking a unified design. This disclosure systematically integrates inference service startup, evaluation execution, monitoring data collection, and visualization analysis through a unified architecture, forming a complete closed loop.

[0186] Furthermore, this disclosure addresses the problem of complex and non-reusable configuration in heterogeneous environments. Compared to existing technologies that heavily rely on manual configuration or fixed scripts, this disclosure achieves automatic adaptation of inference service parameters through environment detection and a differentiated configuration matrix, significantly reducing the cost of migration between different hardware platforms and inference frameworks.

[0187] Furthermore, this disclosure also supports systematic evaluation and analysis capabilities for performance exploration. Existing evaluation tools are mostly based on single tests or static configurations, making it difficult to comprehensively reflect the variation of inference performance under different parameter combinations. This disclosure supports systematic evaluation of different input / output scales and concurrency combinations, and reveals the optimal performance points and bottleneck ranges through visualization, making the technical effects more intuitive and practical.

[0188] Finally, the solution provided in this disclosure also enhances the observability and traceability of the inference service operation process. By continuously monitoring and displaying service logs, process status, and resource usage, this application improves the observability of the inference service operation process, making performance anomalies or operational problems easier to locate, and has higher engineering practical value compared to existing technologies.

[0189] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the inference service method described in any of the above embodiments when executed.

[0190] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 8 As shown, the electronic device 800 of this embodiment includes a processor 801 and a memory 802; wherein, the memory 802 is used to store computer execution instructions; the processor 801 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. For example, the electronic device 800 can be a general-purpose processor, a graphics processing device, a neural network computing device, or a graph neural network computing device.

[0191] In some embodiments, the memory 802 can be either standalone or integrated with the processor 801.

[0192] When the memory 802 is set up independently, the electronic device also includes a bus 803 for connecting the memory 802 and the processor 801.

[0193] It should be understood that the processor 801 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0194] The memory 802 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0195] Bus 803 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0196] This disclosure also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the steps of the inference service method of any of the above method embodiments.

[0197] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the inference service method according to any of the above embodiments.

[0198] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0199] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0200] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0201] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods in the various embodiments of this application.

[0202] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0203] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0204] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0205] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0206] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0207] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A reasoning service method, characterized in that, The method includes: In response to receiving a service request for an inference service and the task configuration parameters corresponding to the service request, obtain the inference service environment information of the inference service framework corresponding to the service request; Based on preset differentiated configuration information, the inference service environment information is mapped to environment parameters; Based on the task configuration parameters and the environment parameters, a startup command corresponding to the service request is generated; The inference service corresponding to the service request is started based on the startup command.

2. The method according to claim 1, characterized in that, The inference service environment information includes at least one of the following: computing device type, acceleration device type, driver version information, runtime version information, and inference framework dependency library information.

3. The method according to claim 1, characterized in that, The differentiated configuration information includes a configuration matrix, which is used to define the available parameter range, function switch status and parameter combination constraint relationship of each environmental parameter under different hardware platforms, inference frameworks and operating conditions. The mapping of the inference service environment information to environment parameters based on preset differentiated configuration information includes: The environment parameters are obtained by querying the configuration matrix for the hardware platform, inference framework, and framework path parameters, library version parameters, and resource allocation parameters defined under the operating conditions that correspond to the inference service environment information.

4. The method according to claim 1, characterized in that, The step of generating the startup command corresponding to the service request based on the task configuration parameters and the environment parameters includes: The task configuration parameters and the environment parameters are concatenated to obtain the startup parameters corresponding to the service request; The startup parameters are converted into command-line instructions that conform to the inference service call specification in the service request to obtain the startup command corresponding to the service request.

5. The method according to claim 1, characterized in that, The service request includes a performance evaluation request, and the performance evaluation request includes an evaluation mode identifier; the method further includes: The test mode of the inference service is determined based on the evaluation mode identifier; Based on the aforementioned test mode, test cases are determined; Based on the test cases, the performance of the inference service is evaluated.

6. The method according to claim 5, characterized in that, The process of determining test cases based on the test mode includes: When the test mode is a targeted combination test mode, at least one set of preset input data, output data, and concurrent request counts are obtained to obtain the test cases; and / or, When the test mode is grid search test mode, a data combination list of input data and output data and a concurrent configuration list of the number of concurrent requests are constructed, and the data combination list and the concurrent configuration list are traversed to obtain a test case set including input data, output data and the number of concurrent requests.

7. The method according to claim 6, characterized in that, The performance evaluation includes throughput evaluation; the performance evaluation of the inference service based on the test cases includes: When the test mode is a targeted combination test mode, the number of concurrent requests is dynamically adjusted based on the preset input and output data in the test cases using a binary search method. Based on the throughput performance test results under different concurrent request numbers, determine the target concurrency number corresponding to the input data and the output data; wherein, the target concurrency number is the number of concurrent requests corresponding to when the throughput performance test results reach the preset conditions.

8. The method according to claim 7, characterized in that, The performance evaluation of the inference service based on the test cases also includes: When the test mode is grid search test mode, during the process of traversing the data combination list and the concurrency configuration list, the throughput test is performed on the current data combination in turn using each concurrency request number in the concurrency configuration list, so as to obtain the throughput test results of the current data combination under different concurrency request numbers. If the throughput test result of the current data combination meets the preset stopping condition, the traversal of the remaining concurrent request count in the concurrency parameter configuration list is terminated, and the test continues for the next data combination in the data combination list.

9. The method according to claim 1, characterized in that, The service request includes an accuracy evaluation request, which is configured with evaluation configuration parameters; the method further includes: Based on the evaluation configuration parameters and the environmental parameters, at least one target evaluation tool is determined from a variety of preset evaluation tools; Based on the target evaluation tool, the inference output results generated by the inference service framework are compared with the benchmark results corresponding to the target evaluation tool, and the accuracy evaluation results of the inference service framework under the inference environment information are determined according to the comparison results.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: During the execution of the inference service, the computing resource information and memory resource information of the computing node where the inference service framework is located are obtained; The computing resource information and the memory resource information are time-aligned to generate a resource usage representation result of the inference service framework under the inference service environment information, and the inference evaluation result is analyzed based on the resource usage representation result.

11. The method according to claim 10, characterized in that, The step of obtaining the computing resource information and memory resource information of the computing node where the inference service framework is located includes: The monitoring mode is determined based on the preset monitoring configuration information; When the monitoring mode is a single-machine monitoring mode, the system resources occupied by the inference service process deployed on the computing node where the inference service framework is located are collected in real time to obtain the computing resource information and the memory resource information. When the monitoring mode is a multi-machine distributed monitoring mode, the system resource operation status on multiple computing nodes is uniformly monitored through a distributed task scheduling and communication mechanism, and the computing resource information and memory resource information of the computing node where the inference service framework is located are extracted from the monitored data; wherein, the multiple computing nodes include the computing node where the inference service framework is located.

12. The method according to claim 10, characterized in that, The computing resource information includes resource metrics of the graphics processing unit (GPU) and the central processing unit (CPU); wherein, the GPU resource metrics include at least one of the following: GPU utilization information, GPU memory usage status information, GPU current temperature, and GPU operating frequency information; the CPU resource metrics include at least one of the following: CPU resource utilization rate and CPU load level information. The memory resource information includes at least one of the following: the system memory occupancy status and the remaining available memory information.

13. The method according to claim 1, characterized in that, The method further includes: During the execution of the inference service, any one of the following is collected: the log information of the inference service, the process status, the system resource usage information, and the result data of the inference service; The collected log information, process status, system resource usage information, and inference service result data can be visualized.

14. A reasoning service device, characterized in that, The device includes: The environment detection unit is used to obtain the inference service environment information of the inference service framework corresponding to the service request in response to receiving a service request for the inference service and the task configuration parameters corresponding to the service request. The parameter decision unit is used to map the inference service environment information into environment parameters based on preset differentiated configuration information. The startup command generation unit is used to generate a startup command corresponding to the service request based on the task configuration parameters and the environment parameters. The service startup execution unit is used to start the inference service corresponding to the service request based on the startup instruction.

15. A reasoning service system, characterized in that, The system includes: The inference service module is used to execute the inference service method as described in any one of claims 1-4; The reasoning evaluation module is used to execute the reasoning service method as described in any one of claims 5-9.

16. The system according to claim 15, characterized in that, The system also includes: The inference monitoring module is used to execute the inference service method as described in any one of claims 10-12.

17. The system according to claim 15, characterized in that, The system also includes: A visualization analysis module is used to execute the inference service method as described in claim 13.

18. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the reasoning service method according to any one of claims 1-13.

19. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the reasoning service method according to any one of claims 1-13.

Images