Test method, device and equipment of service and storage medium

Abstract

The application discloses a service testing method and device, equipment and a storage medium, and belongs to the field of artificial intelligence. The method comprises the following steps: acquiring a test configuration, wherein the test configuration is used for indicating a test mode of an inference service, and the inference service is implemented based on an LLM; initiating a test request for the inference service according to the test configuration; and acquiring test data of the inference service under multiple test indexes according to a test result corresponding to the test request, wherein the multiple test indexes comprise indexes related to output results and running processes of the inference service. The application can comprehensively evaluate the service quality of the inference service.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for testing services. Background Technology

[0002] Large Language Models (LLMs) possess powerful language understanding, generation, and multi-task generalization capabilities, and can be widely applied to scenarios such as question answering systems, text summarization, code generation, logical reasoning, and professional domain knowledge interaction.

[0003] Inference services can be built based on LLM (Liquidity Management System) to leverage the inference capabilities of LLM to provide services externally. Before providing the inference service, it needs to be tested to determine its service quality. In related technologies, the service quality is typically determined by the difference between the inference service's output and the expected result.

[0004] Testing the inference service using the above methods has the drawback of not being able to fully assess the service quality. Summary of the Invention

[0005] This application provides a service testing method, apparatus, device, and storage medium, which can comprehensively evaluate the service quality of inference services. The technical solution is as follows: According to one aspect of this application, a method for testing a service is provided, the method comprising: Obtain the test configuration, which indicates the testing method for the inference service, which is implemented based on LLM; Initiate a test request for the inference service according to the test configuration described above; Based on the test results corresponding to the test request, test data of the inference service under multiple test indicators are obtained. The multiple test indicators include indicators related to the output results and operation process of the inference service. The indicators related to the operation process of the inference service are used to reflect at least one of the performance and resource consumption of the inference service.

[0006] According to another aspect of this application, a service testing apparatus is provided, the apparatus comprising: The acquisition module is used to acquire test configuration, which indicates the testing method for the inference service, which is implemented based on LLM. The testing module is used to initiate test requests to the inference service according to the test configuration. The acquisition module is further configured to acquire test data of the inference service under multiple test indicators based on the test results corresponding to the test request. The multiple test indicators include indicators related to the output results and operation process of the inference service. The indicators related to the operation process of the inference service are used to reflect at least one of the performance and resource consumption of the inference service.

[0007] In an optional design, the acquisition module is used for: Based on the test results corresponding to the test request, obtain multiple types of the first test data, second test data, and third test data of the inference service; Wherein, the first test data is the test data of the inference service under the output quality index, the second test data is the test data of the inference service under the inference performance index, and the third test data is the test data of the inference service under the resource consumption index.

[0008] In an optional design, the output quality metrics include at least one of the following metrics: answer accuracy and logical consistency; The inference performance metrics include at least one of the following metrics: first-token latency, end-to-end response time, error rate, time per token, throughput, and query rate per second; The resource consumption metrics include at least one of the following metrics: GPU memory usage, CPU usage, memory usage, and resource utilization.

[0009] In an optional design, the device further includes: The generation module is used to generate statistical charts corresponding to the test data based on the test data of the inference service and according to the multiple test indicators.

[0010] In an optional design, the generation module is used for: Based on the test data of the inference service, a parent statistical chart corresponding to the test data is generated according to the various test indicators; Based on the test data corresponding to the parent statistical chart, a sub-statistical chart corresponding to the parent statistical chart is generated according to the sub-indicators associated with the various test indicators, and the sub-statistical chart is triggered for display based on the parent statistical chart.

[0011] In an optional design, the device further includes: The verification module is used to perform at least one of hardware consistency verification and software consistency verification on the test node. The test nodes include nodes for deploying the inference service. The hardware consistency check is used to check the consistency of at least one of the GPU model, video memory capacity, CPU architecture, and network bandwidth of different test nodes. The software consistency check is used to check the consistency of at least one of the driver version, acceleration library, runtime environment, and dependency library of different test nodes.

[0012] In an alternative design, the device includes: The loading module is used to load the test cases and semantic scoring logic in the test configuration. The test cases include at least one of general test cases and test cases configured according to the target domain, and the semantic scoring logic is used to score the output results of the inference service.

[0013] In an optional design, the test module is used for: Initiate a test request for the inference service according to the test intensity parameters in the test configuration; The test intensity parameters include at least one of the following: test concurrency, total number of requests, context length, model generation length, request interval, and timeout threshold.

[0014] In an optional design, the device further includes: A module is established to establish the association between the test data of the inference service under the various test metrics and the identifier of the test request.

[0015] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement a test method for the services described above.

[0016] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being loaded and executed by a processor to implement a test method for the services described above.

[0017] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a test method for the services provided in various alternative implementations of the above aspects.

[0018] The beneficial effects of the technical solution provided in this application include at least the following: By testing the LLM-based inference service according to the test configuration, test data on the inference service under various test metrics can be obtained. Since the various test metrics include indicators related to the output results and operation process of the inference service, it is possible to test the inference service with multi-dimensional indicators, thereby achieving a comprehensive evaluation of the service quality of the inference service. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the testing process of the inference service provided in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a testing method for a service provided in an exemplary embodiment of this application; Figure 3 This is a flowchart illustrating a testing method for a service provided in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of a statistical chart provided in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of a testing and analysis process provided in an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the structure of a testing apparatus for the services provided in an exemplary embodiment of this application; Figure 7 This is a schematic diagram of the structure of a testing apparatus for the services provided in an exemplary embodiment of this application; Figure 8 This is a schematic diagram of the structure of a testing apparatus for the services provided in an exemplary embodiment of this application; Figure 9 This is a schematic diagram of the structure of a testing apparatus for the services provided in an exemplary embodiment of this application; Figure 10 This is a schematic diagram of the structure of a testing apparatus for the services provided in an exemplary embodiment of this application; Figure 11 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application.

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0023] First, let me introduce the terms used in this application: Large Language Models (LLMs) are theories, methods, systems, and applications in the field of artificial intelligence that enable computer systems to simulate human intelligent behavior. Their core areas include machine learning, deep learning, natural language processing, computer vision, speech processing, and intelligent reasoning and decision-making. LLMs, as a cutting-edge technology in natural language processing, refer to ultra-large-scale deep neural network models based on a transformer architecture. These models are pre-trained on massive amounts of unlabeled text data through self-supervised learning and are typically fine-tuned through instructions or reinforcement learning based on human feedback to align with human intent. These models typically have billions to hundreds of billions of parameters and possess powerful language understanding, generation, and multi-task generalization capabilities, making them widely applicable in question-answering systems, text summarization, code generation, logical reasoning, and domain-specific knowledge interaction.

[0024] Inference service: Inference service refers to the process of deploying a trained large language model to a computing device, receiving user input prompts, and generating corresponding output text based on the model's internal parameters. This process typically involves steps such as word segmentation of the input sequence, embedding representation, forward computation of a multi-layer Transformer structure, probability distribution sampling, or greedy decoding, ultimately outputting a natural language response. In practical applications, large model inference needs to run on specific hardware platforms (such as Graphics Processing Units (GPUs), Tensor Processing Units (TPUs), or dedicated accelerators). Its performance is affected by factors such as model size, context length, batching strategy, quantization method, and system resource scheduling, manifested in key indicators such as response latency, throughput, memory usage, and energy consumption. Therefore, efficient, stable, and measurable execution and evaluation of inference services are core technical aspects for ensuring the quality and deployment feasibility of large model services.

[0025] YAML: A recursive abbreviation for YAML Ain't a Markup Language. YAML is a highly readable data serialization format.

[0026] With the deep integration of large language models into various software systems, their role as core intelligent components has evolved from traditional "functional modules" to "probabilistic services" with high uncertainty and complexity. Traditional software testing methods face significant limitations when dealing with such models. The output of large language models is nondeterministic, context-sensitive, and semantically open. Their behavior depends not only on the input format but also on multiple factors such as training data distribution, fine-tuning strategies, inference parameters, and deployment environment, making them difficult to verify through pre-set assertions or fixed expected results. Furthermore, applications of these models in specialized fields (such as bioinformatics, quantum physics, and financial compliance) place stringent demands on output accuracy, logical consistency, and factual reliability, while general benchmarks often fail to cover the functional correctness and risk boundaries of specific business scenarios. In addition, large model inference services must also meet non-functional attribute constraints such as latency, throughput, resource consumption, and stability during actual operation; their performance directly affects user experience and system maintainability. Therefore, it is urgent to build a new testing system that integrates the characteristics of artificial intelligence and the principles of software engineering. Through the collaborative evaluation of multi-dimensional indicators (including output quality, inference performance, and resource consumption), configurable domain-specific test cases, automated execution mechanisms, and visualization analysis methods, a systematic verification and guarantee of the entire lifecycle of inference services can be achieved, ensuring their reliability, security, and availability in real-world application scenarios.

[0027] The method provided in this application enables the simultaneous collection and correlated evaluation of multiple dimensions of indicators, such as the output quality, inference performance, and resource consumption of large language models. It supports custom test cases and semantic scoring logic to adapt to the evaluation needs of specialized fields, reducing manual intervention. It provides interactive, drill-down visual statistical charts to intuitively present and efficiently analyze multi-dimensional test data. It establishes a configuration-driven, automated testing process to ensure the reproducibility of the evaluation process and support continuous integration.

[0028] Figure 1 This is a schematic diagram illustrating the testing process of the inference service provided in an exemplary embodiment of this application. For example... Figure 1As shown, the computer device acquires a test configuration 101, which indicates the testing method for the inference service 102. The inference service 102 is implemented based on an LLM 103 and is a service that calls the LLM 103 to perform inference. Optionally, the test configuration 101 includes at least one of test object information, test cases, semantic scoring logic, and test intensity parameters. The test object information includes information related to the test object; the test cases include test cases for testing the inference service 102; the semantic scoring logic is used to score the output of the inference service 102; and the test intensity parameters indicate the intensity or stress of testing the inference service 102. In some embodiments, the test configuration 101 is stored in a test configuration file, for example, in a YAML format test configuration file.

[0029] After obtaining the test configuration 101, the computer device initiates a test request for the inference service 102 according to the test configuration 101. The test request is used to execute a test on the inference service 102, thereby obtaining the test results corresponding to the test request. The test results include results related to the operation and output of the inference service 102. Based on the test results corresponding to the test request, the computer device obtains test data 104 of the inference service 102 under multiple test indicators. Among them, the multiple test indicators include indicators related to the output results and operation process of the inference service 102. Optionally, the test data 104 includes multiple of the following: first test data, second test data, and third test data. The first test data is the test data of the inference service 102 under the output quality indicator, the second test data is the test data of the inference service 102 under the inference performance indicator, and the third test data is the test data of the inference service 102 under the resource consumption indicator.

[0030] In some embodiments, upon obtaining test data 104 from the inference service 102, the computer device will also generate a statistical chart 105 corresponding to the test data 104 according to the aforementioned multiple test indicators. Optionally, the computer device generates a parent statistical chart corresponding to the test data 104 according to the multiple test indicators based on the test data 104 from the inference service 102, and generates a sub-statistical chart corresponding to the parent statistical chart according to the test data 104 corresponding to the parent statistical chart and the sub-indicators associated with the multiple test indicators. The sub-statistical chart is triggered for display based on the parent statistical chart, and the sub-statistical chart can be displayed by drilling down the parent statistical chart.

[0031] By testing the LLM-based inference service according to the test configuration, test data on the inference service under various test metrics can be obtained. Since the various test metrics include indicators related to the output results and operation process of the inference service, it is possible to test the inference service with multi-dimensional indicators, thereby achieving a comprehensive evaluation of the service quality of the inference service.

[0032] Figure 2 This is a flowchart illustrating a testing method for a service provided in an exemplary embodiment of this application. The method can be used with a computer device. Figure 2 As shown, the method includes: Step 202: Obtain the test configuration.

[0033] The inference service in this embodiment is implemented based on an LLM (Liquid Management System). The inference service is a service that calls the LLM to perform inference. For example, the inference service receives a prompt sent by a user account, inputs the prompt into the LLM for inference, obtains the inference result, and then returns the inference result to the user account.

[0034] In some embodiments, the inference service in this application supports invoking one or more LLMs, which may be identical, partially identical, or completely different; this application does not impose any limitations on this. The one or more LLMs may be deployed on one or more test nodes, including nodes used for deploying the inference service. Test nodes may be computer devices such as servers, or they may be virtual nodes; this application does not impose any limitations on this.

[0035] The test configuration indicates the testing method for the inference service. Testing the inference service can also be understood as testing the LLM invoked by the inference service. Optionally, the test configuration includes at least one of the following: test object information, test cases, semantic scoring logic, and test intensity parameters. The test object information includes information related to the test object, such as at least one of the following: username, Internet Protocol (IP) address, and login password of the test object (test node). Test cases include test cases for testing the inference service, such as test cases for testing the LLM invoked by the inference service. The semantic scoring logic is used to score the output of the inference service, such as scoring the output of the LLM invoked by the inference service. The test intensity parameters indicate the intensity or stress of testing the inference service.

[0036] In some embodiments, the test configuration is set by the tester. In other embodiments, the test configuration is stored in a test configuration file, such as a YAML-formatted test configuration file.

[0037] Step 204: Initiate a test request to the inference service according to the test configuration.

[0038] The test configuration is used to trigger the execution of tests on the inference service. After obtaining the test configuration, the computer device will initiate a test request to the inference service according to the test configuration. The test request is used to execute tests on the inference service.

[0039] In some embodiments, the computer device establishes communication connections with different test nodes of the inference service to perform testing of the inference service. Optionally, the computer device logs into the test nodes of the inference service via the Secure Shell (SSH) protocol according to the test configuration to establish communication connections with different test nodes.

[0040] It should be noted that the computer equipment in this application embodiment can be used exclusively for testing the inference service, or it can participate in the testing of the inference service as a test node corresponding to the inference service. This application embodiment does not impose any restrictions on this.

[0041] Step 206: Based on the test results corresponding to the test request, obtain the test data of the inference service under various test metrics.

[0042] The computer equipment initiates a test request to the inference service according to the test configuration to test the inference service, thereby obtaining test results corresponding to the test request. The test results include results related to the operation and output of the inference service. Multiple test metrics include metrics related to the output results and operation process of the inference service. Metrics related to the output results of the inference service are used to reflect the output quality of the inference service, such as the accuracy of the output. Metrics related to the operation process of the inference service are used to reflect at least one of the performance and resource consumption of the inference service, such as the output latency or the GPU utilization of the inference service. Optionally, the multiple test metrics include several metrics among output quality metrics, inference performance metrics, and resource consumption metrics.

[0043] Optionally, the test data includes multiple types of first test data, second test data, and third test data. The first test data is the test data of the inference service under output quality metrics, the second test data is the test data of the inference service under inference performance metrics, and the third test data is the test data of the inference service under resource consumption metrics.

[0044] In summary, the method provided in this embodiment tests an LLM-based inference service according to a test configuration to obtain test data for the inference service under various test metrics. Since these various test metrics include indicators related to the output results and operation process of the inference service, multi-dimensional testing of the inference service can be achieved, thereby enabling a comprehensive evaluation of the service quality of the inference service.

[0045] Figure 3 This is a flowchart illustrating a testing method for a service provided in an exemplary embodiment of this application. The method can be used with a computer device. Figure 3 As shown, the method includes: Step 302: Obtain the test configuration.

[0046] The inference service in this application embodiment is implemented based on an LLM (Liquid Management System). The inference service is a service that calls the LLM to perform inference. In some embodiments, the inference service in this application embodiment supports calling one or more LLMs, which can be deployed on one or more test nodes. Test nodes include nodes used to deploy the inference service. Test nodes can be computer devices such as servers, or they can be virtual nodes; this application embodiment does not impose any limitations on this.

[0047] The test configuration indicates the testing method for the inference service. Testing the inference service can also be understood as testing the LLM invoked by the inference service. Optionally, the test configuration includes at least one of the following: test object information, test cases, semantic scoring logic, and test intensity parameters. The test object information includes information related to the test object, such as at least one of the username, IP address, and login password of the test object (test node). Test cases include test cases for testing the inference service, such as test cases for testing the LLM invoked by the inference service. The semantic scoring logic is used to score the output results of the inference service, such as scoring the output results of the LLM invoked by the inference service. The test intensity parameters indicate the intensity or stress of testing the inference service.

[0048] In some embodiments, the test configuration is set by the tester. In other embodiments, the test configuration is stored in a test configuration file, such as a YAML-formatted test configuration file.

[0049] Optionally, before testing the inference service through the test nodes of the inference service, the computer equipment performs at least one of hardware consistency checks and software consistency checks on the test nodes of the test service. The hardware consistency check verifies the consistency of at least one of the following across different test nodes: GPU model, video memory capacity, central processing unit (CPU) architecture, and network bandwidth, to ensure hardware consistency across different test nodes. The software consistency check verifies the consistency of at least one of the following across different test nodes: driver version, acceleration libraries, runtime environment (e.g., code execution environment), and dependent libraries, to ensure software consistency across different test nodes.

[0050] Step 304: Initiate a test request to the inference service according to the test configuration.

[0051] The test configuration is used to trigger the execution of tests on the inference service. After obtaining the test configuration, the computer device will initiate a test request to the inference service according to the test configuration. The test request is used to execute tests on the inference service.

[0052] In some embodiments, the computer device establishes communication connections with different test nodes of the inference service to test the inference service. Optionally, the computer device logs into the test nodes of the inference service via the SSH protocol according to the test configuration to establish communication connections with different test nodes. Optionally, the test nodes expose test interfaces, which are interfaces used to call the inference service deployed on the test nodes. Optionally, the test interface includes at least one of a Hypertext Transfer Protocol (HTTP) interface and a gRPC (a high-performance remote procedure call framework) interface.

[0053] It should be noted that the computer equipment in this application embodiment can be used exclusively for testing the inference service, or it can participate in the testing of the inference service as a test node corresponding to the inference service. This application embodiment does not impose any restrictions on this.

[0054] In some embodiments, during the process of initiating a test request for the inference service according to the test configuration, the computer device loads the test cases and semantic scoring logic from the test configuration, thereby testing the inference service using the test cases and semantic scoring logic in the test configuration. The test cases include at least one of general test cases and test cases configured according to the target domain. The semantic scoring logic is used to score the output results of the inference service. The general test cases are pre-configured test cases, the target domain is the domain involved in the tests executed based on the test configuration, and the test cases and semantic scoring logic configured according to the target domain can be configured by the tester.

[0055] In some embodiments, the computer device initiates test requests to the inference service according to test intensity parameters in the test configuration. Test intensity parameters indicate the intensity or stress of testing the inference service. Optionally, test intensity parameters include at least one of the following: test concurrency, total number of requests, context length, model generation length, request interval, and timeout threshold. Test concurrency refers to the number of inference requests processed simultaneously at a given moment or time period. Total number of requests refers to the total number of inference requests initiated to the inference service throughout the entire test period. Context length refers to the maximum length of input text that the LLM can receive and process during a single inference process, typically measured in tokens. Model generation length refers to the maximum length of text that the LLM is allowed to output during a single inference process, i.e., the maximum length of the answer or generated content, typically measured in tokens. Request interval refers to the time interval between two consecutively initiated inference requests. Timeout threshold refers to the upper limit of the time waiting for a response from the inference service; if a complete response is not received within this time, the inference request is terminated and determined to have timed out.

[0056] Step 306: Based on the test results corresponding to the test request, obtain the test data of the inference service under various test metrics.

[0057] The computer equipment initiates a test request to the inference service according to the test configuration to test the inference service, thereby obtaining test results corresponding to the test request. The test results include results related to the operation and output of the inference service. Multiple test metrics include metrics related to the output results and operation process of the inference service. Specifically, metrics related to the output results of the inference service reflect the output quality of the inference service, while metrics related to the operation process of the inference service reflect at least one of the following: performance and resource consumption. Optionally, multiple test metrics include several metrics among output quality metrics, inference performance metrics, and resource consumption metrics. Specifically, output quality metrics measure the inference quality of the inference service, inference performance metrics measure the performance of the inference service, and resource consumption metrics measure the resource consumption of the inference service during the inference process.

[0058] In some embodiments, the computer device establishes an association between test data of the inference service under various test metrics and the identifier (ID) of the test request. After establishing the association, the computer device can obtain the corresponding test data based on the identifier of different test requests. In some embodiments, the computer device also performs structured aggregation of the test data according to at least one dimension: test case ID, timestamp, model name, version number, and hardware configuration.

[0059] Optionally, the computer device obtains multiple types of first test data, second test data, and third test data for the inference service based on the test results corresponding to the test request. The first test data is the test data for the inference service under output quality metrics, the second test data is the test data for the inference service under inference performance metrics, and the third test data is the test data for the inference service under resource consumption metrics. The computer device extracts corresponding data from the test results according to multiple output quality metrics, inference performance metrics, and resource consumption metrics, thereby obtaining multiple types of first test data, second test data, and third test data. The first test data, second test data, and third test data include one or more types of test data.

[0060] In some embodiments, output quality metrics include at least one of the following: answer accuracy and logical consistency. Inference performance metrics include at least one of the following: Time To First Token (TTFT), end-to-end response time (also known as Time To Last Token (TTLT)), error rate, time per output token (TPOT), throughput, and queries per second (QPS). Resource consumption metrics include at least one of the following: GPU memory usage, CPU usage, memory usage, and resource utilization.

[0061] Response accuracy refers to the degree to which the LLM-generated response matches the standard answer or the truth. Logical consistency refers to the LLM's ability to maintain consistency in viewpoints, facts, and role settings during long text generation or multi-turn dialogues. First-term latency is the time elapsed from issuing an inference request to receiving the first term generated by the LLM. End-to-end response time is the total time elapsed from issuing an inference request to receiving the last term generated by the LLM. Error rate is the percentage of failed requests out of the total number of requests during testing. Time per term is the average time required for the LLM to generate each term during the generation phase. Throughput is the amount of data or requests successfully processed per unit of time, typically referring to the total number of terms generated per second. Query rate per second is the number of inference requests successfully processed per unit of time.

[0062] Step 308: Based on the test data of the inference service, generate statistical charts corresponding to the test data according to various test indicators.

[0063] Upon obtaining test data for the inference service, the computer device will also generate statistical charts corresponding to the test data according to the aforementioned various test indicators. Optionally, the computer device will generate at least one of the following statistical charts based on the test data for the inference service: one for output quality indicators, one for inference performance indicators, and one for resource consumption indicators. The statistical charts for output quality indicators include at least one of answer accuracy and logical consistency. The statistical charts for inference performance indicators include at least one of first-token latency, end-to-end response time, error rate, time per token, throughput, and queries per second. The statistical charts for resource consumption indicators include at least one of GPU memory usage, CPU usage, memory usage, and resource utilization.

[0064] Optionally, the computer device generates a parent statistical chart corresponding to the test data based on the test data from the inference service, according to various test metrics. Then, based on the test data corresponding to the parent statistical chart, it generates a sub-statistical chart corresponding to the parent statistical chart, according to sub-metrics associated with the various test metrics. The sub-statistical chart is displayed based on the parent statistical chart. For example, when the parent statistical chart is displayed, a touch operation (such as a click) on the parent statistical chart can trigger the display of the corresponding sub-statistical chart.

[0065] For example, Figure 4 This is a schematic diagram of a statistical chart provided in an exemplary embodiment of this application. For example... Figure 4 As shown, Statistical Chart 401 is a first-word delay statistics chart. The horizontal axis represents time (date), and the vertical axis represents the average first-word delay of all test nodes on that date. Statistical Chart 401 can trigger the display of Statistical Chart 402, which is a sub-chart of Statistical Chart 401. For example, clicking on a date in Statistical Chart 401 will display the corresponding Statistical Chart 402. The horizontal axis of Statistical Chart 402 represents the test nodes, and the vertical axis represents the first-word delay of the test nodes on that date.

[0066] It should be noted that this embodiment of the application uses the example of a computer device initiating a test generating a statistical chart for illustration. It is understood that, in addition to the computer device initiating the test, other computer devices can also perform the step of generating the statistical chart. In this case, the computer device initiating the test will provide test data for inference services to other computer devices so that the other computer devices can generate the statistical chart.

[0067] In summary, the method provided in this embodiment tests an LLM-based inference service according to a test configuration to obtain test data for the inference service under various test metrics. Since these various test metrics include indicators related to the output results and operation process of the inference service, multi-dimensional testing of the inference service can be achieved, thereby enabling a comprehensive evaluation of the service quality of the inference service.

[0068] The method provided in this embodiment also acquires multiple test data from the first, second, and third test data of the inference service, enabling the acquisition of test data for the inference service under multi-dimensional indicators, thereby achieving a comprehensive evaluation of the service quality of the inference service. By setting multiple sub-indicators for each of the output quality, inference performance, and resource consumption indicators, the service quality of the inference service can be evaluated from multiple dimensions, including output quality, inference performance, and resource consumption. By generating corresponding statistical charts based on the test data of the inference service, the service quality of the inference service can be intuitively displayed. By generating parent statistical charts and corresponding sub-statistical charts based on the test data of the inference service, an interactive visualization mechanism can be implemented, improving the understandability of test results and the efficiency of problem localization. By performing hardware consistency checks and software consistency checks on test nodes, hardware and software consistency of different test nodes can be ensured, thereby ensuring the rationality of the test. By supporting the configuration of test cases and semantic scoring logic for specific domains, the output quality of the inference service can be accurately evaluated. By initiating tests on the inference service according to test intensity parameters, the test pressure on the inference service can be flexibly controlled to ensure the stability of the inference service. By establishing a correlation between the test data of the inference service and the identifier of the test request, the corresponding test data can be obtained according to the identifier of different test requests, thereby improving the readability and maintainability of the data.

[0069] The method provided in this application aims to solve the technical problems of fragmented evaluation dimensions, insufficient domain adaptability, and low efficiency of result analysis in inference service testing. This method is mainly implemented through the following modules: 1. Machine Basic Information Module: Automatically collects and records the software and hardware configurations of the test execution environment to ensure the traceability and reproducibility of test results. 2. Test Case Management Module: Supports loading general or user-defined domain test cases and configuring corresponding semantic scoring logic. Domain test cases are test cases configured according to the target domain, which is the domain involved in the test executed based on the test configuration. The semantic scoring logic is used to score the output results of the inference service. 3. Multi-Dimensional Test Execution Engine: Concurrently executes model inference requests, synchronously acquiring multi-dimensional raw data such as output quality, inference performance, and resource consumption. 4. Metric Collection and Aggregation Module: Performs request-level alignment on multi-source heterogeneous metrics, that is, associates the test data of the inference service under different test metrics with the identifiers of the test requests, and performs structured storage and multi-dimensional aggregation to form a unified evaluation dataset. 5. Visualization module: Provides an interactive, drill-down chart interface to intuitively present performance comparisons and anomaly analysis results across multiple models and dimensions.

[0070] The main objectives of this application are: 1. To improve the understandability and analysis efficiency of test results: Addressing the characteristics of LLM test results—large data volume, complex dimensions, and semantic abstraction—an interactive visualization interface is constructed to present multi-dimensional indicators such as quality, performance, and resources intuitively in chart form. It supports multi-model comparison, displaying statistical charts corresponding to different inference services within the same user interface, and highlighting and drilling down to view abnormal data. Data exceeding a preset range in the statistical charts is highlighted, and corresponding sub-statistics can be triggered to display them. This significantly lowers the barrier to interpreting results and improves the efficiency of testers in analyzing model behavior, performance bottlenecks, and error patterns. 2. To support flexible adjustment of test intensity and load mode: Allowing users to dynamically adjust key load factors such as test concurrency, total requests, context length, model generation length, request interval, and timeout threshold by configuring test intensity parameters. This simulates various real-world application scenarios, from lightweight verification to high-pressure load testing, meeting the differentiated needs of different stages, such as development debugging, pre-deployment verification, and online inspection, for varying test intensity requirements. 3. Achieve collaborative evaluation of multi-dimensional indicators: Breaking through the limitations of focusing only on output accuracy or monitoring only system throughput, this application embodiment synchronously collects semantic quality (such as task answer accuracy, logical consistency), inference performance (such as first-word latency, end-to-end response time, error rate), and system resource consumption (such as GPU memory, utilization, CPU and memory usage) indicators in the same test process. It also performs precise alignment based on request granularity, establishing a correlation between test data and test request identifiers under different test indicators for the inference service, forming a three-dimensional and integrated evaluation of the overall performance of the large model. 4. Ensure automation and reproducibility of the testing process: The entire testing process is driven by a structured configuration file used to store test configurations, from environment information recording, test case loading, concurrent execution to result storage and visualization, achieving end-to-end automation. All test parameters, model versions, environment configurations, and execution logs are fully archived, ensuring that any test task can be accurately reproduced, providing a reliable basis for model iteration, version comparison, and issue backtracking.

[0071] The technical means of this application include: 1. A technical means to automate the generation and execution of multi-dimensional performance testing tasks for large language models, reducing the complexity of manually constructing test cases: Test tasks are defined through structured configuration files, supporting automatic loading of general test cases or user-defined test cases according to the target domain, and automatically scoring the output results by combining semantic scoring logic. The system does not require manual writing of assertions or maintenance of fixed expected answers; it can automatically initiate requests, collect metrics, and complete evaluation in single-machine or multi-machine environments, significantly reducing the technical threshold for designing effective test cases for the open output of large language models. 2. A technical means to support adjustable test intensity and multi-scenario coverage, improving the simulation capability of actual inference loads: By configuring parameters such as concurrency, context length, generation length, and request interval, the stress test intensity can be flexibly adjusted to simulate various real-world application scenarios, from low-frequency interactions to high-concurrency services. Meanwhile, test cases can cover different semantic types such as question and answer, code generation, logical reasoning, and professional domain question and answer, making the measurement of indicators such as latency, throughput, and resource consumption closer to the actual business load, overcoming the limitation of using only a fixed prompt and being unable to reflect the performance under complex semantic interactions. 3. Possessing unified multi-dimensional indicator collaborative testing capabilities, breaking through the bottleneck of quality and performance evaluation: During the execution of a single request, three types of indicators are collected simultaneously: (1) Output quality, calculating the accuracy rate through a preset scoring strategy; (2) Inference performance, recording TTFT, TTLT, and error rate; (3) System resources, obtaining GPU memory, utilization rate, CPU, memory, etc. The three types of data are precisely aligned through the request ID to achieve correlation analysis of quality and performance. 4. Providing interactive visualization and automated report generation, improving the analysis efficiency and decision support capabilities of test results: The system has a built-in visualization module that dynamically displays multi-dimensional test results in the form of multi-model latency-throughput line graphs, resource time series curves, etc., and supports filtering by domain, task type, or model version; at the same time, error samples can be highlighted and the original input and output can be traced back. This mechanism avoids the tedious process of manually compiling logs and cross-referencing data, enabling R&D and testing personnel to quickly identify performance bottlenecks, quality defects, or resource anomalies, significantly improving the efficiency of problem localization and optimization.

[0072] In summary, this application embodiment effectively solves key technical problems faced by LLM in actual deployment, such as biased evaluation, difficulty in domain adaptation, low analysis efficiency, and non-reproducibility of the process, by constructing a performance evaluation scheme that integrates test configuration, multi-dimensional data collection, automated execution, and interactive visualization. It not only improves the comprehensiveness and professionalism of testing, but also provides solid technical support for the high-quality delivery, stable operation, and continuous optimization of large models.

[0073] For example, Figure 5This is a schematic diagram illustrating a testing and analysis process provided in an exemplary embodiment of this application. For example... Figure 5 As shown, in step S1, the preparation phase begins, preparing the machines for testing: 1. Obtain basic information for all nodes participating in the test, including username, IP address, and login password, and establish a unified test node list, i.e., generate a test node list based on the basic information of the test nodes. 2. Automatically log in to each test node via the SSH protocol to ensure the reliability and security of remote operation. 3. Perform consistency verification on the hardware and software environment of each test node: At the hardware level, it is required to have the same GPU model, video memory capacity, CPU architecture, and network interconnection bandwidth to eliminate the interference of hardware differences on test results; at the software level, ensure that the driver version, acceleration library, code execution environment, and its dependent libraries are consistent, providing a basic guarantee for fair comparison of subsequent inference services. In step S2, the startup phase begins, starting single-machine or multi-machine inference: 1. Determine the running node for the inference service based on the test scale and deployment architecture, which can be a single machine or a distributed multi-machine system. 2. Execute the preset automated startup script, load the specified LLM on the test node, start the inference service process, and expose HTTP or gRPC interfaces for subsequent stress testing calls. In step S3, the testing phase begins, executing multi-dimensional tests: 1. On a dedicated load generator, a YAML-formatted test configuration file is defined to store test configurations. 2. The YAML file configures parameters such as the test cases, concurrency parameters, inference service address, load generation mode, and semantic scoring logic. 3. The test execution engine sends requests to the inference service in parallel according to the configuration file. In step S4, the analysis phase begins, visualizing and analyzing the results: 1. After inference is complete, service logs and test logs, including test results, are input into an automated visualization platform. 2. Quality, performance, and resource data from different sources are associated using a unified request ID and aggregated in a structured manner according to dimensions such as test case ID, timestamp, model name, version number, and hardware configuration. 3. The aggregated results are persistently stored in a time-series database. 4. Using a built web-based front-end system, multi-dimensional visualizations of the test's TTFT, TPOT, throughput, QPS, test model, hardware configuration, and test case ID are generated. 5. Display current test data and historical test data through indicator selection to compare whether the test results meet expectations.

[0074] The method provided in this application synchronously collects and correlates the output quality indicators, inference performance indicators, and resource consumption indicators of large language models within a unified testing process. This covers a wider range of large model inference testing scenarios, achieving multi-dimensional collaborative evaluation and addressing the issue of one-sided evaluation in related methods. An interactive, multi-dimensional data visualization mechanism is used, where indicators such as GPU utilization, memory utilization, TTFT, and TPOT reflect the accuracy and usability of the test results during and after the testing process. It can collect and output refined performance curves and quantitative evaluation results, reducing the need for manual multi-dimensional comparisons and improving testing efficiency and accuracy. Based on SSH automated verification of multi-machine hardware and software environment consistency, and combined with configuration-driven end-to-end test automation execution mechanism, it ensures the reproducibility of the testing process and the feasibility of engineering implementation.

[0075] The method provided in this application embodiment has at least the following beneficial effects: (1) By collecting three types of indicators—output quality, inference performance, and resource consumption—in a single test process and aligning the data based on the request granularity, the defects of the separation between quality assessment and performance monitoring in traditional testing are overcome, and the comprehensiveness and credibility of the assessment results are significantly improved.

[0076] (2) By supporting user-defined test cases and scoring strategies, the testing system can be flexibly adapted to the semantic verification needs of professional fields, thus solving the technical limitation that general benchmarks cannot cover highly professional tasks.

[0077] (3) By introducing an interactive visualization display mechanism, multidimensional indicators are dynamically presented in the form of radar charts, scatter plots, time series and error sample details, and support drill-down analysis by dimension, which greatly improves the understandability of test results and the efficiency of problem location.

[0078] (4) By adopting structured configuration files to drive the entire testing process, the automated execution of environment verification, service startup, stress application and result archiving is achieved, ensuring that the testing process is reproducible and can be seamlessly integrated into the Continuous Integration (CI) / Continuous Delivery (CD) pipeline, thereby improving the engineering level of large model service quality assurance.

[0079] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0080] It should be noted that the order of the method steps provided in the embodiments of this application can be appropriately adjusted, and the steps can also be added or removed as appropriate. Any method variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0081] Figure 6 This is a schematic diagram of the structure of a testing apparatus for a service provided in an exemplary embodiment of this application. Figure 6 As shown, the device includes: The acquisition module 601 is used to acquire the test configuration, which indicates the test method for the inference service, which is implemented based on LLM. Test module 602 is used to initiate a test request to the inference service according to the test configuration; The acquisition module 601 is further configured to acquire test data of the inference service under multiple test indicators based on the test results corresponding to the test request. The multiple test indicators include indicators related to the output results and operation process of the inference service. The indicators related to the operation process of the inference service are used to reflect at least one of the performance and resource consumption of the inference service.

[0082] In an optional design, the acquisition module 601 is used for: Based on the test results corresponding to the test request, obtain multiple types of the first test data, second test data, and third test data of the inference service; Wherein, the first test data is the test data of the inference service under the output quality index, the second test data is the test data of the inference service under the inference performance index, and the third test data is the test data of the inference service under the resource consumption index.

[0083] In an optional design, the output quality metrics include at least one of the following metrics: answer accuracy and logical consistency; The inference performance metrics include at least one of the following metrics: first-token latency, end-to-end response time, error rate, time per token, throughput, and query rate per second; The resource consumption metrics include at least one of the following metrics: GPU memory usage, CPU usage, memory usage, and resource utilization.

[0084] In an optional design, such as Figure 7 As shown, the device further includes: The generation module 603 is used to generate statistical charts corresponding to the test data according to the multiple test indicators based on the test data of the inference service.

[0085] In an optional design, the generation module 603 is used for: Based on the test data of the inference service, a parent statistical chart corresponding to the test data is generated according to the various test indicators; Based on the test data corresponding to the parent statistical chart, a sub-statistical chart corresponding to the parent statistical chart is generated according to the sub-indicators associated with the various test indicators, and the sub-statistical chart is triggered for display based on the parent statistical chart.

[0086] In an optional design, such as Figure 8 As shown, the device further includes: The verification module 604 is used to perform at least one of hardware consistency verification and software consistency verification on the test node; The test nodes include nodes for deploying the inference service. The hardware consistency check is used to check the consistency of at least one of the GPU model, video memory capacity, CPU architecture, and network bandwidth of different test nodes. The software consistency check is used to check the consistency of at least one of the driver version, acceleration library, runtime environment, and dependency library of different test nodes.

[0087] In an optional design, such as Figure 9 As shown, the device includes: Loading module 605 is used to load the test cases and semantic scoring logic in the test configuration; The test cases include at least one of general test cases and test cases configured according to the target domain, and the semantic scoring logic is used to score the output results of the inference service.

[0088] In an optional design, the test module 602 is used for: Initiate a test request for the inference service according to the test intensity parameters in the test configuration; The test intensity parameters include at least one of the following: test concurrency, total number of requests, context length, model generation length, request interval, and timeout threshold.

[0089] In an optional design, such as Figure 10 As shown, the device further includes: Module 606 is used to establish the association between the test data of the inference service under the various test metrics and the identifier of the test request.

[0090] It should be noted that the service testing apparatus provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the service testing apparatus and the service testing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0091] Embodiments of this application also provide a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the testing method provided by the above-described method embodiments.

[0092] For example, Figure 11 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application, which can be implemented as a server.

[0093] The computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including random access memory (RAM) 1102 and read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the CPU 1101. The computer device 1100 also includes a basic input / output system (I / O system) 1106 to facilitate information transfer between various components within the computer device, and a mass storage device 1107 for storing the operating system 1113, application programs 1114, and other program modules 1115.

[0094] The basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109 for user input, such as a mouse or keyboard. Both the display 1108 and the input device 1109 are connected to the central processing unit 1101 via an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include the input / output controller 1110 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, printer, or other types of output devices.

[0095] The mass storage device 1107 is connected to the central processing unit 1101 via a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable storage media provide non-volatile storage for the computer device 1100. That is, the mass storage device 1107 may include computer-readable storage media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0096] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically-erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1104 and mass storage device 1107 described above can be collectively referred to as memory.

[0097] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1101. The one or more programs contain instructions for implementing the above method embodiments, and the central processing unit 1101 executes the one or more programs to implement the methods provided by the above method embodiments.

[0098] According to various embodiments of this application, the computer device 1100 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1100 can be connected to the network 1112 via the network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 can be used to connect to other types of networks or remote computer device systems (not shown).

[0099] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0100] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, which, when loaded and executed by the processor of a computer device, implements a test method for the services provided in the above-described method embodiments.

[0101] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the test method for the services provided in the above-described method embodiments.

[0102] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0103] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for testing a service, characterized in that, The method includes: Obtain the test configuration, which indicates the testing method for the inference service, which is implemented based on the Large Language Model (LLM). Initiate a test request for the inference service according to the test configuration described above; Based on the test results corresponding to the test request, test data of the inference service under multiple test indicators are obtained. The multiple test indicators include indicators related to the output results and operation process of the inference service. The indicators related to the operation process of the inference service are used to reflect at least one of the performance and resource consumption of the inference service.

2. The method according to claim 1, characterized in that, The step of obtaining test data for the inference service under multiple test metrics based on the test results corresponding to the test request includes: Based on the test results corresponding to the test request, obtain multiple types of the first test data, second test data, and third test data of the inference service; Wherein, the first test data is the test data of the inference service under the output quality index, the second test data is the test data of the inference service under the inference performance index, and the third test data is the test data of the inference service under the resource consumption index.

3. The method according to claim 2, characterized in that, The output quality metrics include at least one of the following: answer accuracy and logical consistency; The inference performance metrics include at least one of the following metrics: first-token latency, end-to-end response time, error rate, time per token, throughput, and query rate per second; The resource consumption indicators include at least one of the following: GPU memory usage, CPU usage, memory usage, and resource utilization.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the test data of the inference service, statistical charts corresponding to the test data are generated according to the various test indicators.

5. The method according to claim 4, characterized in that, The step of generating statistical charts corresponding to the test data based on the test data of the inference service, according to the multiple test metrics, includes: Based on the test data of the inference service, a parent statistical chart corresponding to the test data is generated according to the various test indicators; Based on the test data corresponding to the parent statistical chart, a sub-statistical chart corresponding to the parent statistical chart is generated according to the sub-indicators associated with the various test indicators, and the sub-statistical chart is triggered for display based on the parent statistical chart.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Perform at least one of hardware consistency checks and software consistency checks on the test nodes; The test nodes include nodes for deploying the inference service. The hardware consistency check is used to check the consistency of at least one of the GPU model, video memory capacity, CPU architecture, and network bandwidth of different test nodes. The software consistency check is used to check the consistency of at least one of the driver version, acceleration library, runtime environment, and dependency library of different test nodes.

7. The method according to any one of claims 1 to 3, characterized in that, The method includes: Load the test cases and semantic scoring logic from the test configuration; The test cases include at least one of general test cases and test cases configured according to the target domain, and the semantic scoring logic is used to score the output results of the inference service.

8. The method according to any one of claims 1 to 3, characterized in that, The step of initiating a test request to the inference service according to the test configuration includes: Initiate a test request for the inference service according to the test intensity parameters in the test configuration; The test intensity parameters include at least one of the following: test concurrency, total number of requests, context length, model generation length, request interval, and timeout threshold.

9. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Establish the association between the test data of the inference service under the various test metrics and the identifier of the test request.

10. A service testing apparatus, characterized in that, The device includes: The acquisition module is used to acquire test configuration, which indicates the testing method for the inference service, which is implemented based on LLM. The testing module is used to initiate test requests to the inference service according to the test configuration. The acquisition module is further configured to acquire test data of the inference service under multiple test indicators based on the test results corresponding to the test request. The multiple test indicators include indicators related to the output results and operation process of the inference service. The indicators related to the operation process of the inference service are used to reflect at least one of the performance and resource consumption of the inference service.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement a test method for the service as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one program segment, which is loaded and executed by a processor to implement a test method for the service as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform a test method for the service as described in any one of claims 1 to 9.

Images