An inference service evaluation method and device, computer equipment and a storage medium

Abstract

The application relates to an inference service evaluation method and device, computer equipment and a storage medium. The method comprises the following steps: collecting inference data of each inference service, and constructing a data pool by using the inference data, wherein the inference data is real traffic data of a user; obtaining an inference service to be evaluated, and determining a target evaluation index from a preset evaluation index pool, wherein the target evaluation index comprises at least one main index; obtaining test data from the data pool according to a data driving strategy, and performing a test on the inference service to be evaluated by using the test data to obtain a test result; calculating an index value of the target evaluation index according to the test result, and evaluating the inference service to be evaluated by using the index value. The method can collect real traffic data of an inference service as test data, and evaluate the inference service to be evaluated by automatically performing a test.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reasoning service evaluation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Currently, inference services have become a crucial component of artificial intelligence applications, particularly in areas such as autonomous driving, financial fraud detection, and intelligent customer service. Traditional inference services primarily rely on the HTTP protocol for communication, but with the rise of gRPC technology, more and more services are shifting to this efficient communication protocol based on Protocol Buffers serialization and binary encoding. Compared to HTTP, gRPC not only provides cross-language support but also enhances service flexibility and reliability by supporting streaming processing, authentication mechanisms, and flow control. Furthermore, gRPC's HTTP / 2-based design gives it excellent network performance, making it an ideal choice for large-scale model inference and high-performance computing.

[0003] However, existing technical solutions have revealed shortcomings when dealing with the gRPC protocol. For example, Apache JMeter, a widely used open-source performance testing tool, while effective for stress testing and performance evaluation, does not directly support the gRPC protocol. To test gRPC services, users need to install an HTTP / 2 plugin and perform complex configuration adjustments, which not only increases the difficulty of operation but may also affect the accuracy and convenience of the test. Furthermore, JMeter's integration with inference service platforms is low, limiting its effectiveness as a comprehensive evaluation tool and making it difficult to fully leverage its performance testing advantages in practical applications. Summary of the Invention

[0004] Therefore, it is necessary to provide a reasoning service evaluation method, apparatus, computer equipment, and storage medium capable of evaluating gRPC-based reasoning services in response to the aforementioned technical problems.

[0005] On the one hand, a method for evaluating reasoning services is provided, the method comprising:

[0006] Collect inference data from each inference service and use the inference data to build a data pool, wherein the inference data is the user's real traffic data;

[0007] Obtain the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0008] Test data is obtained from the data pool according to the data-driven strategy, and the test data is used to perform tests on the inference service to be evaluated to obtain test results;

[0009] The target evaluation index value is calculated based on the test results, and the inference service to be evaluated is evaluated using the index value.

[0010] In one embodiment, the collection of inference data from each inference service includes:

[0011] In response to the container group creation request of the inference service, check whether a preset evaluation identifier exists in the creation request;

[0012] In response to the presence of a preset evaluation identifier in the creation request, the creation request is intercepted by the admission controller.

[0013] The creation of the container group is completed after adding the proxy container to the container group.

[0014] The proxy container intercepts access requests to the inference service container and creates a reverse proxy object for the access request, wherein the proxy address of the reverse proxy object is the address of the inference service container.

[0015] The inference data of the inference service is obtained through the reverse proxy object.

[0016] In one embodiment, constructing a data pool using the inference data includes:

[0017] A test dataset is constructed based on the collected inference data and / or virtual data;

[0018] In response to constructing the test dataset using the inference data and virtual data, the total amount of data in the test dataset and the mixing ratio of inference data and virtual data in the test dataset are set.

[0019] A data pool is constructed based on the test dataset.

[0020] In one embodiment, the method further includes:

[0021] Construct an evaluation metric pool, which includes preset metrics and custom metrics. The preset metrics include request count, throughput, request latency, and request success rate.

[0022] The indicator information, query conditions, and result parsing formula of the custom indicator are preset and stored in the preset query address.

[0023] In one embodiment, the step of obtaining test data from the data pool according to a data-driven strategy, and performing tests on the inference service to be evaluated using the test data to obtain test results includes:

[0024] Determine the data-driven strategy and set the data-driven parameters. The data-driven strategy includes executing tests a fixed number of times or for a fixed duration. The data-driven parameters include the number of concurrent threads, the execution interval, and the number of execution rounds / execution duration.

[0025] Based on the number of concurrent threads, the corresponding test data is obtained from the data pool, and the inference service interface to be evaluated is accessed to obtain the corresponding test results;

[0026] After the execution interval, the inference service interface to be evaluated is accessed again until the execution round or execution duration is reached.

[0027] In one embodiment, the step of calculating the index value of the target evaluation index based on the test results, and evaluating the inference service to be evaluated using the index value, includes:

[0028] Calculate the value of the preset index based on the test results; and / or,

[0029] Based on the query conditions corresponding to the custom indicator, query parameters are obtained from the test results, and a request is sent to the query address based on the query parameters to obtain the query results;

[0030] The query result is parsed using the result parsing expression corresponding to the custom indicator to obtain the indicator value of the custom indicator;

[0031] In response to the target evaluation index, auxiliary indicators are also included, and it is determined whether the index value of the auxiliary indicators is within a preset range;

[0032] If the value of the auxiliary indicator exceeds the preset range, the score of the reasoning service to be evaluated is determined to be 0.

[0033] If the value of the auxiliary indicator is within the preset range and the selected evaluation type is service recommendation, then there are multiple inference services to be evaluated.

[0034] For each of the aforementioned reasoning services to be evaluated, the index values ​​of the aforementioned reasoning services are scored according to a preset scoring level to obtain the scores of each target evaluation index.

[0035] The score of the target evaluation index is weighted and summed with the preset index weight coefficients to obtain the recommendation score of the reasoning service to be evaluated.

[0036] The reasoning service with the highest recommendation score is selected as the target reasoning service for recommendation.

[0037] In one embodiment, evaluating the inference service to be evaluated using the indicator value further includes:

[0038] If the selected evaluation type is performance testing, then the inference service to be evaluated is one.

[0039] The target evaluation indicator is scored based on the difference between its value and a preset threshold.

[0040] The performance score of the inference service to be evaluated is obtained by weighting and summing the scores of the target evaluation indicators with preset indicator weight coefficients.

[0041] On the other hand, a reasoning service evaluation apparatus is provided, the apparatus comprising:

[0042] The collection module is used to collect inference data from each inference service and to build a data pool using the inference data, wherein the inference data is the user's real traffic data;

[0043] The acquisition module is used to acquire the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0044] The testing module is used to obtain test data from the data pool according to the data-driven strategy, and to perform tests on the inference service to be evaluated using the test data to obtain test results;

[0045] The evaluation module is used to calculate the index value of the target evaluation index based on the test results, and to evaluate the reasoning service to be evaluated using the index value.

[0046] In another aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0047] Collect inference data from each inference service and use the inference data to build a data pool, wherein the inference data is the user's real traffic data;

[0048] Obtain the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0049] Test data is obtained from the data pool according to the data-driven strategy, and the test data is used to perform tests on the inference service to be evaluated to obtain test results;

[0050] The target evaluation index value is calculated based on the test results, and the inference service to be evaluated is evaluated using the index value.

[0051] In another aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0052] Collect inference data from each inference service and use the inference data to build a data pool, wherein the inference data is the user's real traffic data;

[0053] Obtain the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0054] Test data is obtained from the data pool according to the data-driven strategy, and the test data is used to perform tests on the inference service to be evaluated to obtain test results;

[0055] The target evaluation index value is calculated based on the test results, and the inference service to be evaluated is evaluated using the index value.

[0056] The aforementioned inference service evaluation method, apparatus, computer equipment, and storage medium construct a data pool by collecting real traffic data of the inference service, ensuring the authenticity of the test data, improving the accuracy and reliability of the experiment, and allowing selection of target evaluation indicators according to evaluation needs, meeting diverse user evaluation requirements. The score of the inference service to be evaluated is calculated based on the indicator values ​​of the target evaluation indicators, and the test is executed and the test results are obtained through automated execution, improving testing efficiency and shortening the evaluation cycle. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the reasoning service evaluation method in one embodiment;

[0058] Figure 2 This is a flowchart illustrating the reasoning service evaluation method in another embodiment;

[0059] Figure 3 This is a flowchart illustrating the steps for creating an inference service in one embodiment;

[0060] Figure 4 This is a flowchart illustrating the steps for collecting inference data in one embodiment;

[0061] Figure 5 Here is a block diagram of the evaluation index pool in one embodiment;

[0062] Figure 6This is a flowchart illustrating the steps for obtaining the value of a custom indicator in one embodiment.

[0063] Figure 7 This is a flowchart illustrating a service recommendation method in one embodiment;

[0064] Figure 8 This is a structural block diagram of a reasoning service evaluation device in one embodiment;

[0065] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] In one embodiment, such as Figure 1-6 As shown, a reasoning service evaluation method is provided, including the following steps:

[0068] Step S1: Collect inference data from each inference service and construct a data pool using the inference data, wherein the inference data is the user's actual traffic data.

[0069] In one embodiment, the specific steps for collecting the inference data of the inference service to be evaluated include: in response to a container group creation request for the inference service, checking whether a preset evaluation identifier exists in the creation request; in response to the existence of a preset evaluation identifier in the creation request, intercepting the creation request through an admission controller; completing the creation of the container group after adding a proxy container to the container group; intercepting the access request of the inference service business container through the proxy container, and creating a reverse proxy object for the access request, wherein the proxy address of the reverse proxy object is the address of the inference service business container; and obtaining the inference data of the inference service through the reverse proxy object.

[0070] Specifically, data collection for the gRPC inference service is achieved using sidecar injection. A corresponding hook service and admission controller are created in the Kubernetes cluster. The admission controller intercepts the creation of inference service pods with specific preset evaluation identifiers. Before the creation request is sent to the Kubernetes API Server, the request is automatically intercepted and forwarded to the hook service. The hook service then obtains the original pod creation data, adds a new proxy container while retaining the original pod, and forwards the modified data to the Kubernetes API Server to create the pod. The proxy container listens on the service's business container port. When a user requests access to the gRPC inference service business container, the proxy container first intercepts the request and then creates a gRPC reverse proxy object for each access request.

[0071] Furthermore, the access request is fed back to the target address via a gRPC reverse proxy object, and the response information corresponding to the access request is received from the target address. The target address is the actual receiving address of the access request, which is the address of the target server used to process the access request. The gRPC reverse proxy object analyzes the request content of the access request and the content of the response information to extract a preset information set. The preset information set includes the request ID, request time, response ID, response time, response status, request type, length of the request content, response type, and length of the response information. The content in the preset information set is standardized and added to the message header, and then encapsulated together with the request content and response content and stored in a forwarding queue. Multiple message processing threads (workers) retrieve messages from the queue and forward them to the data processing and storage center of the inference platform. The preset information set, request content, and response content together constitute the inference data. By analyzing the inference data, a performance impact factor corresponding to the gRPC inference service is determined. The performance impact factor can be used to evaluate the gRPC inference service.

[0072] Based on the steps described above, inference data can be collected in a Kubernetes cluster via sidecar injection, thereby obtaining real user traffic data. This provides necessary data support for subsequent metric evaluation, making the experimental results more accurate. Furthermore, this method does not require modification to the existing service architecture, making it easy to extend and maintain.

[0073] In one embodiment, the specific steps of constructing a data pool using the inference data include: constructing a test dataset based on the collected inference data and / or virtual data; in response to constructing the test dataset using both the inference data and virtual data, setting the total amount of data in the test dataset and the mixing ratio of inference data and virtual data in the test dataset; and constructing a data pool based on the test dataset.

[0074] Specifically, in actual testing, there are three ways to construct test data: all virtual data, all real data, and a mixture of real and virtual data. When using real data, the collected inference data needs to be queried and exported according to the required conditions, exported to a file, and then imported into the data construction system. When using virtual data, the user needs to fill in the interface definition or import the interface document. The program will then construct random input parameters as evaluation data based on the interface definition. The data construction system allows setting the total amount of data and the mixing ratio. For example, setting the mixing ratio to 3:1 means that the ratio of real data to virtual data is 3:1. Ultimately, the real data pool and the virtual data pool together form the final data pool, which is then sent to the inference service interface to be evaluated according to the rules driven by the data.

[0075] Based on the above steps, appropriate data construction methods can be selected according to the test scenario, enhancing the flexibility and diversity of the test, thereby more comprehensively covering various test scenarios. Furthermore, the use of real data increases the realism of the test process and improves the accuracy of the test results.

[0076] Step S2: Obtain the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0077] It should be noted that the evaluation of inference services mainly involves two experimental scenarios: one is to conduct performance testing on a single service, and the other is to recommend services to different inference services.

[0078] Specifically, the experiment creation interface is called to create corresponding experiment records, and an experiment job is created in the background for each inference service to be evaluated, which is responsible for batch processing short-lived, one-off tasks in Kubernetes. Several important parameters need to be entered when creating an experiment, including the experiment name, the namespace where the experiment is located, the primary metric (including expected trend and actual placeholder values), the secondary metric (including preset thresholds, expected trend, and actual placeholder values), the inference service and interface to be evaluated, and the test type. The test type includes performance testing and service recommendation. If performance testing is selected, one inference service and interface to be evaluated should be entered; if service recommendation is selected, two or more different inference services to be evaluated and their related interfaces should be added (e.g., grpc: / / myservice:8080 / example and grpc: / / otherservice:8080 / example).

[0079] There is only one primary indicator, which is a non-rigid indicator. It mainly focuses on the expected trend. If the trend is upper, the higher the indicator value, the better; if the trend is lower, the lower the indicator value, the better. Secondary indicators are not mandatory and multiple can be added. They are rigid indicators, meaning that secondary indicators that do not reach the predetermined threshold will not be recommended.

[0080] In one embodiment, the method further includes: constructing an evaluation indicator pool, the evaluation indicators including preset indicators and custom indicators, the preset indicators including request count, throughput, request latency and request success rate; and pre-setting the indicator information, query conditions and result parsing formulas of the custom indicators and storing them in a preset query address.

[0081] In actual experiments, pre-defined metrics and custom metrics can be used in combination to evaluate the inference service. Pre-defined metrics can be used for basic performance testing, meeting the basic evaluation requirements of the inference service. Custom metrics can be used through a custom template file, which is a .tpl file, as shown in the example below:

[0082] url:grpc: / / myservice:8080 / example

[0083] name:performance

[0084] metrics:

[0085] -name:request-latency

[0086] type:gauge

[0087] description:|

[0088] Request latency

[0089] params:

[0090] -name:query

[0091] value:latency

[0092] jqExpression:.data.latency|tonumber

[0093] unit:ms

[0094] -name:request-count

[0095]

[0096] The main parameters in this template are analyzed as shown in Table 1:

[0097] Table 1 Custom Template Parameters

[0098] url It supports gRPC and is the metric query address; multiple metrics can be defined using the same URL. name Indicator Provider Name metrics Indicator Definition name Indicator Name type type description Indicator Definition Description params Query conditions JqExpression Result parsing expression unit The unit of measurement is left blank; the default value is 1, meaning there is no unit.

[0099] Specifically, the user provides a URL address and defines the metric calculation rules in the interface pointed to by the URL. The interface can then provide multiple evaluation metrics, requiring the user to specify the name, definition, type, unit, query conditions, and metric result parsing expression for each metric. In actual use, the backend will send a request to the defined interface according to the query conditions and automatically parse the metric value according to the metric result parsing expression, recording the metric result.

[0100] JqExpression follows the parsing syntax of jQuery expressions, allowing data to be converted into numeric types for comparison. For example, `jqExpression:.data.latency|tonumber` means that if the result is `r`, then the value of `r["data"]["latency"]` will be retrieved and converted into a numeric type. The metric type `type` includes two types: `counter` and `gauge`. `counter` represents a statistical type, a monotonically increasing counter, such as the total number of requests. `gauge` represents a real-time numeric type with arbitrary fluctuations, such as the request latency of each request. `params` are query conditions, consisting of a set of key-value pairs (name and value). In the first example metric, `params` actually passes `{"query":"latency"}`. `params` can be set with templated parameters, i.e., placeholders. For example, in the second example metric, `$name` in `params` needs to be replaced with the actual value of `$name` during the experiment.

[0101] Based on the above steps, by combining preset and custom metrics, users can define evaluation metrics through custom template files. They only need to provide a URL address and define the metric calculation rules in the interface pointed to by that address. Different query results can be obtained through different query conditions, that is, different evaluation metrics can be generated. This not only meets basic performance evaluation needs, but also provides high flexibility and extensibility, thereby meeting the diverse evaluation needs of users.

[0102] Step S3: Obtain test data from the data pool according to the data-driven strategy, and perform tests on the inference service to be evaluated using the test data to obtain test results.

[0103] In one embodiment, the specific steps of obtaining test data from the data pool according to the data-driven strategy and performing tests on the inference service to be evaluated using the test data to obtain test results include: determining the data-driven strategy and setting the data-driven parameters of the data-driven strategy, wherein the data-driven strategy includes performing tests a fixed number of times or for a fixed duration, and the data-driven parameters include the number of concurrent threads, the execution interval, and the number of execution rounds / execution duration; obtaining corresponding test data from the data pool according to the number of concurrent threads and accessing the interface of the inference service to be evaluated to obtain the corresponding test results; and after the execution interval, accessing the interface of the inference service to be evaluated again until the number of execution rounds or execution duration is reached.

[0104] Among them, the number of concurrent threads refers to the number of requests initiated at the same time, the number of execution rounds refers to the number of times the request operation is repeated, the execution interval refers to the interval between each group of concurrent threads, in seconds, and the execution duration refers to the time for conducting the experiment.

[0105] Based on the above steps, a suitable data-driven strategy can be selected according to different testing needs. An experiment can be created for each service to be evaluated, and then a fixed number of tests or a fixed duration of tests can be performed. Data is retrieved from the data pool and requests are initiated according to preset data-driven parameters, reducing the need for manual intervention.

[0106] Step S4: Calculate the index value of the target evaluation index based on the test results, and evaluate the reasoning service to be evaluated using the index value.

[0107] In one embodiment, the step of calculating the indicator value of the target evaluation indicator based on the test results and evaluating the inference service to be evaluated using the indicator value includes: calculating the indicator value of the preset indicator based on the test results; and / or, obtaining query parameters from the test results based on the query conditions corresponding to the custom indicator, and sending a request to the query address based on the query parameters to obtain the query result; parsing the query result using the result parsing expression corresponding to the custom indicator to obtain the indicator value of the custom indicator; and, in response to the target evaluation indicator further including auxiliary indicators, determining whether the indicator value of the auxiliary indicator is within the preset range. Within the preset range; in response to the indicator value of the auxiliary indicator exceeding the preset range, the score of the reasoning service to be evaluated is determined to be 0; in response to the indicator value of the auxiliary indicator being within the preset range, and the selected evaluation type being service recommendation, then there are multiple reasoning services to be evaluated; for each reasoning service to be evaluated, the indicator value of the reasoning service to be evaluated is scored according to a preset scoring level to obtain the score of each target evaluation indicator; the score of the target evaluation indicator is weighted and summed with a preset indicator weight coefficient to obtain the recommendation score of the reasoning service to be evaluated; the reasoning service to be evaluated with the highest recommendation score is selected as the target reasoning service for recommendation.

[0108] Specifically, the values ​​of several preset indicators can be calculated in the following ways:

[0109] Request count = count endtime -count starttime That is, the difference between the number of requests at the end of the experiment and the number of requests at the beginning of the experiment;

[0110] Throughput QPS = (count t+10 -count t ) / 10, which is the query rate per second, equal to the number of response requests per second, reflects the maximum throughput capacity;

[0111] Request latency (50%, 90%, 95%, 99%): Taking 50% request latency TP50 as an example, sort all request latencies within the experimental period in ascending order. Then, calculate the total number of requests multiplied by 50% to find the number of requests C at the 50th percentile. Finally, the request latency at position C is the 50% request latency. Similarly, the 90%, 95%, and 99% request latencies can be calculated.

[0112] Success Rate = count OK / count, or request success rate, is equal to the number of gRPC requests with an OK status divided by the total number of requests in all statuses.

[0113] For example, when the evaluation type is selected as service recommendation, and the target evaluation metrics include a primary metric and two secondary metrics, the metric values ​​for each inference service to be evaluated are shown in Table 2:

[0114] Table 2. Indicator values ​​of the reasoning service to be evaluated

[0115]

[0116] Users set the weight ratio p of the primary indicator and each secondary indicator based on their own judgment. For example, if the ratio is 2:1:1, then the primary indicator will have a weight coefficient of 2 / 4 throughout the entire evaluation process, secondary indicator 1 will have a weight of 1 / 4, and secondary indicator 2 will have a weight of 1 / 4. After setting the weight ratio, the analysis can be submitted.

[0117] The analysis interface first determines whether the secondary indicators meet the preset range. For example, if the lower threshold of secondary indicator 1 is 1.5, and the value of secondary indicator 1 for Service1 is 1.0, then Service1 is directly determined to have a final score of 0, and Service1 will not be recommended during service recommendation. If the secondary indicators are within the range, then the indicator values ​​of the secondary indicators are sorted according to the expected trend; similarly, the indicator values ​​of the primary indicators are also sorted according to the expected trend.

[0118] When calculating the score for each reasoning service to be evaluated, the score for each indicator of that service is first calculated. For example, for the main indicator, the minimum value of the main indicator is subtracted from the maximum value, and then the difference is divided by the number of rating levels n to obtain an interval difference.

[0119]

[0120] Where R max The maximum value among the main indicators, R min The minimum value among the main indicators, n is the number of rating levels, which defaults to a 10-point scale, i.e., n is 10.

[0121] Next, the actual values ​​of the main indicator will be segmented and scored. Since the expected trend of the main indicator is upper, R is set. min ~R min +Δ is 1 point, R min +Δ~R min +2Δ equals 2 points, and so on. Similarly, the scores for other target evaluation indicators can be obtained. Then, based on the user-set weight ratio p and the score S of each target evaluation indicator, a summation calculation is performed to obtain the recommendation score for the service to be evaluated.

[0122]

[0123] The reason for not providing a recommended score directly during the experiment is that users' subjective ratings may change. Therefore, the direct experimental results are first fed back to the user, who then sets the weights for their ratings. This allows users to analyze the experimental results multiple times, rather than having to re-run the experiment every time a different rating weight is applied. Both the final experimental and analytical results are persistently stored for further analysis by the user.

[0124] In one embodiment, evaluating the inference service to be evaluated using the indicator value further includes: in response to the selected evaluation type being performance testing, the inference service to be evaluated is one; scoring the target evaluation indicator based on the difference between the indicator value of the target evaluation indicator and a preset threshold; and weighting and summing the score of the target evaluation indicator with a preset indicator weighting coefficient to obtain the performance score of the inference service to be evaluated.

[0125] It should be understood that, although Figure 1-7 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-7 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0126] In one embodiment, such as Figure 8 As shown, a reasoning service evaluation device is provided, comprising: a collection module, an acquisition module, a testing module, and an evaluation module, wherein:

[0127] The collection module is used to collect inference data from each inference service and to build a data pool using the inference data, wherein the inference data is the user's real traffic data;

[0128] The acquisition module is used to acquire the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0129] The testing module is used to obtain test data from the data pool according to the data-driven strategy, and to perform tests on the inference service to be evaluated using the test data to obtain test results;

[0130] The evaluation module is used to calculate the index value of the target evaluation index based on the test results, and to evaluate the reasoning service to be evaluated using the index value.

[0131] In one embodiment, the collection module is further configured to, in response to the container group creation request of the inference service, check whether a preset evaluation identifier exists in the creation request; in response to the existence of the preset evaluation identifier in the creation request, intercept the creation request through an admission controller; complete the creation of the container group after adding a proxy container to the container group; intercept the access request of the inference service business container through the proxy container, and create a reverse proxy object for the access request, wherein the proxy address of the reverse proxy object is the address of the inference service business container; and obtain the inference data of the inference service through the reverse proxy object.

[0132] In one embodiment, the collection module is further configured to construct a test dataset based on the collected inference data and / or virtual data; in response to constructing the test dataset using the inference data and virtual data, set the total amount of data in the test dataset and the mixing ratio of inference data and virtual data in the test dataset; and construct a data pool based on the test dataset.

[0133] In one embodiment, the apparatus further includes a construction module for constructing an evaluation metric pool. The evaluation metrics include preset metrics and custom metrics. The preset metrics include the number of requests, throughput, request latency, and request success rate. The custom metrics' metric information, query conditions, and result parsing formulas are preset and stored in a preset query address.

[0134] In one embodiment, the testing module is further configured to determine a data-driven strategy and set data-driven parameters. The data-driven strategy includes executing tests a fixed number of times or for a fixed duration. The data-driven parameters include the number of concurrent threads, the execution interval, and the number of execution rounds / duration. Based on the number of concurrent threads, the module retrieves corresponding test data from the data pool and accesses the inference service interface to be evaluated to obtain the corresponding test results. After the execution interval, the module accesses the inference service interface to be evaluated again until the number of execution rounds or the execution duration are reached.

[0135] In one embodiment, the evaluation module is further configured to calculate the index value of the preset index based on the test results; and / or, obtain query parameters from the test results based on the query conditions corresponding to the custom index, and send a request to the query address based on the query parameters to obtain the query results; parse the query results through the result parsing expression corresponding to the custom index to obtain the index value of the custom index; in response to the target evaluation index further including auxiliary indexes, determine whether the index value of the auxiliary index is within a preset range; in response to the index value of the auxiliary index exceeding the preset range, determine that the score of the reasoning service to be evaluated is 0; in response to the index value of the auxiliary index being within the preset range, and the selected evaluation type being service recommendation, then there are multiple reasoning services to be evaluated; for each reasoning service to be evaluated, score the index value of the reasoning service to be evaluated according to a preset scoring level to obtain the score of each target evaluation index; weight and sum the score of the target evaluation index with a preset index weight coefficient to obtain the recommendation score of the reasoning service to be evaluated; select the reasoning service to be evaluated with the highest recommendation score as the target reasoning service for recommendation.

[0136] In one embodiment, the evaluation module is further configured to respond to the selected evaluation type being performance testing, wherein the inference service to be evaluated is one; to score the target evaluation indicator based on the difference between the indicator value of the target evaluation indicator and a preset threshold; and to perform a weighted summation of the score of the target evaluation indicator with a preset indicator weight coefficient to obtain the performance score of the inference service to be evaluated.

[0137] Specific limitations regarding the reasoning service evaluation device can be found in the limitations of the reasoning service evaluation method described above, and will not be repeated here. Each module in the aforementioned reasoning service evaluation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0138] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores inference data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an inference service evaluation method.

[0139] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0141] Collect inference data from each inference service and use the inference data to build a data pool, wherein the inference data is the user's real traffic data;

[0142] Obtain the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0143] Test data is obtained from the data pool according to the data-driven strategy, and the test data is used to perform tests on the inference service to be evaluated to obtain test results;

[0144] The target evaluation index value is calculated based on the test results, and the inference service to be evaluated is evaluated using the index value.

[0145] In one embodiment, when the processor executes the computer program, it further performs the following steps: in response to the container group creation request of the inference service, checks whether a preset evaluation identifier exists in the creation request; in response to the existence of the preset evaluation identifier in the creation request, intercepts the creation request through an admission controller; completes the creation of the container group after adding a proxy container to the container group; intercepts the access request of the inference service business container through the proxy container, and creates a reverse proxy object for the access request, wherein the proxy address of the reverse proxy object is the address of the inference service business container; and obtains the inference data of the inference service through the reverse proxy object.

[0146] In one embodiment, when the processor executes the computer program, it further performs the following steps: constructing a test dataset based on the collected inference data and / or virtual data; in response to constructing the test dataset using the inference data and virtual data, setting the total amount of data in the test dataset and the mixing ratio of inference data and virtual data in the test dataset; and constructing a data pool based on the test dataset.

[0147] In one embodiment, when the processor executes the computer program, it further performs the following steps: constructing an evaluation metric pool, the evaluation metrics including preset metrics and custom metrics, the preset metrics including request count, throughput, request latency and request success rate; and pre-setting the metric information, query conditions and result parsing formulas of the custom metrics and storing them in a preset query address.

[0148] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a data-driven strategy and setting data-driven parameters, wherein the data-driven strategy includes executing a fixed number of tests or a fixed duration of tests, and the data-driven parameters include the number of concurrent threads, the execution interval, and the number of execution rounds / execution duration; obtaining corresponding test data from the data pool according to the number of concurrent threads, and accessing the inference service interface to be evaluated to obtain the corresponding test results; and after the execution interval, accessing the inference service interface to be evaluated again until the number of execution rounds or execution duration is reached.

[0149] In one embodiment, when the processor executes the computer program, it further implements the following steps: calculating the index value of the preset index based on the test results; and / or, obtaining query parameters from the test results based on the query conditions corresponding to the custom index, and sending a request to the query address based on the query parameters to obtain the query results; parsing the query results through the result parsing expression corresponding to the custom index to obtain the index value of the custom index; in response to the target evaluation index further including auxiliary indexes, determining whether the index value of the auxiliary index is within a preset range; in response to the index value of the auxiliary index exceeding the preset range, determining that the score of the reasoning service to be evaluated is 0; in response to the index value of the auxiliary index being within the preset range, and the selected evaluation type being service recommendation, then there are multiple reasoning services to be evaluated; for each reasoning service to be evaluated, scoring the index value of the reasoning service to be evaluated according to a preset scoring level to obtain the score of each target evaluation index; weighting and summing the score of the target evaluation index with a preset index weight coefficient to obtain the recommendation score of the reasoning service to be evaluated; selecting the reasoning service to be evaluated with the highest recommendation score as the target reasoning service for recommendation.

[0150] In one embodiment, when the processor executes the computer program, it further performs the following steps: in response to the selected evaluation type being performance testing, the inference service to be evaluated is one; the target evaluation indicator is scored based on the difference between the indicator value of the target evaluation indicator and a preset threshold; the score of the target evaluation indicator is weighted and summed with a preset indicator weight coefficient to obtain the performance score of the inference service to be evaluated.

[0151] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0152] Collect inference data from each inference service and use the inference data to build a data pool, wherein the inference data is the user's real traffic data;

[0153] Obtain the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0154] Test data is obtained from the data pool according to the data-driven strategy, and the test data is used to perform tests on the inference service to be evaluated to obtain test results;

[0155] The target evaluation index value is calculated based on the test results, and the inference service to be evaluated is evaluated using the index value.

[0156] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: in response to the container group creation request of the inference service, checking whether there is a preset evaluation identifier in the creation request; in response to the presence of the preset evaluation identifier in the creation request, intercepting the creation request through an admission controller; completing the creation of the container group after adding a proxy container to the container group; intercepting the access request of the inference service business container through the proxy container, and creating a reverse proxy object for the access request, wherein the proxy address of the reverse proxy object is the address of the inference service business container; and obtaining the inference data of the inference service through the reverse proxy object.

[0157] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: constructing a test dataset based on the collected inference data and / or virtual data; in response to constructing the test dataset using the inference data and virtual data, setting the total amount of data in the test dataset and the mixing ratio of inference data and virtual data in the test dataset; and constructing a data pool based on the test dataset.

[0158] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: constructing an evaluation metric pool, the evaluation metrics including preset metrics and custom metrics, the preset metrics including request count, throughput, request latency and request success rate; and pre-setting the metric information, query conditions and result parsing formulas of the custom metrics and storing them in a preset query address.

[0159] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a data-driven strategy and setting data-driven parameters, the data-driven strategy including executing a fixed number of tests or a fixed duration, the data-driven parameters including the number of concurrent threads, the execution interval, and the number of execution rounds / execution duration; obtaining corresponding test data from the data pool according to the number of concurrent threads, and accessing the inference service interface to be evaluated to obtain the corresponding test results; after the execution interval, accessing the inference service interface to be evaluated again until the number of execution rounds or execution duration is reached.

[0160] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: calculating the index value of the preset index based on the test results; and / or, obtaining query parameters from the test results based on the query conditions corresponding to the custom index, and sending a request to the query address based on the query parameters to obtain the query results; parsing the query results through the result parsing expression corresponding to the custom index to obtain the index value of the custom index; in response to the target evaluation index further including auxiliary indexes, determining whether the index value of the auxiliary index is within a preset range; in response to the index value of the auxiliary index exceeding the preset range, determining that the score of the reasoning service to be evaluated is 0; in response to the index value of the auxiliary index being within the preset range, and the selected evaluation type being service recommendation, then there are multiple reasoning services to be evaluated; for each reasoning service to be evaluated, scoring the index value of the reasoning service to be evaluated according to a preset scoring level to obtain the score of each target evaluation index; weighting and summing the score of the target evaluation index with a preset index weight coefficient to obtain the recommendation score of the reasoning service to be evaluated; selecting the reasoning service to be evaluated with the highest recommendation score as the target reasoning service for recommendation.

[0161] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: in response to the selected evaluation type being performance testing, the inference service to be evaluated is one; the target evaluation indicator is scored based on the difference between the indicator value of the target evaluation indicator and a preset threshold; the score of the target evaluation indicator is weighted and summed with a preset indicator weight coefficient to obtain the performance score of the inference service to be evaluated.

[0162] In one embodiment, a computer product is provided that stores a computer program, which, when executed by a processor, performs the following steps:

[0163] Collect inference data from each inference service and use the inference data to build a data pool, wherein the inference data is the user's real traffic data;

[0164] Obtain the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index.

[0165] Test data is obtained from the data pool according to the data-driven strategy, and the test data is used to perform tests on the inference service to be evaluated to obtain test results;

[0166] The target evaluation index value is calculated based on the test results, and the inference service to be evaluated is evaluated using the index value.

[0167] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: in response to the container group creation request of the inference service, checking whether there is a preset evaluation identifier in the creation request; in response to the presence of the preset evaluation identifier in the creation request, intercepting the creation request through an admission controller; completing the creation of the container group after adding a proxy container to the container group; intercepting the access request of the inference service business container through the proxy container, and creating a reverse proxy object for the access request, wherein the proxy address of the reverse proxy object is the address of the inference service business container; and obtaining the inference data of the inference service through the reverse proxy object.

[0168] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: constructing a test dataset based on the collected inference data and / or virtual data; in response to constructing the test dataset using the inference data and virtual data, setting the total amount of data in the test dataset and the mixing ratio of inference data and virtual data in the test dataset; and constructing a data pool based on the test dataset.

[0169] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: constructing an evaluation metric pool, the evaluation metrics including preset metrics and custom metrics, the preset metrics including request count, throughput, request latency and request success rate; and pre-setting the metric information, query conditions and result parsing formulas of the custom metrics and storing them in a preset query address.

[0170] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a data-driven strategy and setting data-driven parameters, the data-driven strategy including executing a fixed number of tests or a fixed duration, the data-driven parameters including the number of concurrent threads, the execution interval, and the number of execution rounds / execution duration; obtaining corresponding test data from the data pool according to the number of concurrent threads, and accessing the inference service interface to be evaluated to obtain the corresponding test results; after the execution interval, accessing the inference service interface to be evaluated again until the number of execution rounds or execution duration is reached.

[0171] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: calculating the index value of the preset index based on the test results; and / or, obtaining query parameters from the test results based on the query conditions corresponding to the custom index, and sending a request to the query address based on the query parameters to obtain the query results; parsing the query results through the result parsing expression corresponding to the custom index to obtain the index value of the custom index; in response to the target evaluation index further including auxiliary indexes, determining whether the index value of the auxiliary index is within a preset range; in response to the index value of the auxiliary index exceeding the preset range, determining that the score of the reasoning service to be evaluated is 0; in response to the index value of the auxiliary index being within the preset range, and the selected evaluation type being service recommendation, then there are multiple reasoning services to be evaluated; for each reasoning service to be evaluated, scoring the index value of the reasoning service to be evaluated according to a preset scoring level to obtain the score of each target evaluation index; weighting and summing the score of the target evaluation index with a preset index weight coefficient to obtain the recommendation score of the reasoning service to be evaluated; selecting the reasoning service to be evaluated with the highest recommendation score as the target reasoning service for recommendation.

[0172] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: in response to the selected evaluation type being performance testing, the inference service to be evaluated is one; the target evaluation indicator is scored based on the difference between the indicator value of the target evaluation indicator and a preset threshold; the score of the target evaluation indicator is weighted and summed with a preset indicator weight coefficient to obtain the performance score of the inference service to be evaluated.

[0173] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for evaluating reasoning services, characterized in that, include: Inference data from various inference services is collected, and a data pool is constructed using the inference data, wherein the inference data is real user traffic data. The data pool is constructed in the following manner: a test dataset is constructed based on the collected inference data and / or virtual data; in response to constructing the test dataset using both inference data and virtual data, the total amount of data in the test dataset and the mixing ratio of inference data and virtual data in the test dataset are set; the data pool is constructed based on the test dataset; wherein the test dataset is constructed based on at least one of the following test data types: all virtual data, all real data, and a mixture of real data and virtual data. Obtain the inference service to be evaluated, wherein the inference service to be evaluated is a gRPC-based inference service; Create an evaluation metric pool, which includes preset metrics and custom metrics. The preset metrics include request count, throughput, request latency, and request success rate. Pre-set the metric information, query conditions, and result parsing formulas of the custom metrics and store them in a preset query address. Determine the target evaluation metric from the evaluation metric pool, which includes at least one main metric. Test data is obtained from the data pool according to the data-driven strategy, and the test data is used to perform tests on the inference service to be evaluated to obtain test results; The target evaluation index is calculated based on the test results, and the inference service to be evaluated is evaluated using the index value. The test type selected during the evaluation includes at least one of the following: performance testing of a single service, or service recommendation of different inference services. If performance testing is selected, one inference service to be evaluated and its interface are filled in. If service recommendation is selected, two or more different inference services to be evaluated and their related interfaces are added.

2. The reasoning service evaluation method according to claim 1, characterized in that, The collection of reasoning data from each reasoning service includes: In response to the container group creation request of the inference service, check whether a preset evaluation identifier exists in the creation request; In response to the presence of a preset evaluation identifier in the creation request, the creation request is intercepted by the admission controller. The creation of the container group is completed after adding the proxy container to the container group. The proxy container intercepts access requests to the inference service container and creates a reverse proxy object for the access request, wherein the proxy address of the reverse proxy object is the address of the inference service container. The inference data of the inference service is obtained through the reverse proxy object.

3. The reasoning service evaluation method according to claim 1, characterized in that, The step of obtaining test data from the data pool according to a data-driven strategy, and performing tests on the inference service to be evaluated using the test data to obtain test results includes: Determine the data-driven strategy and set the data-driven parameters. The data-driven strategy includes executing tests a fixed number of times or for a fixed duration. The data-driven parameters include the number of concurrent threads, the execution interval, and the number of execution rounds / execution duration. Based on the number of concurrent threads, the corresponding test data is obtained from the data pool, and the inference service interface to be evaluated is accessed to obtain the corresponding test results; After the execution interval, the inference service interface to be evaluated is accessed again until the execution round or execution duration is reached.

4. The reasoning service evaluation method according to claim 1, characterized in that, The step of calculating the target evaluation index value based on the test results, and evaluating the reasoning service to be evaluated using the index value, includes: Calculate the value of the preset index based on the test results; and / or, Based on the query conditions corresponding to the custom indicator, query parameters are obtained from the test results, and a request is sent to the query address based on the query parameters to obtain the query results; The query result is parsed using the result parsing expression corresponding to the custom indicator to obtain the indicator value of the custom indicator; In response to the target evaluation index, auxiliary indicators are also included, and it is determined whether the index value of the auxiliary indicators is within a preset range; If the value of the auxiliary indicator exceeds the preset range, the score of the reasoning service to be evaluated is determined to be 0. If the value of the auxiliary indicator is within the preset range and the selected evaluation type is service recommendation, then there are multiple inference services to be evaluated. For each of the aforementioned reasoning services to be evaluated, the index values ​​of the aforementioned reasoning services are scored according to a preset scoring level to obtain the scores of each target evaluation index. The score of the target evaluation index is weighted and summed with the preset index weight coefficients to obtain the recommendation score of the reasoning service to be evaluated. The reasoning service with the highest recommendation score is selected as the target reasoning service for recommendation.

5. The reasoning service evaluation method according to claim 1 or 4, characterized in that, The evaluation of the reasoning service to be evaluated using the indicator values ​​also includes: If the selected evaluation type is performance testing, then the inference service to be evaluated is one. The target evaluation indicator is scored based on the difference between its value and a preset threshold. The performance score of the inference service to be evaluated is obtained by weighting and summing the scores of the target evaluation indicators with preset indicator weight coefficients.

6. A reasoning service evaluation device, characterized in that, For implementing the method as described in any one of claims 1-5, the apparatus comprises: The collection module is used to collect inference data from each inference service and to build a data pool using the inference data, wherein the inference data is the user's real traffic data; The acquisition module is used to acquire the reasoning service to be evaluated and determine the target evaluation index from the preset evaluation index pool. The target evaluation index includes at least one main index. The testing module is used to obtain test data from the data pool according to the data-driven strategy, and to perform tests on the inference service to be evaluated using the test data to obtain test results; The evaluation module is used to calculate the index value of the target evaluation index based on the test results, and to evaluate the reasoning service to be evaluated using the index value.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images