Service performance evaluation method, computer program product, and electronic device

Abstract

The application discloses a service performance evaluation method, a computer program product and an electronic device, relates to the technical field of performance evaluation, and is characterized in that a data cache pool is used to cache a plurality of experimental test data, the plurality of experimental test data is obtained by processing a plurality of data sets by a plurality of data processing modes, and a performance evaluation experiment is performed based on the data cache pool. Therefore, the technical problem that a plurality of repeated calculations exist in a data processing process of multi-service evaluation and service evaluation efficiency is reduced can be solved, and the technical effects of improving evaluation efficiency and resource utilization and providing strong data support are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of performance evaluation, and in particular to a service performance evaluation method, a computer program product, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the large-scale landing of artificial intelligence technology, pre-training language models have penetrated into many fields such as finance, medicine and technology. The scene requirements are quite different, and new requirements such as inference efficiency and resource consumption are brought about. Therefore, inference service evaluation technology plays a crucial role in pre-training language model inference optimization. At present, there are many repeated calculations in the data processing process of multi-service evaluation, which reduces the service evaluation efficiency. SUMMARY

[0003] The present application provides a service performance evaluation method, a computer program product, an electronic device and a computer readable storage medium to at least solve the problem of repeated calculations in the data processing process of multi-service evaluation in the related art, which reduces the service evaluation efficiency.

[0004] The present application provides a service performance evaluation method, comprising:

[0005] Creating a performance evaluation experiment corresponding to a service to be evaluated, the service to be evaluated including a pre-training language model service;

[0006] Based on a data cache pool, performing a performance evaluation experiment, wherein the data cache pool is used to partition cache a plurality of experimental test data, and the plurality of experimental test data is obtained by processing a plurality of data sets by a plurality of data processing methods;

[0007] In the process of performing the performance evaluation experiment, data collection is performed on the streaming response of the service to be evaluated to dynamically evaluate the service to be evaluated.

[0008] The present application also provides a computer program product, comprising:

[0009] A creating module for creating a performance evaluation experiment corresponding to a service to be evaluated, the service to be evaluated including a pre-training language model service;

[0010] An execution module for performing a performance evaluation experiment based on a data cache pool, wherein the data cache pool is used to partition cache a plurality of experimental test data, and the plurality of experimental test data is obtained by processing a plurality of data sets by a plurality of different data processing methods;

[0011] An evaluation module for collecting data on the streaming response of the service to be evaluated in the process of performing the performance evaluation experiment to dynamically evaluate the service to be evaluated.

[0012] The application further provides an electronic device, comprising a memory for storing a computer program, and a processor for executing the computer program to implement the steps of any of the service performance evaluation methods.

[0013] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the service performance evaluation methods.

[0014] According to the application, the data cache pool is used to cache various experimental test data, the various experimental test data is obtained by processing a plurality of data sets by a plurality of different data processing manners, and the performance evaluation experiment is performed based on the data cache pool. Therefore, the technical problem that there are many repeated calculations in the data processing process of multi-service evaluation and the service evaluation efficiency is reduced can be solved, and the technical effects of improving the evaluation efficiency and resource utilization and providing strong data support are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A flowchart of a service performance evaluation method provided by an embodiment of the application is shown in the figure.

[0017] Figure 2 A flowchart of a service performance evaluation method provided by an embodiment of the application is shown in the figure.

[0018] Figure 3 A flowchart of a service performance evaluation method provided by an embodiment of the application is shown in the figure.

[0019] Figure 4 A flowchart of a service performance evaluation method provided by an embodiment of the application is shown in the figure.

[0020] Figure 5 A flowchart of a service performance evaluation method provided by an embodiment of the application is shown in the figure.

[0021] Figure 6 A flowchart of a service performance evaluation method provided by an embodiment of the application is shown in the figure.

[0022] Figure 7 A flowchart of a service performance evaluation method provided by an embodiment of the application is shown in the figure.

[0023] Figure 8A flowchart of a service performance evaluation method provided by an embodiment of the present application is shown in FIG. 1.

[0024] Figure 9 A flowchart of a service performance evaluation method provided by an embodiment of the present application is shown in FIG. 1.

[0025] Figure 10 A flowchart of a service performance evaluation method provided by an embodiment of the present application is shown in FIG. 1.

[0026] Figure 11 A module diagram of a computer program product provided by an embodiment of the present application is shown in FIG. 1.

[0027] Figure 12 A module diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1.

[0028] Figure 13 A module diagram of a computer readable storage medium provided by an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0030] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0031] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0032] Please refer to Figure 1 and Figure 2 The embodiments of the present application provide a service performance evaluation method, which is described in detail in combination with the execution flow of the service performance evaluation method.

[0033] The service performance evaluation method comprises:

[0034] 010: create a performance evaluation experiment corresponding to the service to be evaluated, the service to be evaluated including a pre-trained language model service;

[0035] 020: based on the data cache pool, performing the performance evaluation experiment, wherein the data cache pool is used to partition cache a plurality of experimental test data, the plurality of experimental test data being obtained by a plurality of data sets respectively processed by a plurality of data processing methods;

[0036] 030: in the process of performing the performance evaluation experiment, data collection is performed on the streaming response of the service to be evaluated to dynamically evaluate the service to be evaluated.

[0037] In the service performance evaluation method of the embodiments of the present application, the data cache pool partitions caches a plurality of experimental test data, the plurality of experimental test data being obtained by a plurality of data sets respectively processed by a plurality of different data processing methods, and the performance evaluation experiment is performed based on the data cache pool. Therefore, the technical problem of repeated calculation in the data processing process of multi-service evaluation can be solved, the evaluation efficiency is improved, the resource utilization rate is improved, and the technical effect of providing strong data support is achieved.

[0038] Specifically, the service to be evaluated is determined first, and the service to be evaluated refers to the pre-trained language model service to be evaluated. The service to be evaluated can include a plurality of sub-services, and the plurality of sub-services can be a plurality of different pre-trained language model services or a plurality of different sub-services in one pre-trained language model service.

[0039] The performance evaluation experiment corresponding to the service to be evaluated is created, different sub-services can create different sub-experiments according to actual application requirements to jointly constitute the performance evaluation experiment. After the creation is completed, the performance evaluation experiment can be started. In the performance evaluation experiment, the processing of the input data is an important link to ensure the effectiveness of the evaluation.

[0040] A data cache pool is set to perform the performance evaluation experiment based on the experimental test data in the data cache pool. For example, the corresponding experimental test data can be selected from the data cache pool according to the data set and the data processing method required for performing the performance evaluation experiment.

[0041] In the data cache pool, a plurality of different experimental test data are partitioned and cached. The data cache pool can include a plurality of cache partitions, and each experimental test data can be cached in a cache partition. The plurality of experimental test data correspond to a plurality of data sets and a plurality of data processing methods, and each experimental test data is obtained by processing a data set by a data processing method.

[0042] The plurality of data sets are, for example, MNIST data sets and CIFAR data sets. The plurality of data processing methods include basic data processing methods and professional data processing methods.

[0043] Basic data processing methods include outlier processing, damage detection, sensitive information desensitization, etc. Outlier processing, such as filtering long text, pure noise audio, damage detection, such as checking file integrity (based on cyclic redundancy check or message digest algorithm), and sensitive information desensitization, such as automatically masking ID cards, bank card numbers, etc. Professional data processing includes word segmentation, stem extraction, subword segmentation, sliding window segmentation, cross-modal alignment, etc.

[0044] During the performance evaluation experiment, the performance of the service to be evaluated can be dynamically evaluated according to the performance evaluation experiment. For example, during the performance evaluation experiment, the service to be evaluated generates a streaming response, and the streaming response of the service to be evaluated can be collected. According to the data collection result, the performance of the service to be evaluated is dynamically evaluated to determine whether the performance of the service to be evaluated meets the standard.

[0045] In the related art, when each service is evaluated, the data processing method is selected to process the data set to obtain test data for evaluation. When repeated experiments are performed, multiple processing is required. In the multi-service evaluation process, the same data set and the same data processing method may be used for different services. Each evaluation requires a large amount of time and resources to calculate and process complex data in real time, which reduces the efficiency of service evaluation and wastes computing resources.

[0046] In the embodiments of the present application, multi-zone caching is added for evaluation acceleration. Multiple data sets are processed by multiple different data processing methods to obtain multiple experimental test data. The multiple experimental test data are partitioned and cached by a data cache pool. When performing a performance evaluation experiment, the required data set and the corresponding experimental test data of the required data processing method can be directly read from the data cache pool. In this way, the evaluation efficiency and resource utilization can be effectively improved. For multi-service evaluation scenarios, strong data support can be provided, and the efficiency of multi-service evaluation can be improved.

[0047] The specific process of creating a performance evaluation experiment is described in detail below.

[0048] Please refer to Figure 2 and Figure 3 In some embodiments, creating a performance evaluation experiment corresponding to the service to be evaluated (i.e., 010) includes:

[0049] 011: Determine the experimental target corresponding to the service to be evaluated, and define the corresponding experimental rules;

[0050] 012: Perform legality verification on the experimental target and the experimental rules;

[0051] 013: In a case where the experiment target and the experiment rule pass the legality check, creating a performance evaluation experiment based on the experiment target and the experiment rule.

[0052] Specifically, the experiment target corresponding to the service to be evaluated is determined, and based on the experiment target, the corresponding experiment rule can be defined through a configuration file.

[0053] In some embodiments, the experiment target includes any one or more of the service to be evaluated, the version relationship, the core indicator, and the experiment type, and the indicator data includes any one or more of the ordinary delay, the success rate, and the dynamic streaming delay; and / or the experiment rule includes any one or more of the version representation, the traffic allocation strategy, the indicator threshold, the statistical significance requirement, the experiment execution period, and the indicator collection frequency.

[0054] The experiment target includes any one or more of the service to be evaluated, the version relationship, the indicator data, and the experiment type. The version relationship refers to the relationship between different versions of the service to be evaluated, which can be divided into a baseline version and a candidate version. The baseline version is the current running version of the service to be evaluated, and the candidate version is a new version of the service to be evaluated. The evaluation of the service to be evaluated includes the evaluation of the improvement effect of the candidate version relative to the baseline version.

[0055] The indicator data is used for subsequent dynamic evaluation and can be determined according to actual evaluation requirements. For example, the indicator data can include any one or more of the ordinary delay, the success rate, and the dynamic streaming delay.

[0056] The experiment type refers to the type of the created performance evaluation experiment and can be determined according to actual evaluation requirements. For example, the experiment type can adopt A / B testing, canary release, etc.

[0057] The experiment rule includes any one or more of the version identification, the traffic allocation strategy, the indicator threshold, the statistical significance requirement, the experiment execution period, and the indicator collection frequency. The version identification refers to the identification of the aforementioned version relationship. The traffic allocation strategy refers to the strategy of allocating traffic to different versions. In one example, the experiment type is A / B testing, 90% of the traffic is allocated to the baseline version, and 10% of the traffic is allocated to the candidate version.

[0058] The indicator threshold refers to the threshold corresponding to the indicator data, which can be set according to actual evaluation requirements. For example, the increase of the delay of no more than 10% can be set as the indicator threshold corresponding to the dynamic streaming delay.

[0059] The statistical significance requirement is used to determine whether the indicator data has statistical significance, ensuring that the observed difference (e.g., the difference between the indicator data of the baseline version and the candidate version) is not caused by random fluctuations. In one example, for the delay, the delay p99 value <0.05 is set as the statistical significance requirement.

[0060] The experiment execution period refers to a period in which a request is initiated to the service to be evaluated in the process of performing the performance evaluation experiment. The index collection frequency refers to a frequency of collecting data to determine the index data in the process of performing the performance evaluation experiment. In an example, the index collection frequency is set to be collected once every 30 seconds.

[0061] In the related art, general indexes are mostly used for evaluation, and the evaluation result is not accurate enough.

[0062] In the embodiments of the present application, the evaluation dimension is refined, and the user attention index is further improved. The index data includes ordinary delay, success rate and dynamic streaming delay, and the evaluation is performed in combination with the statistical significance requirement. The accuracy of the evaluation result is high.

[0063] After determining the experiment target and the experiment rule, the legality of the experiment target and the experiment rule is verified. In the case that the experiment target and the experiment rule do not pass the legality verification, the experiment target corresponding to the service to be evaluated is returned, and the corresponding experiment rule is defined. In the case that the experiment target and the experiment rule pass the legality verification, the performance evaluation experiment can be created based on the experiment target and the experiment rule.

[0064] It should be noted that the service to be evaluated can be deployed in a container orchestration system, such as a kubernetes (K8s) cluster. K8s is open source and is used to manage containerized applications on multiple hosts in a cloud platform. The goal of K8s is to make it simple and efficient to deploy containerized applications. K8s provides a mechanism for application deployment, planning, updating and maintenance.

[0065] When creating the performance evaluation experiment, the metadata corresponding to the performance evaluation experiment is injected into the template of a task (Job) or a timing task (CronJob) in the K8s cluster as an environment variable or volume data. The metadata includes the experiment name, the namespace, the experiment target, the experiment rule, etc. Then, the resources are initialized, and a temporary task for periodically collecting data is created, and the creation of the performance evaluation experiment is completed.

[0066] Job is a controller in K8s used to manage one-time batch tasks, which ensures that one or more basic computing units (Pods) are successfully completed and then terminated. The core goal is to ensure the reliability of task execution, and even if the node fails or the Pod is terminated unexpectedly, the Job will automatically retry or reschedule the task.

[0067] CronJob is a resource object in K8s used to manage periodic tasks, which automatically triggers Job execution based on Cron expression. Its essence is a controller that creates Job at a fixed time interval, which is suitable for tasks that need to be executed repeatedly at fixed time intervals.

[0068] As shown in Figure 2 When performing performance evaluation experiments, traffic can be configured by a traffic configuration tool (for example, Istio, etc.) to proportionally split requests to different versions of the service to be evaluated; and the data source of a monitoring alarm system (for example, Prometheus, etc.) can be associated to collect, store and query the data corresponding to the index data of different versions of the service to be evaluated in real time.

[0069] The specific process of legality verification of the experimental target and the experimental rule is described in detail below.

[0070] Referring to Figure 4 In some embodiments, before creating the performance evaluation experiment corresponding to the service to be evaluated (i.e., 010), the service performance evaluation method further includes:

[0071] 040: adding the access address of the service to be evaluated, and configuring the corresponding model name and access key;

[0072] At this time, legality verification is performed on the experimental target and the experimental rule (i.e., 012), including:

[0073] 0121: initiating a request to the service to be evaluated based on the access address, the model name and the access key;

[0074] 0122: determining whether the experimental target and the experimental rule pass the legality verification according to the request result.

[0075] Specifically, the service to be evaluated is deployed in a K8s cluster and has an externally accessible interface, such as a Hypertext Transfer Protocol (HTTP) path or a google Remote Procedure Call Protocol (gRPC) port. By adding the access address of the service to be evaluated and configuring the corresponding model name and access key, a request can be initiated to the service to be evaluated. The access address can refer to a Uniform Resource Locator (URL), for example, the access address can be https: / / api.example.com / v1 / chat / completions.

[0076] When performing legality verification on the experimental target and the experimental rule, a request is initiated to the service to be evaluated based on the access address, the model name and the access key, and the request result can be used to determine whether the experimental target and the experimental rule pass the legality verification.

[0077] For example, according to the request result, it can be determined whether the to-be-evaluated service exists, whether the data corresponding to the index data can be collected, and whether the experimental rule is correct. If the to-be-evaluated service exists, the data corresponding to the index data can be collected, and the experimental rule is correct, it indicates that the experimental target and the experimental rule pass the legality check. If the to-be-evaluated service does not exist, or the data corresponding to the index data cannot be collected, or the experimental rule is incorrect, it indicates that the experimental target and the experimental rule do not pass the legality check.

[0078] In this way, the reliability of the created performance evaluation experiment is ensured, and the subsequent evaluation result is avoided from being affected by the errors of the experimental target and the experimental rule, and the invalid experiment is avoided.

[0079] Please refer to Figure 2 and Figure 5 In some embodiments, before creating the performance evaluation experiment corresponding to the to-be-evaluated service (i.e., 010), the service performance evaluation method further comprises:

[0080] 050: adding an access address of the to-be-evaluated service, and configuring a corresponding model name and an access token;

[0081] At this time, based on the data cache pool, the performance evaluation experiment is executed (i.e., 020), which comprises:

[0082] 021: initiating a request to the to-be-evaluated service based on the access address, the model name and the access token, and reading the target test data from the data cache pool and sending it to the to-be-evaluated service;

[0083] 022: receiving the streaming response of the to-be-evaluated service;

[0084] At this time, the streaming response of the to-be-evaluated service is collected for data collection to dynamically evaluate the to-be-evaluated service (i.e., 030), which comprises:

[0085] 031: collecting data of the streaming response according to a preset collection frequency to determine index data;

[0086] 032: dynamically evaluating the to-be-evaluated service according to the index data.

[0087] Specifically, the to-be-evaluated service is deployed in a K8s cluster and has an externally accessible interface. By adding an access address of the to-be-evaluated service and configuring a corresponding model name and an access token, a request can be initiated to the to-be-evaluated service.

[0088] After the performance evaluation experiment is started, a request is initiated to the service to be evaluated according to an experiment execution period set in the experiment rule based on an access address, a model name and an access key. When the request is initiated, an Application Programming Interface (API) specification defined by the service to be evaluated needs to be followed to ensure stability of the service to be evaluated and reliability of data.

[0089] When the request is initiated, target test data required for performing the performance evaluation experiment can be read from the data cache pool and sent to the service to be evaluated. The service to be evaluated receives the target test data, processes the target test data, and generates a streaming response.

[0090] The streaming response generated by the service to be evaluated is received, and data collection is performed on the streaming response according to a preset collection frequency set in the experiment rule. Index data can be determined according to the collected data. The service to be evaluated can be evaluated according to the index data. Different versions of the service to be evaluated are requested and data collection is performed respectively, and the index data is associated to the corresponding version after the index data is obtained.

[0091] The specific process of reading the target test data from the data cache pool is described in detail below.

[0092] Referring to Figure 6 In some embodiments, reading the target test data from the data cache pool and sending the target test data to the service to be evaluated (i.e., 021) comprises:

[0093] 0211: determining a target data set corresponding to the target test data and a target data processing mode;

[0094] 0212: determining a cache tag of the target test data according to the target data set and the target data processing mode;

[0095] 0213: reading the target test data from the data cache pool according to the cache tag and sending the target test data to the service to be evaluated.

[0096] Specifically, it can be understood that the target test data is obtained by processing the target data set by the target data processing mode, and the target data set corresponding to the target test data and the target data processing mode are determined. The cache tag of the target test data is determined according to the target data set and the target data processing mode. The target test data can be read from the data cache pool according to the query result according to the cache tag, and the target test data is sent to the service to be evaluated.

[0097] Referring to Figure 7 In some embodiments, the cache tag comprises a first tag and a second tag. Determining the cache tag of the target test data according to the target data set and the target data processing mode (i.e., 0212) comprises:

[0098] determining whether the target data set is included in the plurality of data sets corresponding to the plurality of experimental test data;

[0099] in the case where the target data set is not included, importing the target data set to determine the first label;

[0100] in the case where the target data set is included, taking the label of the target data set as the first label;

[0101] determining whether the target data processing mode is included in the plurality of data processing modes corresponding to the plurality of experimental test data;

[0102] in the case where the target data processing mode is not included, importing the target data processing mode to determine the second label;

[0103] in the case where the target data processing mode is included, taking the label of the target data processing mode as the second label.

[0104] Specifically, for the plurality of data sets and the plurality of data processing modes corresponding to the experimental test data included in the data cache pool, in order to facilitate the user to view, the corresponding data set pool and data processing mode pool are generated, and the data set pool and the data processing mode pool are equivalent to the index directory of the data set and the data processing mode.

[0105] By querying in the data set pool, it can be determined whether the target data set is included in the plurality of data sets corresponding to the plurality of experimental test data. In the case where the target data set is not included, the target data set is imported into the data set pool, so that the label of the imported target data set can be taken as the first label. In the case where the target data set is included, the label of the target data set in the data cache pool can be directly taken as the first label.

[0106] Similarly, by querying in the data processing mode pool, it can be determined whether the target data processing mode is included in the plurality of data processing modes corresponding to the plurality of experimental test data. In the case where the target data processing mode is not included, the target data processing mode is imported into the data processing mode pool, so that the label of the imported target data processing mode can be taken as the second label. In the case where the target data processing mode is included, the label of the target data processing mode in the data cache pool can be directly taken as the second label.

[0107] The cache label includes the first label and the second label. In one example, the cache label is four-dimensional data, as follows:

[0108]

[0109] wherein, is the cache label; a label of the target data set, i.e., a first label; a label of the target data processing mode, i.e., a second label; a timestamp of the last update of the target data set, a timestamp of the last update of the target data processing mode.

[0110] Referring to Figure 7 In some embodiments, the target test data is read from the data cache pool according to the cache label and sent to the service to be evaluated (i.e., 0213), including:

[0111] Based on the cache label, it is determined whether there is experimental test data corresponding to the target test data in the data cache pool;

[0112] In the case where there is no corresponding experimental test data, the target data set is processed according to the target data processing mode to obtain the target test data;

[0113] The target test data is cached to the data cache pool, and the step of determining whether there is experimental test data corresponding to the target test data in the data cache pool based on the cache label is returned;

[0114] In the case where there is corresponding experimental test data, the experimental test data is obtained as the target test data and sent to the service to be evaluated.

[0115] Specifically, based on the determined cache label, it can be queried in the data cache pool to automatically detect whether there is experimental test data corresponding to the target test data in the data cache pool.

[0116] Although in the foregoing embodiments, the target data set is imported when it does not exist in the data set pool, and the target data processing mode is imported when it does not exist in the data processing mode pool, the target data processing mode may not be used to process the target data set, i.e., there may be no corresponding experimental test data in the data cache pool.

[0117] In the case where there is no corresponding experimental test data in the data cache pool, the target data set is first processed according to the target data processing mode to obtain the target test data. Then, the target test data is cached to the data cache pool as the experimental test data, and the step of determining whether there is experimental test data corresponding to the target test data in the data cache pool based on the cache label is returned. Again, the corresponding experimental test data can be queried.

[0118] In the case where there is corresponding experimental test data in the data cache pool, the experimental test data is directly obtained as the target test data and sent to the service to be evaluated.

[0119] It can be understood that the cache tag is a unique tag, and according to the cache tag, the corresponding experimental test data in the data cache pool can be determined. In this way, the accuracy of the read experimental test data is ensured, and the normal execution of the performance evaluation experiment is ensured, and the accuracy of the evaluation result is ensured.

[0120] Please refer to Figure 8 In some embodiments, the dynamic evaluation of the to-be-evaluated service according to the index data (i.e., 032) includes:

[0121] 0321: comparing the data difference between the index data of the baseline version of the to-be-evaluated service and the index data of the candidate version;

[0122] 0322: verifying the statistical significance of the data difference by a statistical hypothesis testing algorithm;

[0123] 0323: respectively checking whether the index data of the baseline version and the index data of the candidate version meet the index threshold corresponding to the index data.

[0124] Specifically, in the process of performing the performance evaluation experiment, the index data of the baseline version of the to-be-evaluated service and the index data of the candidate version of the to-be-evaluated service can be determined. The data difference between the index data of the baseline version and the index data of the candidate version is compared, and the statistical significance of the data difference is verified by a statistical hypothesis testing algorithm, such as a t-test algorithm.

[0125] The core use of the t-test algorithm is to judge whether the difference between the means of two or more groups of data is statistically significant, that is, to judge whether such a difference is truly present or is only caused by random fluctuations (such as the randomness of sample selection).

[0126] For example, according to the statistical significance requirement set in the experimental rule: delay p99 value < 0.05, if the delay p99 value calculated based on the t-test algorithm is < 0.05, it means that the statistical significance of the data difference between the index data of the baseline version and the index data of the candidate version meets the standard. If the delay p99 value calculated based on the t-test algorithm is > 0.05, it means that the statistical significance of the data difference between the index data of the baseline version and the index data of the candidate version does not meet the standard.

[0127] In addition, it can also be checked whether the index data of the baseline version and the index data of the candidate version meet the index threshold corresponding to the index data, respectively. For example, taking the dynamic streaming delay as an example, it is checked whether the increase of the dynamic streaming delay of the baseline version exceeds the corresponding increase threshold. If the increase of the dynamic streaming delay of the baseline version exceeds the corresponding increase threshold, it means that the dynamic streaming delay of the baseline version is abnormal; if the increase of the dynamic streaming delay of the baseline version does not exceed the corresponding increase threshold, it means that the dynamic streaming delay of the baseline version is normal.

[0128] In the embodiments of the present application, the index data is combined with statistical significance for dynamic evaluation, to evaluate whether the data difference between the index data of the baseline version and the index data of the candidate version is truly existent, to provide a scientific and objective basis for service evaluation, and the accuracy of the evaluation result is higher.

[0129] Please refer to Figure 9 In some embodiments, based on the data cache pool, the performance evaluation experiment (i.e., 020) is performed, including:

[0130] In the data cache pool, in the case that the data set and / or data processing mode corresponding to the experimental test data is updated, a cache partition is established in the data cache pool, and the updated test data corresponding to the updated data set and / or data processing mode is cached as the experimental test data.

[0131] It is judged whether the performance evaluation experiment includes a sub-experiment corresponding to the data set and / or data processing mode.

[0132] In the case that the sub-experiment corresponding to the data set and / or data processing mode is included, the sub-experiment is waited to complete, and the experimental test data corresponding to the data set and / or data processing mode is cleared.

[0133] In the case that the sub-experiment corresponding to the data set and / or data processing mode is not included, the experimental test data corresponding to the data set and / or data processing mode is directly cleared.

[0134] Specifically, in the process of performing the performance evaluation experiment, there may be a case that the data set and / or data processing mode corresponding to the experimental test data in the data cache pool is updated.

[0135] In the case that the data set corresponding to the experimental test data is updated, the data processing mode related to the data set before the update is obtained, the related data processing mode is used to perform data processing on the updated data set to obtain updated test data. A new cache partition is established in the data cache pool, and the updated test data corresponding to the updated data set is cached as the experimental test data.

[0136] If the service to be evaluated includes multiple sub-services, the performance evaluation experiment includes multiple corresponding sub-experiments. It is judged whether the performance evaluation experiment includes a sub-experiment corresponding to the data set before the update. In the case that the performance evaluation experiment includes a sub-experiment corresponding to the data set before the update, a correlation list of the corresponding sub-experiment can be established, and it is dynamically monitored whether all the sub-experiments in the correlation list are completed. The sub-experiment is waited to complete, the sub-experiment in the correlation list is cleared, and the experimental test data corresponding to the data set before the update is cleared.

[0137] In the case that the performance evaluation experiment does not include the sub-experiment corresponding to the data set before the update, the experimental test data corresponding to the data set before the update can be directly cleared.

[0138] In the case that the data processing mode corresponding to the experimental test data is updated, the data set related to the data processing mode before the update is obtained, and the updated data processing mode processes the related data set to obtain updated test data. A new cache partition is established in the data cache pool, and the updated test data corresponding to the updated data processing mode is cached as experimental test data.

[0139] It is judged whether the performance evaluation experiment includes a sub-experiment corresponding to the data processing mode before the update. In the case that the performance evaluation experiment includes a sub-experiment corresponding to the data processing mode before the update, a correlation list corresponding to the sub-experiment can be established, and it is dynamically monitored whether all the sub-experiments in the correlation list are completed. Wait for the sub-experiment to be completed, clear the sub-experiment in the correlation list, and clear the experimental test data corresponding to the data processing mode before the update.

[0140] In the case that the performance evaluation experiment does not include the sub-experiment corresponding to the data processing mode, the experimental test data corresponding to the data processing mode can be directly cleared.

[0141] In this way, the smooth upgrade of the data cache pool is realized when the updated test data is introduced, which can provide strong data support for the evaluation of multiple inference services. In addition, it does not affect the normal execution of the performance evaluation experiment, and further improves the efficiency of the multi-service evaluation.

[0142] In addition, in the case that the data set corresponding to the experimental test data is updated, the timestamp of the data set will change, so that when the performance evaluation experiment is executed subsequently, the latest experimental test data can be queried in the data cache pool according to the cache tag.

[0143] Similarly, in the case that the data processing mode corresponding to the experimental test data is updated, the timestamp of the data processing mode will change, so that when the performance evaluation experiment is executed subsequently, the latest experimental test data can be queried in the data cache pool according to the cache tag.

[0144] Please refer to Figure 10 In some embodiments, the index data includes a dynamic streaming delay, and the index data (i.e., 031) is determined according to data collection of the streaming response at a preset collection frequency, including:

[0145] 0311: Real-time acquisition of generation timestamps of multiple markers in the streaming response;

[0146] 0312: Calculate the arrival delay between adjacent markers according to the generation timestamps;

[0147] 0313: determining the weight corresponding to the plurality of tokens according to the generation order of the plurality of tokens;

[0148] 0314: calculating the dynamic streaming delay according to the arrival delay and the weight.

[0149] Specifically, the streaming response of the service to be evaluated includes a plurality of tokens, which can refer to tokens. The following is an example of expanding the token as a token. When the token is generated, the generation timestamp of each token is captured in real time and recorded. Let the number of tokens be n, and the token sequence be , wherein denotes the arrival time of the i-th token (i.e., the generation timestamp, in seconds) from the start of the request.

[0150] The arrival delay of the i-th token is defined as the difference between the arrival time of the previous token, i.e.:

[0151]

[0152] , wherein denotes the delay of the i-th token, i.e., the arrival delay between the i-th token and the (i-1)-th token, denotes the time from the start of the request.

[0153] To control the steepness of the weight increment through the parameter and take into account the influence of the early delay (i is smaller), the weight formula of each token can be as follows:

[0154]

[0155] , wherein is the weight of the i-th token delay . 0 < k < 1, by default 0.1, to adjust the size of the exponential, the larger k (e.g., 0.5), the faster the late weight (i is larger) grows, the relatively greater influence on the result, c (c > 0) is a constant term, adding a constant term c in the numerator of the weight function to increase the weight of the early delay. By adding a constant term c, the weight of the first delay is increased from 0 to a non-zero value, and the larger c, the closer the weight of the early delay to the late, and the more balanced the influence of each delay.

[0156] Combining the definition of the delay, the overall delay, i.e., the dynamic streaming delay, is:

[0157]

[0158] The final formula of the dynamic streaming delay is:

[0159]

[0160] In the related art, the total delay of the entire response completion is calculated, and the different sensitivities of the user to the intermediate token and the end token in the streaming generation process are ignored.

[0161] In the embodiments of the present application, a dynamic streaming evaluation index is proposed to solve the performance evaluation problem of streaming output (such as token-by-token generation) in pre-trained language model services. The evaluation weight can be dynamically adjusted according to the position of token generation, the sensitivity of tokens at different positions is distinguished, the contribution of tail token delay to the overall delay index is greater, and the user experience is more accurately reflected, thereby providing a new and more scientific standard for performance optimization and evaluation of pre-trained language model services.

[0162] Please refer to Figure 2 In some embodiments, after performing the performance evaluation experiment based on the data cache pool (i.e., 020), the service performance evaluation method further comprises:

[0163] determining whether the performance evaluation experiment meets an experiment termination condition according to the index data;

[0164] terminating the performance evaluation experiment and outputting the experiment result in the case of meeting the experiment termination condition;

[0165] returning to the step of performing the performance evaluation experiment based on the data cache pool in the case of not meeting the experiment termination condition.

[0166] Specifically, in the process of performing the performance evaluation experiment, after determining the index data, it can be determined whether the performance evaluation experiment meets the experiment termination condition according to the index data. The experiment termination condition includes a first termination condition and a second termination condition, the first termination condition corresponds to the termination under the normal execution of the performance evaluation experiment, and the second termination condition corresponds to the termination under the abnormal situation of the performance evaluation experiment.

[0167] The first termination condition includes that the performance evaluation experiment is executed to a first preset time, the performance evaluation experiment is executed a preset number of times, the index data meets a corresponding index threshold value for a second preset time, the statistical significance of the candidate version reaches a statistical significance requirement, etc.

[0168] The second termination condition includes that the index data of the candidate version does not meet the corresponding index threshold value, the index data of the baseline version is abnormal (for example, the delay suddenly increases or crashes), the user manually terminates, etc.

[0169] Of course, the first termination condition and the second termination condition can be set and adjusted according to the actual evaluation situation.

[0170] In the case where the performance evaluation experiment meets the experiment termination condition, the performance evaluation experiment is terminated, and the experiment result is output. In the case where the performance evaluation experiment does not meet the experiment termination condition, the data cache pool is returned, and the steps of performing the performance evaluation experiment are executed to continue the performance evaluation experiment until the experiment termination condition is met.

[0171] In the embodiments of the present application, multiple experiment termination conditions are set, and the experiment is automatically terminated when the performance of the service to be evaluated meets the standard; in the case where the difference between the candidate version and the baseline version is not significant, the experiment can be prevented from being executed indefinitely; and in the case where the experiment is abnormally executed, the experiment is terminated in a timely manner. In this way, resource waste can be avoided, and evaluation efficiency can be improved.

[0172] In some embodiments, the experiment result is output, including:

[0173] In the case where a persistent volume is configured, a report file corresponding to the experiment result is generated and saved to the persistent volume;

[0174] In the case where cloud storage is configured, a report file corresponding to the experiment result is generated and uploaded to the cloud storage.

[0175] In the case where mail information is configured, a report file corresponding to the experiment result is generated and sent to the corresponding mailbox.

[0176] Specifically, when the performance evaluation experiment is terminated, the experiment state of the performance evaluation experiment is detected, and the experiment state includes success and failure. For example, if the performance evaluation experiment is terminated by triggering the early termination condition, the experiment state is failure. Regardless of whether the performance evaluation experiment is successful or not, an experiment report is immediately generated, which summarizes the comparison results of the index data, the statistical significance conclusion, and the like.

[0177] The experiment result can be output in multiple ways. In the case where a persistent volume (Persistent Volume Claim, PVC) is configured, a report file corresponding to the experiment result is generated and saved to the persistent volume for subsequent access.

[0178] In the case where cloud storage is configured, a report file corresponding to the experiment result is generated and uploaded to the cloud storage. The cloud storage platform is, for example, Amazon Simple Storage Service (AWS S3).

[0179] In the case where mail information is configured, a report file corresponding to the experiment result is generated and sent to the corresponding mailbox through a network callback (Webhook).

[0180] Taking cloud storage as an example, an example of a report file corresponding to the experiment result is as follows:

[0181] report:

[0182] format: "html"

[0183] destination:

[0184] type: "s3"

[0185] bucket: "reports-bucket"

[0186] path: "reports / {{.Experiment.Name}} / {{.Timestamp}}.html"

[0187] credentials:

[0188] secretName: "s3-creds"

[0189] trigger: "on_termination"

[0190] Among them, the parameter format is used to configure the report file format, supporting JavaScript Object Notation (JSON) format or HyperText Markup Language (html).

[0191] The parameter destination.type is used to configure the cloud storage type, which matches the target cloud storage platform, for example, "s3" matches AWS S3.

[0192] The parameter destination.bucket is used to configure the target storage bucket, which needs to be created in advance and ensured to be accessible. The storage bucket is a basic container in cloud storage services for organizing and managing data, similar to a "folder" in a local file system, but with more functions and features adapted to cloud scenarios. The storage bucket can hold a large number of files, supports flexible permission management, and can seamlessly integrate with various cloud services, making it a core unit for organizing data in cloud storage.

[0193] The parameter destination.path is used to configure the path of the report file in the storage bucket, supporting Go template syntax and dynamic variables such as experiment name and timestamp. {{.Timestamp}} represents the generated timestamp, which can avoid file name conflicts.

[0194] The parameter destination.credentials.secretName is used to reference the K8sSecret name of the storage authentication information.

[0195] The parameter trigger is used to configure the upload trigger occasion, which can be configured as on_termination, indicating automatic upload when the experiment terminates, or periodic, indicating periodic upload, which needs to be combined with the interval field.

[0196] After the report file is sent, the related resources of the performance evaluation experiment can be cleaned up, such as the temporary test Pod, the traffic distribution policy, and the like, to avoid residual resource occupation of the system.

[0197] Through the description of the above implementation, those skilled in the art can clearly understand that the method according to the above embodiment can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better implementation.

[0198] Please refer to Figure 11 The embodiment of the present application also provides a computer program product 100. The computer program product 100 comprises a creating module 10, an executing module 20 and an evaluating module 30. The creating module 10 is used to create a performance evaluation experiment corresponding to a service to be evaluated, the service to be evaluated comprising a pre-trained language model service. The executing module 20 is used to execute the performance evaluation experiment based on a data cache pool, wherein the data cache pool is used to cache a plurality of experimental test data in a partitioned manner, and the plurality of experimental test data is obtained by processing a plurality of data sets by a plurality of different data processing manners. The evaluating module 30 is used to collect data of a streaming response of the service to be evaluated in the process of executing the performance evaluation experiment, so as to dynamically evaluate the service to be evaluated.

[0199] In some embodiments, the creating module 10 is specifically used to determine an experimental target corresponding to the service to be evaluated, and define corresponding experimental rules; perform legality verification on the experimental target and the experimental rules; and create the performance evaluation experiment based on the experimental target and the experimental rules in a case where the experimental target and the experimental rules pass the legality verification.

[0200] In some embodiments, the computer program product 100 further comprises a configuring module. Before creating the performance evaluation experiment corresponding to the service to be evaluated, the configuring module is used to add an access address of the service to be evaluated, and configure a corresponding model name and an access key. The creating module 10 is specifically used to initiate a request to the service to be evaluated based on the access address, the model name and the access key; and determine whether the experimental target and the experimental rules pass the legality verification according to a request result.

[0201] In some embodiments, before creating a performance evaluation experiment corresponding to the service to be evaluated, the configuration module is configured to add an access address of the service to be evaluated, and configure a corresponding model name and an access mark. The execution module 20 is specifically configured to initiate a request to the service to be evaluated based on the access address, the model name and the access mark, and read target test data from the data cache pool and send it to the service to be evaluated; and receive a streaming response of the service to be evaluated. The evaluation module 30 is specifically configured to collect data according to a preset collection frequency, determine index data, and dynamically evaluate the service to be evaluated according to the index data.

[0202] In some embodiments, the execution module 20 is specifically configured to determine a target data set and a target data processing mode corresponding to the target test data; determine a cache tag of the target test data according to the target data set and the target data processing mode; and read the target test data from the data cache pool according to the cache tag and send it to the service to be evaluated.

[0203] In some embodiments, the cache tag includes a first tag and a second tag. The execution module 20 is specifically configured to determine whether the target data set is included in a plurality of data sets corresponding to a plurality of experimental test data; in the case where the target data set is not included, import the target data set to determine the first tag; in the case where the target data set is included, take the tag of the target data set as the first tag; determine whether the target data processing mode is included in a plurality of data processing modes corresponding to a plurality of experimental test data; in the case where the target data processing mode is not included, import the target data processing mode to determine the second tag; in the case where the target data processing mode is included, take the tag of the target data processing mode as the second tag.

[0204] In some embodiments, the execution module 20 is specifically configured to determine, based on the cache tag, whether there is experimental test data corresponding to the target test data in the data cache pool; in the case where there is no corresponding experimental test data, perform data processing on the target data set according to the target data processing mode to obtain the target test data; cache the target test data to the data cache pool, and return to the step of determining, based on the cache tag, whether there is experimental test data corresponding to the target test data in the data cache pool; in the case where there is corresponding experimental test data, obtain the experimental test data as the target test data and send it to the service to be evaluated.

[0205] In some embodiments, the evaluation module 30 is specifically configured to compare the data difference between the index data of the baseline version and the index data of the candidate version of the service to be evaluated; verify the statistical significance of the data difference through a statistical hypothesis testing algorithm; and respectively check whether the index data of the baseline version and the index data of the candidate version meet the index threshold corresponding to the index data.

[0206] In some embodiments, the execution module 20 is further configured to, in the case that the data set corresponding to the experimental test data and / or the data processing mode is updated in the data cache pool, establish a cache partition in the data cache pool, cache the updated test data corresponding to the updated data set and / or data processing mode as the experimental test data, determine whether the performance evaluation experiment includes a sub-experiment corresponding to the data set and / or data processing mode, in the case that the corresponding sub-experiment is included, wait for the completion of the sub-experiment, and clear the experimental test data corresponding to the data set and / or data processing mode, and in the case that the corresponding sub-experiment is not included, directly clear the experimental test data corresponding to the data set and / or data processing mode.

[0207] In some embodiments, the index data includes a dynamic streaming delay. The evaluation module 30 is specifically configured to acquire generation timestamps of a plurality of marks in the streaming response in real time, calculate an arrival delay between adjacent marks according to the generation timestamps, determine weights corresponding to the plurality of marks according to a generation order of the plurality of marks, and calculate the dynamic streaming delay according to the arrival delay and the weights.

[0208] In some embodiments, the computer program product 100 further includes a termination module, which is specifically configured to, after the performance evaluation experiment is executed based on the data cache pool, determine whether the performance evaluation experiment satisfies an experiment termination condition according to the index data, in the case that the experiment termination condition is satisfied, terminate the performance evaluation experiment and output an experimental result, and in the case that the experiment termination condition is not satisfied, return to the step of executing the performance evaluation experiment based on the data cache pool.

[0209] In some embodiments, the termination module is specifically configured to, in the case that a persistent volume is configured, generate a report file corresponding to the experimental result and save the report file into the persistent volume, in the case that a cloud storage is configured, generate a report file corresponding to the experimental result and upload the report file to the cloud storage, and in the case that mail information is configured, generate a report file corresponding to the experimental result and send the report file to a corresponding mailbox.

[0210] The descriptions of the features in the embodiments of the computer program product 100 can refer to the descriptions of the embodiments of the service performance evaluation method, which will not be repeated here.

[0211] Please refer to Figure 12 The embodiments of the present application also provide an electronic device 200, which includes a memory 210 and a processor 220, the memory 210 stores a computer program, and the processor 220 is configured to run the computer program to execute the steps in any one of the embodiments of the service performance evaluation method.

[0212] Please refer to Figure 13The embodiments of the present application further provide a computer readable storage medium 300, which stores a computer program 310. The computer program 310 is configured to perform the steps in any of the service performance evaluation method embodiments when running.

[0213] In an example embodiment, the computer readable storage medium 300 can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing the computer program 310.

[0214] The embodiments of the present application further provide another computer program product, which comprises a non-volatile computer readable storage medium. The non-volatile computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps in any of the service performance evaluation method embodiments.

[0215] The skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0216] The above provides a detailed description of the service performance evaluation method, computer program product, electronic device and computer readable storage medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the examples is only applicable to help understand the method of the present application and its core idea. It should be noted that for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application. These improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method of service performance evaluation, characterized by, The method comprises the following steps: creating a performance evaluation experiment corresponding to a service to be evaluated, the service to be evaluated comprising a pre-trained language model service; performing the performance evaluation experiment based on a data cache pool, wherein the data cache pool is used to cache a plurality of experimental test data in partitions, and the plurality of experimental test data is obtained by processing a plurality of data sets by a plurality of data processing methods; in the process of performing the performance evaluation experiment, data collection is performed on the streaming response of the service to be evaluated to dynamically evaluate the service to be evaluated; the data collection on the streaming response of the service to be evaluated to dynamically evaluate the service to be evaluated comprises: collecting data on the streaming response according to a preset collection frequency to determine index data; the index data comprises a dynamic streaming delay, and the collection of data on the streaming response according to the preset collection frequency to determine the index data comprises: real-time acquisition of generation timestamps of a plurality of marks in the streaming response; calculation of arrival delays between adjacent marks according to the generation timestamps; determination of weights corresponding to the plurality of marks according to the generation order of the plurality of marks; calculation of the dynamic streaming delay according to the arrival delays and the weights; wherein the calculation formula of the weights is: wherein, is the weight corresponding to the i-th marker among the plurality of markers, 0 < k < 1, c > 0.

2. The service performance evaluation method according to claim 1, characterized by, the creation of the performance evaluation experiment corresponding to the service to be evaluated comprises: determination of an experimental target corresponding to the service to be evaluated, and definition of corresponding experimental rules; legitimacy verification of the experimental target and the experimental rules; in the case that the experimental target and the experimental rules pass the legitimacy verification, creation of the performance evaluation experiment based on the experimental target and the experimental rules.

3. The service performance evaluation method according to claim 2, characterized by, Before the creation of the performance evaluation experiment corresponding to the service to be evaluated, the service performance evaluation method further comprises: adding an access address of the service to be evaluated, and configuring a corresponding model name and an access key; the legitimacy verification of the experimental target and the experimental rules comprises: initiating a request to the service to be evaluated based on the access address, the model name and the access key; determination of whether the experimental target and the experimental rules pass the legitimacy verification according to the request result.

4. The service performance evaluation method according to claim 1, characterized by, Before the creation of the performance evaluation experiment corresponding to the service to be evaluated, the service performance evaluation method further comprises: adding an access address of the service to be evaluated, and configuring a corresponding model name and an access key; the performance evaluation experiment based on the data cache pool comprises: initiating a request to the service to be evaluated based on the access address, the model name and the access key, and reading target test data from the data cache pool and sending the target test data to the service to be evaluated; receiving the streaming response of the service to be evaluated; after the collection of data on the streaming response according to the preset collection frequency to determine the index data, the data collection on the streaming response of the service to be evaluated to dynamically evaluate the service to be evaluated further comprises: dynamically evaluating the service to be evaluated according to the index data.

5. The service performance evaluation method according to claim 4, characterized by, the reading of the target test data from the data cache pool and the sending of the target test data to the service to be evaluated comprises: determining a target data set and a target data processing mode corresponding to the target test data; determining a cache tag of the target test data according to the target data set and the target data processing mode; reading the target test data from the data cache pool according to the cache tag and sending the target test data to the service to be evaluated.

6. The service performance evaluation method according to claim 5, characterized by, The cache tag includes a first tag and a second tag, and the determination of the cache tag of the target test data according to the target data set and the target data processing mode includes: determining whether the target data set is included in a plurality of data sets corresponding to a plurality of experimental test data; in the case that the target data set is not included, importing the target data set to determine the first tag; in the case that the target data set is included, taking the tag of the target data set as the first tag; determining whether the target data processing mode is included in a plurality of data processing modes corresponding to a plurality of experimental test data; in the case that the target data processing mode is not included, importing the target data processing mode to determine the second tag; in the case that the target data processing mode is included, taking the tag of the target data processing mode as the second tag.

7. The service performance evaluation method according to claim 5, characterized by, The reading of the target test data from the data cache pool according to the cache tag and the sending of the target test data to the service to be evaluated include: determining whether the experimental test data corresponding to the target test data exists in the data cache pool based on the cache tag; in the case that the corresponding experimental test data does not exist, performing data processing on the target data set according to the target data processing mode to obtain the target test data; caching the target test data to the data cache pool and returning to the step of determining whether the experimental test data corresponding to the target test data exists in the data cache pool based on the cache tag; in the case that the corresponding experimental test data exists, obtaining the experimental test data as the target test data and sending the target test data to the service to be evaluated.

8. The service performance evaluation method according to claim 4, characterized by, The dynamic evaluation of the service to be evaluated according to the index data includes: comparing the data difference between the index data of a baseline version of the service to be evaluated and the index data of a candidate version; verifying the statistical significance of the data difference by a statistical hypothesis testing algorithm; respectively checking whether the index data of the baseline version and the index data of the candidate version satisfy the index threshold corresponding to the index data.

9. The method for service performance evaluation according to claim 1, wherein, The performance evaluation experiment based on the data cache pool includes: in the case that the data set and / or the data processing mode corresponding to the experimental test data in the data cache pool is updated, establishing a cache partition in the data cache pool, caching the updated test data corresponding to the updated data set and / or the updated data processing mode as the experimental test data; determining whether a sub-experiment corresponding to the data set and / or the data processing mode is included in the performance evaluation experiment; In the case of including a corresponding sub-experiment, wait for the completion of the sub-experiment, clear the experimental test data corresponding to the data set and / or the data processing mode; In the case of not including the corresponding sub-experiment, directly clear the experimental test data corresponding to the data set and / or the data processing mode.

10. The method for evaluating service performance according to claim 4, wherein, After executing the performance evaluation experiment based on the data cache pool, the service performance evaluation method further comprises: determining whether the performance evaluation experiment meets an experiment termination condition according to the index data; terminating the performance evaluation experiment and outputting an experiment result in the case of meeting the experiment termination condition; returning to the step of executing the performance evaluation experiment based on the data cache pool in the case of not meeting the experiment termination condition.

11. The service performance evaluation method according to claim 10, characterized by, The output experiment result comprises: generating a report file corresponding to the experiment result and saving it into the persistent volume in the case of configuring a persistent volume; generating a report file corresponding to the experiment result and uploading it to cloud storage in the case of configuring cloud storage; generating a report file corresponding to the experiment result and sending it to a corresponding mailbox in the case of configuring email information.

12. A computer program product, characterised in that, Comprise: a creating module for creating a performance evaluation experiment corresponding to a service to be evaluated, the service to be evaluated comprising a pre-trained language model service; an executing module for executing the performance evaluation experiment based on a data cache pool, wherein the data cache pool is used to partition and cache a plurality of experimental test data, and the plurality of experimental test data is obtained by a plurality of data sets respectively processed by a plurality of different data processing modes; an evaluation module for collecting data of a streaming response of the service to be evaluated in the process of executing the performance evaluation experiment, so as to dynamically evaluate the service to be evaluated; The evaluation module is specifically used for: collecting data of the streaming response according to a preset collection frequency to determine index data; The evaluation module is specifically used for: real-time acquisition of generation time stamps of a plurality of marks in the streaming response; calculation of arrival delays between adjacent marks according to the generation time stamps; determination of weights corresponding to a plurality of marks according to generation orders of the plurality of marks; calculation of a dynamic streaming delay according to the arrival delays and the weights; wherein the calculation formula of the weights is: wherein, is the weight corresponding to the i-th marker among the plurality of markers, 0 < k < 1, c > 0.

13. An electronic device, comprising: Comprise: a memory for storing a computer program; a processor for executing the computer program to realize the steps of the service performance evaluation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to realize the steps of the service performance evaluation method according to any one of claims 1 to 11.

Images