Benchmark test method and device, electronic equipment and program product

Abstract

The invention relates to the technical field of benchmark testing, and discloses a benchmark testing method, device and system and a program product. In the method, in the Nth (N is an integer greater than 1) round of benchmark test process, a first characteristic threshold value of a first load operation characteristic can be input into a first benchmark test model to obtain a first load. Then, whether the change of the first load relative to a second load meets a test ending condition or not is judged, and the second load is obtained by inputting the first characteristic threshold value into a second benchmark test model in the (N-1) th round of benchmark test process. If yes, a benchmark test result obtained based on the first load is obtained, if not, the first benchmark test model is adjusted, a third benchmark test model is obtained, and then the process is repeated till the obtained load meets the test ending condition. The method can improve the accuracy of the benchmark test result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a benchmark testing method, apparatus, electronic device, and program product. Background Technology

[0002] Benchmarking is a way to measure and evaluate the performance level of an object under test (e.g., a single server or a cluster of servers) before it is deployed to a real-world application environment to ensure that the object under test has the ability to handle the expected load. Benchmarking is typically performed using benchmarking tools. During benchmarking, a series of loads (website backend programs, artificial intelligence models, databases, etc., loaded onto the object under test during the benchmarking process) can be run on the object under test. The benchmarking tool can then evaluate the performance level of the object under test in the relevant application scenario based on its performance under these loads.

[0003] Benchmarking typically involves multiple rounds of testing. For example, in each round, the load on the tested object is gradually increased compared to the previous round until a preset load limit is reached. The performance level of the tested object under this preset load limit is then used as the benchmark result. For instance, taking a cluster as the tested object and a website backend program used for information retrieval as the load, in each round of benchmarking, the number of backend programs running on the cluster can be gradually increased compared to the previous round, such as increasing the number of backend programs, until the cluster reaches a preset load limit. The performance level of the cluster under this preset load limit (such as the cluster's response time to query requests) is then used as the benchmark result.

[0004] It is understandable that the closer the preset load limit is to the actual load capacity of the tested object, the more accurate the benchmark test results will be. However, in some embodiments, the preset load limit is set based on experience, and the difference between this preset load limit and the actual load capacity of the tested object may be significant. Therefore, if the benchmark test results are obtained based on a preset load limit set based on experience, the accuracy of the benchmark test results may be low. Summary of the Invention

[0005] This application provides a benchmarking method, apparatus, electronic device, and program product. This method can determine the load that needs to be applied to the object under test based on a benchmarking model, thereby improving the accuracy of benchmarking results.

[0006] In a first aspect, this application provides a benchmarking method, which includes: obtaining a first feature threshold of a first load operating characteristic when the load is running on the test object; and inputting the first feature threshold into a benchmarking model to obtain the load of the test object corresponding to the first load operating characteristic.

[0007] It is understood that in this application, the first load operation characteristic is the characteristic category corresponding to the state that the user expects the tested object to reach when benchmarking the tested object. Therefore, the first feature threshold corresponding to the first load operation characteristic is used to represent the feature value corresponding to the state that the user expects the tested object to reach, and this first feature threshold can be preset by the user based on experience, etc. Taking the load as a website backend program used for querying, and the user expects the average response time of the tested object to the query request to be 0.5s during benchmarking, then the first load operation characteristic is the average response time of the query request, and the first feature threshold is 0.5s.

[0008] Then, by inputting this first feature threshold into the benchmark model, the load corresponding to the state that the user expects the tested object to reach can be obtained. Compared with the load manually set based on user experience and the linear accumulation method of load, the load in this application is obtained based on the benchmark model, which makes the load more accurate.

[0009] In one possible implementation of the first aspect, the method further includes: obtaining a first feature threshold of the first load operating characteristic when the load is running on the test object; during the Nth round of benchmark testing, inputting the first feature threshold into a first benchmark model to obtain a first load of the test object corresponding to the first load operating characteristic, where N is an integer greater than 1; corresponding to the change of the first load relative to the second load satisfying the test termination condition, obtaining a benchmark result based on the first load, wherein the second load is obtained based on inputting the first feature threshold into a second benchmark model during the N-1th round of benchmark testing, and the first benchmark model is obtained based on adjusting the second benchmark model.

[0010] It is understood that the first load operating characteristics include at least one of load service characteristics and device resource characteristics, and the first characteristic threshold includes at least one of load service characteristic threshold and device resource characteristic threshold. Therefore, compared with the load directly set by the user based on experience, the method provided in this application obtains the first load based on the first characteristic threshold of the first load operating characteristics, resulting in higher accuracy of the first load.

[0011] Furthermore, this application predicts the first load corresponding to the Nth round of benchmark testing based on a first benchmark testing model, and terminates the benchmark testing when the difference between the first load and the second load meets the test termination condition, and obtains the benchmark test result based on the first load. It is understood that since the load is typically increased round by round during multi-round benchmark testing, the test termination condition in this application indicates that the load corresponding to the current round of benchmark testing (e.g., the first load) has no room for further increase compared to the load corresponding to the previous round of benchmark testing (e.g., the second load), provided that the user expects the tested object to reach the characteristic threshold (e.g., the first characteristic threshold) corresponding to the load operation characteristics (e.g., the first load) when running the load (e.g., the first load). At this point, the benchmark test result based on the load corresponding to the current round of benchmark testing (e.g., the first load) can be obtained. Taking the first load as an example, the first load corresponding to the benchmark test result obtained in this way meets the test termination condition. Therefore, compared to the load set based on experience, the first load of this application is closer to the actual load capacity of the tested object under the first characteristic threshold, thus making the benchmark test result more accurate.

[0012] In one possible implementation of the first aspect, the test termination condition may include: the absolute value of the difference between the first load amount corresponding to the first load and the second load amount corresponding to the second load is less than or equal to a difference threshold. In this approach, the benchmark test ends when the change in the first load amount relative to the second load amount is small, and the benchmark test result corresponding to the first load is obtained.

[0013] Since the first load meets the test termination condition, it is closer to the actual load capacity of the test object, resulting in higher accuracy of the benchmark test results. Furthermore, determining whether the first load meets the test termination condition based on the change in the first load relative to the second load makes the test termination condition more intuitive and concise, which helps improve the speed of benchmark testing and thus enhances testing efficiency.

[0014] In one possible implementation of the first aspect, the first benchmark model is obtained by: acquiring a first feature value of the first load running characteristics during the second load operation; adjusting the second benchmark model based on the difference between the first feature value and the first feature threshold to obtain the first benchmark model.

[0015] It is understandable that the above-described method of adjusting the second benchmark model to obtain the first benchmark model ensures that the first benchmark model has higher accuracy than the second benchmark model. In this case, inputting the first feature threshold into the first benchmark model results in a higher accuracy of the first load, thereby improving the accuracy of the benchmark results.

[0016] In one possible implementation of the first aspect, when the change of the first load relative to the second load does not meet the test termination condition, the first benchmark test model is adjusted to obtain the third benchmark test model.

[0017] The process of adjusting the first benchmark model to obtain the third benchmark model includes: adjusting the first benchmark model based on the difference between the second feature value and the first feature threshold corresponding to the first load running characteristics during the first load operation to obtain the third benchmark model.

[0018] It is understandable that if the change of the first load relative to the second load does not meet the test termination condition, the first benchmark model needs to be trained (or adjusted) to obtain the third benchmark model. The accuracy of the third benchmark model is higher than that of the first benchmark model. Therefore, the accuracy of the third load obtained based on the third benchmark model is also higher. That is, the feature value of the first running feature during the third load operation is closer to the first feature threshold, thus making the benchmark results more accurate.

[0019] In one possible implementation of the first aspect, corresponding to the change of the first load relative to the second load satisfying the test termination condition, obtaining the benchmark test result based on the first load includes: obtaining a third feature value of the second load running feature generated when the first load runs on the test object, and using the third feature value as the benchmark test result, wherein the first load running feature and the second load running feature are the same or different.

[0020] As can be understood, based on the foregoing, the first load operation characteristic is the feature category corresponding to the state that the user expects the test object to reach when benchmarking the test object, and the first feature threshold corresponding to the first load operation characteristic is used to determine the load (such as the first load) applied to the test object. The first load, when running on the test object, corresponds to a third feature value of the second load operation characteristic, and the third feature value is: the actual feature value that the first load operation characteristic and / or other load operation characteristics can actually reach when the user expects the test object to reach the first feature threshold of the first load operation characteristic. Therefore, the second load operation characteristic can be the same as the first load operation characteristic, or it can be different from the first load operation characteristic (for example, the second load operation characteristic is another load operation characteristic different from the first load operation characteristic, or the second load operation characteristic includes both the first load operation characteristic and other load operation characteristics different from the first load operation characteristic).

[0021] The second load operation characteristic may include at least one of load service characteristics and equipment resource characteristics, or may include other characteristics other than load service characteristics and equipment resource characteristics. This application does not limit this.

[0022] In one possible implementation of the first aspect, the load includes a website backend program for information retrieval, the tested object includes a cluster, and the load business characteristics include at least one of the number of query requests processed by the tested object per unit time, the number of concurrent processing requests processed, and the average response time of query requests; the device resource characteristics include at least one of the memory occupied by the website backend program during runtime and the utilization rate of the central processing unit of the website backend program during runtime.

[0023] Taking the first load operation characteristic and the second load operation characteristic as being the same, both being the average response time of query requests, as an example, after the first load (i.e., the website backend program) runs on the tested object, the specific value of the corresponding average response time of query requests (i.e., the third characteristic value) can be used as the benchmark test result. In this case, the third characteristic value and the first characteristic threshold may be the same or different.

[0024] Taking the difference between the first and second load operating characteristics as an example, if the first load operating characteristic is the average response time of a query request, then the second load operating characteristic may include at least one of the following: the number of query requests processed by the tested object per unit time, the number of concurrent requests processed, the memory occupied by the website backend program during runtime, and the CPU utilization rate of the website backend program during runtime. If the first load operating characteristic is the memory occupied by the website backend program during runtime, the second load operating characteristic may include at least one of the following: the number of query requests processed by the tested object per unit time, the number of concurrent requests processed, the average response time of a query request, and the CPU utilization rate of the website backend program during runtime.

[0025] Furthermore, the second load performance characteristic may also include the first load performance characteristic. If the first load performance characteristic is the average response time of a query request, then the second load performance characteristic may also include the average response time of a query request. In addition, the second load performance characteristic may also include at least one of the following: the number of query requests processed by the tested object per unit time, the number of concurrent requests processed, the memory occupied by the website backend program during runtime, and the CPU utilization rate of the website backend program during runtime.

[0026] In this method, the second load operating characteristics and the first load operating characteristics can be the same or different, which makes the benchmark test results include a wider range of operating characteristics, and thus makes the benchmark test results more comprehensively reflect the performance level of the tested object.

[0027] In one possible implementation of the first aspect, the first benchmark model includes any one of a linear regression model, a decision tree model, a convolutional neural network model, a recurrent neural network model, and a transformer model.

[0028] It is understood that the embodiments of this application do not limit the type of the first benchmark model, as long as the first load corresponding to the first feature threshold can be obtained based on the first feature threshold. In this case, the selection of the type of the first benchmark model is more flexible, thereby making the implementation of the benchmark method provided by this application more flexible.

[0029] Secondly, this application provides a benchmark testing apparatus, which includes:

[0030] The first acquisition module is used to acquire the first feature threshold of the first load running characteristics when the load is running on the object under test.

[0031] The input module is used to input the first feature threshold into the first benchmark model during the Nth round of benchmark testing to obtain the first load of the tested object corresponding to the first load operating feature, where N is an integer greater than 1;

[0032] The second acquisition module is used to acquire the benchmark test results based on the first load when the change of the first load relative to the second load meets the test termination condition. The second load is obtained by inputting the first feature threshold into the second benchmark test model during the (N-1)th round of benchmark testing. The first benchmark test model is obtained by adjusting the second benchmark test model.

[0033] In one possible implementation of the second aspect, the test termination condition includes: the absolute value of the difference between the first load amount corresponding to the first load and the second load amount corresponding to the second load is less than or equal to the difference threshold.

[0034] In one possible implementation of the second aspect, the first benchmark model is obtained by: acquiring a first feature value of the first load running characteristics during the second load operation; adjusting the second benchmark model based on the difference between the first feature value and the first feature threshold to obtain the first benchmark model.

[0035] Therefore, the benchmark testing apparatus involved in this application may further include a runtime feature acquisition module, which is used to: acquire a first feature value of the runtime feature of the first load during the second load operation. The runtime feature acquisition module may, for example, include the service feature acquisition module and / or the resource feature acquisition module described later.

[0036] In addition, the benchmarking device may also include a feature modeling module, which is used to adjust the second benchmarking model based on the difference between the first feature value and the first feature threshold to obtain the first benchmarking model.

[0037] In one possible implementation of the second aspect, the feature modeling module is further used to: adjust the first benchmark model to obtain a third benchmark model when the change of the first load relative to the second load does not meet the test termination condition.

[0038] In one possible implementation of the second aspect, the feature modeling module is used to: adjust the first benchmark model based on the difference between the second feature value and the first feature threshold corresponding to the first load running feature during the first load running, to obtain the third benchmark model.

[0039] In one possible implementation of the second aspect, the second acquisition module is used to: acquire a third feature value of a second load running feature generated when the first load runs on the object under test; and use the third feature value as a benchmark test result, wherein the first load running feature and the second load running feature are the same or different.

[0040] It is understood that the second acquisition module can be the aforementioned operational feature acquisition module, or it can be a different module; this application does not limit this. If the second acquisition module and the operational feature acquisition module are the same, the operational feature acquisition module can acquire the feature values ​​of both the first and second load operational features. If the second acquisition module and the operational feature acquisition module are different, the operational feature acquisition module is used to acquire the feature values ​​of the first load operational feature, and the second acquisition module is used to acquire the feature values ​​of the second load operational feature.

[0041] In one possible implementation of the second aspect, the first load operation characteristic includes at least one of load service characteristics and device resource characteristics; the first characteristic threshold includes at least one of load service characteristic threshold and device resource characteristic threshold.

[0042] In one possible implementation of the second aspect, the load includes a website backend program for information retrieval, the tested object includes a cluster, and the load business characteristics include at least one of the number of query requests processed by the tested object per unit time, the number of concurrent processing requests processed, and the average response time of query requests; the device resource characteristics include at least one of the memory occupied by the website backend program during runtime and the utilization rate of the central processing unit of the website backend program during runtime.

[0043] In one possible implementation of the second aspect, the first benchmark model includes any one of a linear regression model, a decision tree model, a convolutional neural network model, a recurrent neural network model, and a transformer model.

[0044] Thirdly, this application provides an electronic device comprising: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the benchmark testing method of the first aspect and any possible implementation thereof.

[0045] Fourthly, this application provides a computer program product comprising: computer instructions that, when executed on an electronic device, cause the electronic device to perform the benchmark testing method of the first aspect and any possible implementation thereof.

[0046] The beneficial effects of the second to fourth aspects can be found in the first aspect and the beneficial effects of any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0047] Figure 1 According to some embodiments of this application, a schematic diagram of a scenario for benchmarking a computing cluster 100 is shown;

[0048] Figure 2 According to some embodiments of this application, a flowchart of a benchmark testing method is shown;

[0049] Figure 3A According to some embodiments of this application, a flowchart illustrating a method for determining a benchmark model M3 is shown;

[0050] Figure 3B According to some embodiments of this application, a schematic diagram of the interactive flow of a benchmarking method is shown;

[0051] Figure 4A According to some embodiments of this application, a schematic diagram of the structure of a first electronic device for performing a benchmark testing method is shown;

[0052] Figure 4B According to some embodiments of this application, a schematic diagram of the structure of an electronic device for performing a second benchmark testing method is shown.

[0053] Figure 5 According to some embodiments of this application, a schematic diagram of the acquisition process of a business feature acquisition module 401 is shown;

[0054] Figure 6According to some embodiments of this application, a schematic diagram of the acquisition process of a resource feature acquisition module 402 is shown;

[0055] Figure 7 According to some embodiments of this application, a schematic diagram of the cleaning process of a feature cleaning module 403 is shown;

[0056] Figure 8 According to some embodiments of this application, a schematic diagram of the workflow of a feature modeling module 405 is shown;

[0057] Figure 9 According to some embodiments of this application, a flowchart of a benchmark testing method in the related art is shown;

[0058] Figure 10 According to some embodiments of this application, a schematic diagram of the structure of a third electronic device is shown. Detailed Implementation

[0059] The illustrative embodiments of this application include, but are not limited to, benchmark testing methods, apparatuses, electronic devices, and program products.

[0060] It is understood that the technical solution of this application is applicable to various scenarios requiring benchmarking, including, in addition to the scenario mentioned above of running a website backend program on a cluster to benchmark the cluster, scenarios such as running an artificial intelligence model on a cluster to benchmark the cluster. In other words, the embodiments of this application do not limit the type of the object under test corresponding to the benchmarking test, or the type of load running on the object under test.

[0061] In this context, a website backend program refers to applications and services running on a server, used to handle business logic, data storage, and user requests. In this application, a website backend program can serve as a load. When benchmarking a load as the object of test, the website backend program can include a website backend program for querying, a website backend program for shopping, and a website backend program for storage. For a website backend program for querying, it can provide query services when running; for a website backend program for shopping, it can provide query services, transaction services, etc.; and for a website backend program for storage, it can provide data storage services when running.

[0062] Furthermore, a cluster is a system composed of multiple computers or servers. The nodes in a cluster can share computing tasks and load to achieve higher performance levels. Clusters can include computing clusters, storage clusters, database clusters, etc.

[0063] The following section provides a detailed description of a scenario where a website backend program is run on a computing cluster for benchmark testing, using the computing cluster as the test object and the website backend program as the load.

[0064] Figure 1 A schematic diagram illustrating a scenario for benchmarking a computing cluster 100 is shown. For example... Figure 1 As shown, before benchmarking, computing cluster 100 is in an idle state (i.e., not running any load). During a certain round of benchmarking, computing cluster 100 can load the website backend program (e.g., ...) corresponding to the load of that round of benchmarking. Figure 1 Then run the website backend program (A, B).

[0065] It's understandable that load is a specific manifestation or metric of load, used to describe the specific value or quantity of the load. If the load is a website backend program, then the load can be the number of website backend programs. If the load is an artificial intelligence model, then the load can be the number of artificial intelligence models. If the load is a database, then the load can be the number of databases. Therefore, in Figure 1 In this context, the load capacity refers to the number of backend programs on the website.

[0066] When a website's backend program is run, it provides certain services and has corresponding load characteristics. Therefore, the characteristic values ​​of the load characteristics of the website's backend program corresponding to this round of benchmark testing can be determined. Here, load characteristics are the evaluation dimensions of the performance level of the tested object under load. In this application, load characteristics may include load business characteristics and / or device resource characteristics. Load business characteristics are used to indicate different dimensions of business processing of the tested object under load, and device resource characteristics are used to indicate different dimensions of resource consumption when the load runs on the tested object. Therefore, in Figure 1 In this context, the load corresponds to the website's backend program. Load characteristics could include, for example, the number of query requests processed per unit of time, the number of concurrent requests it can handle, and the average response time for query requests. Device resource characteristics could include, for example, the amount of memory occupied by the website's backend program during runtime, and the utilization rate of the website's backend program's central processing units (CPU).

[0067] Furthermore, the characteristic values ​​of load operation characteristics are the specific manifestations or metrics of these characteristics, and can be concrete numerical values. The characteristic values ​​of load operation characteristics can be used to represent the performance level of the tested object under load. Therefore, in Figure 1In this context, the load corresponds to the website backend program. If the average response time for a query request is 0.5 seconds, then the characteristic value of the load operation characteristic (i.e., the average response time for a query request) is 0.5 seconds. If the number of query requests processed per unit time by the tested object is 100, then the characteristic value of the load operation characteristic (i.e., the number of query requests processed per unit time) is 100. If the number of concurrent requests that the tested object can handle is 100, then the characteristic value of the load operation characteristic (i.e., the number of concurrent requests that can be handled) is 100. If the CPU utilization rate of each website backend program during runtime is 10%, then the characteristic value of the load operation characteristic (i.e., the CPU utilization rate of each website backend program during runtime) is 10%.

[0068] Then, when the running load reaches the preset load limit, the performance level of computing cluster 100 is obtained by using the feature value of the load running characteristics corresponding to the load running at the preset load limit, that is, the benchmark test result.

[0069] Based on the foregoing, in some embodiments, the preset load limit is set by the user based on experience. Taking the load as a website backend program, and the user's predicted preset load limit indicating ten website backend programs as an example, during the benchmark test, ten website backend programs can be run on the test object, and the characteristic values ​​of the load operation characteristics of the ten website backend programs (such as the number of query requests processed per unit time and the number of concurrent processing requests that can be processed) are used as the benchmark test results.

[0070] However, in the benchmarking methods described above, the user-set preset load limit differs significantly from the actual load capacity of the tested object. The actual load capacity of the tested object refers to the maximum load it can withstand under stable and normal operating conditions. If the load exceeds the actual capacity, the tested object may experience performance degradation (e.g., response latency or malfunction). For example, if the preset load limit is greater than the actual load capacity, it will lead to overload; if the preset load limit is less than the actual load capacity, it will lead to underload. Both situations result in lower accuracy of the benchmarking results.

[0071] To address the aforementioned technical problems, this application provides a benchmarking method. In this application, the benchmarking method may include multiple rounds of testing. In the first round of benchmarking, a first feature threshold (which can be determined empirically or otherwise) of a first load operating characteristic is input into benchmarking model M1 (as an example of a second benchmarking model) to obtain load Load-1 (as an example of a second load). Then, in the second round of benchmarking, benchmarking model M1 is trained based on the difference between the feature value C1 (as an example of a first feature value) corresponding to the first load operating characteristic during load Load-1 operation and the first feature threshold, resulting in benchmarking model M2 (as an example of a first benchmarking model). The first feature threshold is input into benchmarking model M2 to obtain load Load-2 (as an example of a first load). Then, it is determined whether the change in load Load-2 relative to load Load-1 satisfies the test termination condition, for example, whether the absolute value of the difference between the first load amount corresponding to load Load-2 and the second load amount corresponding to load Load-1 is less than a difference threshold.

[0072] If yes, then obtain the benchmark results based on Load-2. If not, continue training the quasi-test model M2 until the load obtained based on the trained benchmark model meets the test termination condition, and then obtain the benchmark results based on the load that meets the test termination condition. For example, benchmark model M2 is trained based on the difference between the feature value C2 (as an example of the second feature value) corresponding to the first load running feature during Load-2 and the first feature threshold, resulting in benchmark model M3 (as an example of the third benchmark model).

[0073] In the method provided in this application, the load Load-2 is obtained based on a trained benchmark model M2, and the benchmark result based on Load Load-2 is obtained when Load Load-2 meets the test termination condition. The benchmark result obtained in this way corresponds to a Load Load-2 that meets the test termination condition. Therefore, compared to a preset load set based on experience, the load Load-2 of this application is closer to the actual load capacity of the tested object, resulting in higher accuracy of the benchmark results.

[0074] In some embodiments, the feature value C3 of the second load running feature generated by Load-2 running on the test object (as an example of the third feature value) can be used as the benchmark test result.

[0075] Taking the load as a website backend program as an example, if the second load corresponding to load Load-1 is ten, and the first load corresponding to load Load-2 is also ten, that is, the first load equals the second load, which meets the test termination condition, and the benchmark test results based on ten website backend programs can be obtained.

[0076] In some embodiments, the first load operation characteristic includes load service characteristics and / or device resource characteristics. The load service characteristics are used to indicate different dimensions of service processing when the tested object is running a load, and the device resource characteristics are used to indicate different dimensions of resource consumption when the load is running on the tested object. In some embodiments, the second load operation characteristic may also include load service characteristics and / or device resource characteristics, and the first load operation characteristic and the second load operation characteristic may be the same or different.

[0077] In some embodiments, the benchmark model (e.g., benchmark model M1, benchmark model M2, etc.) can be a machine learning model such as linear regression or Q-learning, or a deep learning model such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer. This application does not limit the type of benchmark model. It is understood that regardless of the type of benchmark model in this application, the benchmark model can obtain the corresponding load based on a feature threshold.

[0078] Before describing the benchmarking method of this application in detail below, the subject of the benchmarking method will first be described. It is understood that the benchmarking method involved in the embodiments of this application can be applied to electronic devices. In some embodiments, electronic devices include, but are not limited to, servers, tablets, computers with wireless transceiver capabilities, virtual reality (VR) devices, augmented reality (AR) devices, wireless devices in industrial control, and so on.

[0079] Furthermore, the test object of the benchmarking method involved in this application may include one or more devices under test, including but not limited to (cloud) servers, virtual machines, servers for heterogeneous computing, etc. In some embodiments, corresponding to the test object including multiple devices under test, and the devices under test being servers, the test object may be a cluster including multiple servers.

[0080] It is understood that, when the object under test includes only one device under test, the electronic device executing the benchmark test method of this application may be the same as or different from the device under test. When the object under test includes multiple devices under test, the electronic device executing the benchmark test method of this application may be a device different from the multiple devices under test, or it may be any one of the multiple devices under test.

[0081] The benchmark testing method provided in this application will be described in detail below with reference to the accompanying drawings. Figure 2 A flowchart illustrating a benchmark testing method provided in this application is shown, which can be applied to electronic devices. Figure 2 As shown, taking a maximum of three rounds of benchmark testing as an example, the method includes:

[0082] 201: Obtain the first feature threshold of the first load running characteristic when the load is running on the object under test.

[0083] In this application embodiment, the object under test may include one or more devices under test.

[0084] In this application, the first load operation feature is the feature category corresponding to the state that the user expects the object under test to reach when benchmarking the object under test. Therefore, the first feature threshold corresponding to the first load operation feature is used to represent the feature value corresponding to the state that the user expects the object under test to reach.

[0085] The first load operation characteristic includes at least one of load service characteristics and device resource characteristics. The load service characteristics indicate different dimensions of service processing during load operation on the tested object, while the device resource characteristics indicate different dimensions of resource consumption during load operation on the tested object.

[0086] Taking a website backend program as an example, load balancing characteristics can include the number of query requests processed per unit of time, the number of concurrent requests it can handle, and the average response time of query requests. Device resource characteristics can include the amount of memory occupied by the website backend program during runtime and the CPU utilization rate of the website backend program during runtime.

[0087] Based on this, the first characteristic threshold may include at least one of the load service characteristic threshold and the device resource characteristic threshold. It can be understood that the first characteristic threshold can be preset by the user based on experience.

[0088] If the load is a website backend program used for querying, and the user expects the tested object to achieve an average response time of 0.5 seconds for query requests during benchmarking, then the first load performance characteristic is the average response time for query requests, and the first characteristic threshold is 0.5 seconds. If the user expects the tested object to achieve 100% CPU utilization during benchmarking, then the first load performance characteristic is CPU utilization, and the first characteristic threshold is 100%.

[0089] This application does not limit the method of obtaining the first feature threshold. For example, the electronic device can read a configuration file to obtain the first feature threshold from the configuration file. In this case, the first feature threshold in the configuration file can be pre-configured by the user based on experience or other methods. Taking the first load operation feature as including load business features, and the load business feature being the average response time of a query request, as an example, the user can, based on experience, pre-store the first feature threshold (e.g., 0.5s) corresponding to the average response time of a query request in the configuration file for the electronic device to obtain. Taking the first load operation feature as including device resource features, and the device resource feature being the CPU utilization rate of the website backend program during operation, as an example, the user can, based on experience, pre-store the first feature threshold (e.g., 100%) corresponding to the CPU utilization rate of the website backend program during operation in the configuration file for the electronic device to obtain.

[0090] In other embodiments, the electronic device can determine a first feature threshold of the first load operating characteristic based on the physical parameters of the object under test. For example, if the first load operating characteristic includes device resource characteristics, and the device resource characteristic is the amount of memory occupied by the website's backend program during runtime, then if the electronic device obtains that the total memory parameter of the object under test is 1T, then it can determine that the first feature threshold corresponding to this device resource characteristic is 1T. In some embodiments, the electronic device can determine the physical parameters of the object under test through acquisition. In this case, the electronic device can acquire the physical parameters of the object under test based on the resource feature acquisition module, and then determine the first feature threshold of the first load operating characteristic based on the physical parameters.

[0091] 202: In the first round of benchmark testing, the first feature threshold is input into the benchmark testing model M1 to obtain the load Load-1 of the tested object corresponding to the first load operation feature.

[0092] This application does not limit the type of the benchmark model M1, as long as the load Load-1 corresponding to the first feature threshold can be obtained based on the first feature threshold. For example, the benchmark model M1 can be any one of a linear regression model, a decision tree model, a convolutional neural network model, a recurrent neural network model, or a transformer model. It can be understood that the load Load-1 can be the load corresponding to the second load amount.

[0093] It is understood that in this embodiment of the application, when the object under test includes multiple devices under test, Load-1 can include the load corresponding to each device under test, and the load corresponding to each device under test can be the same or different. For example, the load amount corresponding to each device under test can be the same or different. In this way, when Load-1 is executed on the object under test later, the corresponding load can be executed on different devices under test.

[0094] 203: Obtain the characteristic value C1 corresponding to the first load running characteristic during the operation of Load-1.

[0095] It can be understood that the characteristic value C1 is the actual characteristic value corresponding to the first load running characteristic when the load is Load-1. Taking the load as a website backend program and the first load running characteristic as the average response time of a query request being 0.4s as an example, the characteristic value C1 of the first load running characteristic when the website backend program is running under load-1 is 0.4s.

[0096] 204: In the second round of benchmark testing, the benchmark model M1 is adjusted based on the difference between the feature value C1 and the first feature threshold to obtain the benchmark model M2.

[0097] It is understandable that, since the first feature threshold is directly input into the benchmark model M1 in the first round of benchmark testing to obtain the load Load-1, the benchmark model M1 can be directly trained based on the difference between the feature value C1 corresponding to the load Load-1 and the first feature threshold in the second round of benchmark testing to obtain the benchmark model M2.

[0098] This application does not limit the type of the benchmark model M2. For example, the benchmark model M2 can be any one of a linear regression model, a decision tree model, a convolutional neural network model, a recurrent neural network model, or a transformer model. Furthermore, the benchmark model M2 and the benchmark model M1 are of the same type.

[0099] 205: Input the first feature threshold into the benchmark test model M2 to obtain the load Load-2 of the tested object corresponding to the first load operation feature.

[0100] 206: Determine whether the change in load Load-2 relative to load Load-1 meets the test termination condition.

[0101] Understandably, if the change in load Load-2 relative to load Load-1 satisfies the test termination condition, then execute step 207 to obtain the benchmark test results based on load Load-2. If not, if the change in load Load-2 relative to load Load-1 does not satisfy the test termination condition, then execute step 208 to adjust the benchmark test model M2 to obtain the benchmark test model M3.

[0102] It is understandable that, since the load is typically increased round by round during multi-round benchmarking, the test termination condition in this application indicates that, compared to the load of the previous round of benchmarking (e.g., the second load), there is no room for further increases in the load (e.g., the first load) compared to the load of the previous round of benchmarking (e.g., the second load), provided that the user expects the first load operating characteristic of the tested object to reach the first characteristic threshold. At this point, the benchmarking result based on the load (e.g., the first load) of the current round of benchmarking can be obtained.

[0103] In some embodiments, the test termination condition may include: the absolute value of the difference between the load corresponding to the load in the current benchmark test and the load corresponding to the load in the previous benchmark test is less than or equal to a difference threshold. For example, in 206, the test termination condition may be that the absolute value of the difference between the first load corresponding to load Load-2 and the second load corresponding to load Load-1 is less than or equal to the difference threshold. This application does not limit the selection of the difference threshold; it can be set based on experience or flexibly adjusted according to the actual application scenario. For example, the difference threshold can be 0. In this case, if the first load and the second load are the same, then the change in load Load-2 relative to load Load-1 satisfies the test termination condition.

[0104] It is understood that the above example only uses the load amount corresponding to the load as an example to illustrate the test termination condition. However, it should be understood that the test termination condition can also be other metrics corresponding to the load, and this application embodiment does not limit this.

[0105] 207: Obtain the benchmark test results based on Load-2.

[0106] In some embodiments, obtaining the benchmark test result based on Load-2 may include: obtaining the feature value C3 of the second load running feature generated by Load-2 when it runs on the object under test; and using the feature value C3 as the benchmark test result.

[0107] Based on the content described in section 202 above, taking an example where the test object includes multiple devices under test, each device under test can run a corresponding load and generate a corresponding feature value C3. In this way, the electronic device can obtain the feature value C3 and use the feature value C3 as the benchmark test result.

[0108] The second load operation characteristic may include at least one of load service characteristics and device resource characteristics, or it may include other types of characteristics besides load service characteristics and device resource characteristics. This application embodiment does not limit this. For example, the content of the second load operation characteristic may be based on the actual application scenario or user requirements.

[0109] As can be understood, based on the foregoing, the first load operation characteristic is the feature category corresponding to the state that the user expects the test object to reach when benchmarking the test object. Furthermore, the first feature threshold corresponding to this first load operation characteristic is used to determine the load (such as the first load) applied to the test object. When the first load runs on the test object, it corresponds to a third feature value of the second load operation characteristic, and the third feature value is: the actual feature value that the first load operation characteristic and / or other load operation characteristics can actually reach when the user expects the test object to reach the first feature threshold of the first load operation characteristic. Therefore, the second load operation characteristic can be the same as the first load operation characteristic, or it can be different from the first load operation characteristic (for example, the second load operation characteristic is another load operation characteristic different from the first load operation characteristic, or the second load operation characteristic includes both the first load operation characteristic and other load operation characteristics different from the first load operation characteristic).

[0110] When the first load operating characteristic and the second load operating characteristic are the same, taking the load as a website backend program and both the first load operating characteristic and the second load operating characteristic as the average response time of a query request as an example, the electronic device can use the specific value of the average response time of the query request corresponding to the operation of load Load-2 (website backend program) (i.e., characteristic value C3) as the benchmark test result. It is understood that characteristic value C3 and the first characteristic threshold can be the same or different, and this application embodiment does not limit this.

[0111] When the first load operating characteristic and the second load operating characteristic differ, taking a website backend program as the load and the average response time of query requests as the first load operating characteristic as an example, the electronic device can use the characteristic value C3 of the second load operating characteristic, which differs from the first load operating characteristic, as the benchmark test result. For example, the second load operating characteristic can be a load business characteristic, such as the number of query requests processed per unit time or the number of concurrent requests processed by the tested object. Another example is that the second load operating characteristic can be a device resource characteristic, such as the memory occupied by the website backend program during runtime or the CPU utilization rate of the website backend program during runtime.

[0112] In this case, the second operational characteristic may also include the first load operational characteristic. For example, the second load operational characteristic includes the average response time of query requests. In addition, the second load operational characteristic may also include at least one of the following: the number of query requests processed by the tested object per unit time, the number of concurrent requests processed, the memory occupied by the website backend program during runtime, and the CPU utilization rate of the website backend program during runtime.

[0113] When the first load operating characteristic and the second load operating characteristic differ, taking a website backend program as the load, and the first load operating characteristic as the memory occupied by the website backend program during runtime as an example, the electronic device can use the characteristic value C3 of the second load operating characteristic, which differs from the first load operating characteristic, as the benchmark test result. For example, the second load operating characteristic can be a load business characteristic, such as the number of query requests processed per unit time, the number of concurrent requests processed, and the average response time of query requests. Another example is that the second load operating characteristic can be a device resource characteristic, such as the CPU utilization rate of the website backend program during runtime.

[0114] In this case, the second operational characteristic may also include the first load operational characteristic. For example, the second load operational characteristic includes the memory occupied by the website backend program during runtime. In addition, the second load operational characteristic may also include at least one of the following: the number of query requests processed by the tested object per unit time, the number of concurrent processing requests, the average response time of query requests, and the CPU utilization of the website backend program during runtime.

[0115] 208: During the third round of benchmark testing, the benchmark model M2 was adjusted to obtain the benchmark model M3.

[0116] It can be understood that adjusting the benchmark model M2 to obtain the benchmark model M3 can be understood as training the benchmark model M2 and using the trained benchmark model M2 as the benchmark model M3.

[0117] In some embodiments, such as Figure 3A As shown, adjusting the benchmark model M2 to obtain the benchmark model M3 can include the following schemes:

[0118] 2081: Obtain the characteristic value C2 corresponding to the first load running characteristic during the Load-2 runtime.

[0119] It can be understood that the characteristic value C2 is the actual characteristic value corresponding to the first load running characteristic when the load-2 is running. Taking the load as the website backend program and the first load running characteristic as the average response time of the query request being 0.4s as an example, the characteristic value C2 of the first load running characteristic when the load-2 (i.e., the website backend program) is running is 0.4s.

[0120] 2082: Adjust the benchmark model M2 based on the difference between the eigenvalue C2 and the first eigenvalue threshold to obtain the benchmark model M3.

[0121] This application does not limit the type of benchmark model M3. For example, benchmark model M3 can be any of the following: linear regression model, decision tree model, convolutional neural network model, recurrent neural network model, or transformer model. Furthermore, benchmark model M3 is of the same type as benchmark models M1 and M2.

[0122] 209: Input the first feature threshold into the benchmark test model M3 to obtain the load Load-3 of the tested object corresponding to the first load operation feature.

[0123] 210: The test ends when the change in load Load-3 relative to load Load-2 meets the test termination condition, and the benchmark test results based on load Load-3 are obtained.

[0124] It is understood that the test termination condition involved in this process may include: the absolute value of the difference between the third load quantity corresponding to load Load-3 and the first load quantity corresponding to load Load-2 is less than or equal to the difference threshold. This difference threshold may be the same as or different from the difference threshold in section 206 above; this embodiment of the application does not limit this.

[0125] In some embodiments, obtaining the benchmark test result based on Load-3 may include: obtaining the feature value C4 of the third load operation feature generated when Load-3 runs on the object under test; and using the feature value C4 as the benchmark test result. The third load operation feature includes at least one of load service features and device resource features. Furthermore, the third load operation feature may be the same as or different from the first and second load operation features.

[0126] The benchmarking method provided in this application, considering both load business characteristics and device resource characteristics, obtains the load corresponding to each round of benchmarking based on a benchmarking model. The benchmarking results obtained in this way can reflect the true load capacity of the tested object, thus improving the accuracy of the benchmarking results. Furthermore, this benchmarking method can be used to test objects with relatively simple structures, such as a single server, as well as those with more complex structures, such as those including multiple tested devices (e.g., a cluster of multiple servers), demonstrating strong adaptability to different testing scenarios.

[0127] Understandable. Figure 2 Taking the example where the change in load Load-3 relative to load Load-2 satisfies the test termination condition, the benchmark testing method provided in this application is described. In some other embodiments, if the change in load Load-3 relative to load Load-2 does not satisfy the test termination condition, a process similar to the principles described in sections 204 to 206 above can be repeated, that is, the benchmark testing model M3 is adjusted to obtain the benchmark testing model M4. The first feature threshold is input into the benchmark testing model M4 to obtain the load Load-4 of the tested object corresponding to the first load operating characteristic. It is then determined whether the change in load Load-4 relative to load Load-3 satisfies the test termination condition, and so on. This process continues until a load that satisfies the test termination condition is obtained, and the benchmark testing result based on that load is acquired.

[0128] It is understandable that the above Figure 2 The benchmark testing method described in this application is based on an electronic device as the execution subject. As described above, when the object under test includes only one device under test, the electronic device executing the benchmark testing method of this application may be the same as or different from the device under test. When the object under test includes multiple devices under test, the electronic device executing the benchmark testing method of this application may be a device different from the object under test, or it may be any one of the devices under test.

[0129] Therefore, taking electronic devices as a different type of device from the object being tested, and taking the example of obtaining benchmark test results through two rounds of benchmark testing, Figure 3B A schematic diagram of the interactive flow of a benchmarking method is shown. Figure 3B As shown, this method can be executed interactively by an electronic device and the object under test, and the method includes, but is not limited to, the following schemes.

[0130] 301: The first feature threshold for acquiring the first load running characteristics of an electronic device when the load is running on the test object.

[0131] 302: During the first round of benchmark testing, the electronic device inputs the first feature threshold into the benchmark model M1 to obtain the load Load-1 of the tested object corresponding to the first load operating feature.

[0132] It is understood that the principles of the contents shown in 301 and 302 above are the same as those of the contents shown in 201 and 202 above, and will not be repeated here.

[0133] 303: The electronic device sends Load-1 to the test object.

[0134] It is understandable that after receiving the load Load-1, the electronic device can send the load Load-1 to the object under test so that the object under test can run.

[0135] 304: The tested object runs under load Load-1 and generates the characteristic value C1 corresponding to the first load running characteristic.

[0136] It is understandable that the relevant description of eigenvalue C1 in 304 can be found in the relevant description in 203 above, and will not be repeated here.

[0137] 305: The object under test sends the feature value C1 to the electronic device.

[0138] 306: During the second round of benchmark testing, the electronic device adjusts the benchmark test model M1 based on the difference between the feature value C1 and the first feature threshold to obtain the benchmark test model M2.

[0139] 307: The electronic device inputs the first feature threshold into the benchmark test model M2 to obtain the load Load-2 of the tested object corresponding to the first load operation feature.

[0140] It is understood that the principles of the contents shown in 306 and 307 above are the same as those of the contents shown in 204 and 205 above, and will not be repeated here.

[0141] 308: When the change of load Load-2 relative to load Load-1 meets the test termination condition, the electronic device sends load Load-2 to the object under test.

[0142] As you can understand, the relevant details regarding the test termination conditions can be found in the previous section 206, and will not be repeated here.

[0143] 309: The tested object runs under load Load-2 and generates the characteristic value C3 corresponding to the second load running characteristic.

[0144] It is understood that the relevant content regarding the second load operating characteristics can be found in the relevant description in section 207 above, and will not be repeated here.

[0145] 310: The object under test sends the feature value C3 to the electronic device.

[0146] 312: Electronic devices use the characteristic value C3 as the benchmark test result.

[0147] It is understandable that, corresponding to the case where the test object includes multiple devices under test, and the electronic device is any one of the devices under test, the interaction process when the electronic device performs benchmark testing on other devices under test is similar to... Figure 3B The process is similar, so it will not be repeated here. Furthermore, the test procedure for benchmarking an electronic device against itself (i.e., the electronic device being one of the devices under test) is the same as described above. Figure 2 The process shown is similar and will not be repeated here.

[0148] The structure of the electronic device involved in this application is described below. Taking an example where the electronic device executing the benchmark test method of this application differs from the object under test,... Figure 4A A schematic diagram of the structure of an electronic device is shown. Figure 4A As shown, the electronic device may include a service feature acquisition module 401, a resource feature acquisition module 402, a feature cleaning module 403, a database module 404, a feature modeling module 405, a configuration module 406, a load sending module 407, and a reporting module 408.

[0149] Taking the current round as the Nth round of benchmark testing as an example, the service feature acquisition module 401 is used to acquire the feature values ​​of the load service features during the (N-1)th round of benchmark testing. The resource feature acquisition module 402 is used to acquire the feature values ​​of the device resource features during the (N-1)th round of benchmark testing. It can be understood that when N is 2, the feature values ​​of the load service features and the device resource features during the (N-1)th round of benchmark testing can include the feature value C1 mentioned above, or the feature values ​​corresponding to other load operation features during the Load-1 runtime. It can be understood that the feature values ​​acquired by the service feature acquisition module 401 and the resource feature acquisition module 402 can be used to train the benchmark testing model. For example, the difference between feature value C1 and the first feature threshold corresponding to the first load operation feature can be used to train the second benchmark testing model to obtain the first benchmark testing model.

[0150] The feature cleaning module 403 is used to clean the feature values ​​of the collected load service features and device resource features, such as handling abnormal data and missing data. The database module 404 is used to store the cleaned feature values ​​of the load service features and device resource features.

[0151] The feature modeling module 405 is used to train the benchmark model (e.g., benchmark model M1) based on the feature values ​​(e.g., feature value C1) of the load service characteristics and device resource characteristics to obtain the trained benchmark model (e.g., benchmark model M2), and predict the corresponding load (e.g., load Load-2) based on the trained benchmark model.

[0152] The configuration module 406 includes the specific types of load service characteristics and device resource characteristics that need to be collected, such as a characteristic value representing the average response time of a query request. In some embodiments, the service characteristic collection module 401 and the resource characteristic collection module 402 can collect the characteristic values ​​of relevant features based on the configuration module 406.

[0153] The load sending module 407 is used to send the load output by the feature modeling module 405 to the object under test. The reporting module 408 is used to report the benchmark test results. Taking N as 2 as an example, when the first load meets the test termination condition, the service feature acquisition module 401 and the resource feature acquisition module 402 can send the third feature value (i.e., the benchmark test result) of the acquired second load running features to the reporting module 408. Then, the reporting module 408 can report the benchmark test results.

[0154] It is understandable that if the current round is the first round of benchmark testing, then the business feature acquisition module 401 is not working, and the resource feature acquisition module 402 can acquire the feature values ​​of the device resource features. At this time, the feature values ​​of the device resource features can be used as the feature thresholds corresponding to the device resource features. In addition, in this case, the feature modeling module 405 can directly input the feature thresholds of the device resource features into the preset benchmark test model to obtain the load. The functions of the remaining modules are similar to those of the relevant modules in the Nth round of benchmark testing, and will not be described in detail here.

[0155] Furthermore, if the electronic device performing the benchmark test method of this application is the same as the object under test, then the structural schematic diagram of the electronic device can be as follows: Figure 4B As shown. Figure 4B As shown, the electronic device also includes a business feature acquisition module 401, a resource feature acquisition module 402, a feature cleaning module 403, a database module 404, a feature modeling module 405, a configuration module 406, and a reporting module 408. The functions of each of these modules are the same as described above. Figure 4A The corresponding modules have the same function, so they will not be described in detail here.

[0156] and Figure 4A The structures shown are different, Figure 4BThe system does not include a load sending module 407, but instead includes a load driving module 409. The load driving module 409 is used to run the load (such as load Load-1) output by the benchmark model.

[0157] The workflows of the above-mentioned business feature acquisition module 401, resource feature acquisition module 402, feature cleaning module 403, and feature modeling module 405 are described in detail below.

[0158] Figure 5 A schematic diagram of the data acquisition process of a business feature acquisition module 401 is shown. For example... Figure 5 As shown, the business feature acquisition module 401 can acquire a load business feature information table and a business feature acquisition worksheet. The load business feature information table may include feature names, feature meanings, feature units, etc.

[0159] Taking the load as the website backend program and the load business characteristic as the number of query requests processed by the tested object per unit time as an example, the characteristic name can be, for example, "the number of query requests processed per unit time"; the characteristic meaning can be "the number of query requests that the tested object can process per unit time when running the website backend program"; the characteristic unit is "number".

[0160] Taking the load business characteristic as the number of concurrent requests that the tested object can handle as an example, the characteristic name can be, for example, "the number of concurrent requests that can be handled"; the characteristic meaning can be "the number of concurrent requests that the tested object can handle at a certain moment when running the website backend program"; the characteristic unit is "number".

[0161] Taking the average response time of query requests as a characteristic of load business as an example, the characteristic name can be "average response time of query requests"; the characteristic meaning can be "average response time of multiple query requests when the tested object runs the website backend program"; the characteristic unit is "seconds (s)".

[0162] It is understood that the feature names listed above are merely examples, and this application does not limit the form of the feature names, as long as the feature names correspond one-to-one with the load service features they represent. For example, feature names can also be represented by feature identity documents (IDs).

[0163] Furthermore, in some embodiments, the business feature collection worksheet may include the collection methods for various load business features. Taking the load business feature as the number of query requests processed by the tested object per unit time as an example, the collection method for this load business feature is, for example, to obtain the number of query requests processed by each website backend program running on the tested object within 1 second, obtain multiple query request numbers, and then add the multiple query request numbers together to obtain the number of query requests that the tested object can process within 1 second.

[0164] Taking the load business characteristic as the number of concurrent requests that the tested object can handle as an example, the collection method for this load business characteristic is, for example, to obtain the number of query requests that all website backend programs running on the tested object can handle at a certain moment.

[0165] Taking the average response time of query requests as a load business characteristic as an example, the method for collecting this load business characteristic is, for example, to obtain the response time of each query request in a certain number of query requests, resulting in multiple response times. Then, by averaging the multiple response times, the average response time of the query request can be obtained.

[0166] After obtaining the load service characteristic information table and the service characteristic collection worksheet, the service characteristic collection module 401 can begin collecting the characteristic values ​​of the relevant load service characteristics. In some embodiments, the service characteristic collection module 401 can collect the characteristic values ​​of the load service characteristics at different times. For example, the service characteristic collection module 401 can collect the characteristic values ​​of the load service characteristics for the first time at time T1, the second time at time T2, and the third time at time T3. This embodiment does not limit the collection interval of the load service characteristic characteristic values ​​(e.g., the time interval between time T1 and time T2), which can be set based on experience or flexibly adjusted according to the actual application scenario. Furthermore, the time intervals can be the same or different; for example, the time interval between time T1 and time T2 can be the same or different from the time interval between time T2 and time T3.

[0167] After collecting the feature values ​​of load service characteristics at different times, the feature values ​​corresponding to different times are combined with the information in the load service feature information table to obtain the time-series data of load service characteristics (hereinafter referred to as service time-series data). Then, the service time-series data is sent to the feature cleaning module 403.

[0168] It is understandable that in this application, feature values ​​of load business characteristics are collected multiple times at different times. The obtained feature values ​​can more comprehensively reflect the performance of the tested object during a certain load operation, thereby making the model trained based on the feature values ​​more accurate.

[0169] Figure 6 A schematic diagram of the acquisition process of a resource feature acquisition module 402 is shown. Figure 6 As shown, the resource feature acquisition module 402 can acquire the equipment resource feature information table and the resource feature acquisition worksheet. The equipment resource feature information table may include feature name, feature meaning, feature unit, etc.

[0170] Taking the load as the website backend program and the device resource characteristic as the memory size occupied by the website backend program when it runs as an example, the characteristic name can be, for example, "memory size occupied by the tested object"; the characteristic meaning can be "memory occupied by the tested object when running the website backend program"; the characteristic unit is "bytes".

[0171] Taking the CPU utilization rate of a website's backend program as an example, the feature name can be "CPU utilization rate"; the feature meaning can be "CPU utilization when the tested object runs the website's backend program"; and the feature unit is "%".

[0172] It is understood that the feature names listed above are merely examples, and this application does not limit the form of the feature names, as long as the feature names can correspond one-to-one with the device resource features they represent. For example, feature names can also be represented by feature identification numbers (IDs).

[0173] Furthermore, in some embodiments, the resource feature collection worksheet may include the collection methods for various device resource features. Taking the memory size occupied by a website backend program on the tested object during runtime as an example, the collection method for this device resource feature is, for example, to obtain the memory size occupied by each website backend program on the tested object during runtime, obtain multiple memory sizes, and then add the multiple memory sizes together to obtain the memory size occupied by the website backend program on the tested object during runtime.

[0174] Taking the CPU utilization rate of a website's backend program during runtime as an example, the method for collecting this device resource characteristic is as follows: obtain the CPU utilization rate of each website's backend program on the tested object during runtime, obtain multiple CPU utilization rates, and then add the multiple CPU utilization rates together to obtain the CPU utilization rate of multiple website backend programs on the tested object during runtime.

[0175] After obtaining the device resource feature information table and the resource feature acquisition worksheet, the resource feature acquisition module 402 can begin collecting the feature values ​​of the relevant device resource features. In some embodiments, the resource feature acquisition module 402 can collect the feature values ​​of the aforementioned device resource features at different times. For example, the device resource feature acquisition module 402 can collect the feature values ​​of the device resource features for the first time at time T1, the feature values ​​of the device resource features for the second time at time T2, and the feature values ​​of the device resource features for the third time at time T3.

[0176] In this embodiment, the acquisition interval of the feature values ​​of the device resource characteristics (e.g., the time interval between time T1 and time T2) is not limited. It can be set based on experience or flexibly adjusted according to the actual application scenario. In addition, the time interval can be the same or different. For example, the time interval between time T1 and time T2 can be the same or different from the time interval between time T2 and time T3.

[0177] After collecting the feature values ​​of the equipment resource characteristics at different times, the feature values ​​corresponding to different times are combined with the information in the equipment resource feature information table to obtain the time-series data of the equipment resource characteristics (hereinafter referred to as resource time-series data). Then, the resource time-series data is sent to the feature cleaning module 403.

[0178] Figure 7 A schematic diagram of the cleaning process of a feature cleaning module 403 is shown. For example... Figure 7 As shown, the cleaning process includes:

[0179] 701: Merge business time-series data and resource time-series data to obtain merged data.

[0180] In this embodiment of the application, after the feature cleaning module 403 obtains the service time-series data sent by the service feature acquisition module 401 and the resource time-series data sent by the resource feature acquisition module 402, it can first merge the service time-series data and the resource time-series data, for example, merge the time-series data corresponding to the same time.

[0181] Taking business time-series data as an example, it can include the feature values ​​of the load business characteristics at time T1 and the feature values ​​of the load business characteristics at time T2. Similarly, taking resource time-series data as an example, it can include the feature values ​​of the device resource characteristics at time T1 and the feature values ​​of the device resource characteristics at time T2.

[0182] In the above example, the feature cleaning module 403 can merge the service time-series data and resource time-series data corresponding to time T1 and time T2 respectively to obtain merged time-series data (i.e., merged data). For example, the feature values ​​of the load service feature and the device resource feature corresponding to time T1 can be merged to obtain the merged data corresponding to time T1. As another example, the feature values ​​of the load service feature and the device resource feature corresponding to time T2 can be merged to obtain the merged data corresponding to time T2.

[0183] 702: Process abnormal data in the merged data to obtain the first merged data after abnormal data processing.

[0184] In this embodiment, after the feature cleaning module 403 obtains the merged data corresponding to each of the multiple time points, it can process the abnormal data in the merged data corresponding to the multiple time points. It can be understood that if the merged data corresponding to a certain time point is too large or too small compared to the merged data corresponding to other time points, the merged data corresponding to that time point is abnormal data.

[0185] This application does not limit the processing method for abnormal data. In some embodiments, the feature cleaning module 403 can delete (or filter) abnormal data or correct abnormal data. Taking the merged data corresponding to a certain moment as abnormal data as an example, the feature cleaning module 403 can delete the merged data corresponding to that moment. In addition, the feature cleaning module 403 can also correct the abnormal data corresponding to that moment to normal data based on the merged data (normal data) corresponding to other moments. For example, the average value of the merged data corresponding to other moments can be used as the merged data corresponding to that moment.

[0186] It is understandable that since the merged data is used to train the benchmark model, the operation of processing the outlier data in the merged data can not only improve the accuracy of the merged data, but also improve the accuracy of the benchmark model obtained by the subsequent feature modeling module 405, thereby making the load obtained by the model more accurate.

[0187] In some embodiments, 702 is optional, that is, after the feature cleaning module 403 obtains the merged data, it may not process the abnormal data in the merged data, but instead perform the processing procedures shown in subsequent steps 703 on the merged data.

[0188] 703: Process the missing data in the first merged data to obtain the second merged data after processing the missing data.

[0189] This application does not limit the processing method for missing data. In some embodiments, the feature cleaning module 403 can delete the remaining merged data at the time corresponding to the missing data, or fill in the missing data. Taking the merged data at a certain time as an example, which only includes resource time-series data and lacks service time-series data, the feature cleaning module 403 can delete the resource time-series data in the merged data at that time, that is, delete the remaining merged data at that time, and obtain the second merged data.

[0190] In addition, the feature cleaning module 403 can also fill the missing data corresponding to this time based on the merged data corresponding to other times. For example, the merged data (missing data) corresponding to this time can be obtained by interpolation (such as linear interpolation), and then the second merged data after processing the missing data can be obtained.

[0191] It is understandable that since the merged data is used to train the benchmark model, processing operations such as imputing missing data in the merged data can not only improve the richness and accuracy of the merged data, but also improve the accuracy of the benchmark model obtained by the subsequent feature modeling module 405, thereby making the accuracy of the load obtained by the model higher.

[0192] In some embodiments, 703 is optional. That is, after obtaining the first merged data, the feature cleaning module 403 may not process the missing data in the first merged data, but instead perform the processing procedures shown in subsequent steps such as 704 on the merged data.

[0193] It is understood that the order in which 702 and 703 are executed is not limited in the embodiments of this application. For example, the feature cleaning module 403 may process the missing data in the merged data first and then process the abnormal data in the merged data after obtaining the merged data.

[0194] 704: The second merged data is serialized to obtain the serialized second merged data.

[0195] It is understood that data serialization is the process of converting a data structure into a storable or transmittable format. It is typically used to convert complex data (such as arrays) into readable byte streams or text formats (such as JSON or XML) for storage in files or transmission over a network. Therefore, in this embodiment, the second merged data can be serialized to convert its data structure into a storable or transmittable data format, resulting in serialized second merged data. It is understood that persistent processing of the second merged data can reduce data storage size and facilitate data storage and transmission operations.

[0196] In some embodiments, 704 is optional, that is, the feature cleaning module 403 may not perform serialization processing on the second merged data after obtaining the second merged data.

[0197] 705: Persist the serialized second merged data to obtain the third merged data.

[0198] It is understood that data persistence is the process of saving data to a permanent storage medium (such as a database, file system, or cloud storage) to ensure that the data remains available after the program ends or the system restarts, preventing data loss. Therefore, in this embodiment, the serialized second merged data can be persisted, for example, by adding the second merged data to a database or creating a file to obtain the third merged data.

[0199] Figure 8 A schematic diagram of the workflow of a feature modeling module 405 is shown. Figure 8 As shown, the process includes:

[0200] 801: Obtain the third merged data, deserialize the third merged data, and obtain the fourth merged data.

[0201] In this embodiment of the application, if the feature cleaning module 403 performs serialization processing on the merged data, such as the process shown in 704 above, then in 801, the feature modeling module 405 can perform deserialization processing on the merged data that has undergone serialization processing (such as the third merged data mentioned above), that is, convert the byte stream or text format of the data back to the original data structure to obtain the fourth merged data, so as to facilitate the subsequent preprocessing operation of the merged data.

[0202] 802: Perform feature selection on the fourth merged data to obtain the feature-selected fourth merged data.

[0203] It is understood that feature selection is a process used to select the most relevant features to improve model performance and reduce overfitting. This application does not limit the specific method of feature selection; for example, feature selection can be performed through principal component analysis (PCA) or correlation analysis.

[0204] In the embodiments of this application, feature selection of data can remove redundant or irrelevant features in the fourth merged data, that is, help identify and remove noisy data, improve the overall quality of the dataset, thereby reducing model overfitting and improving the model's generalization ability.

[0205] 803: Perform feature transformation on the fourth merged data after feature selection to obtain the fourth merged data after feature transformation.

[0206] It is understood that feature transformation can convert original features into new features to improve model performance or adapt to model requirements. Feature transformation can improve model accuracy by changing features to make them more suitable for the model. This application does not limit the specific method of feature transformation; for example, feature transformation can be performed on the fourth merged data through normalization, standardization, logarithmic transformation, etc.

[0207] 804: The benchmark model is trained based on the fourth merged data after feature transformation to obtain the trained benchmark model.

[0208] This application does not limit the type of benchmark model, as long as it can generate a predictive load. For example, the benchmark model can be any of the following: linear regression model, decision tree model, convolutional neural network model, recurrent neural network model, or transformer model. Furthermore, the training process of the benchmark model has been described in detail in section 204 above and will not be repeated here.

[0209] 805: Predict load based on a trained benchmark model.

[0210] It is understandable that after obtaining the trained benchmark model, the load of the current round can be predicted based on the benchmark model. For example, the first feature threshold mentioned above can be input into the benchmark model to obtain the first load.

[0211] like Figure 9 As shown, a benchmarking method includes the following scheme:

[0212] 901: In each round of benchmark testing, the load is linearly accumulated, and the characteristic values ​​of the load running characteristics corresponding to each round of benchmark testing are obtained.

[0213] Understandably, this benchmarking method automatically sets the load for each round of benchmarking, and the load in each round increases linearly compared to the previous round. For example, each round of benchmarking can increase the load by a fixed number based on the load of the previous round. Taking website backend programs as an example, the first round of benchmarking can run ten website backend programs, the second round can run twenty, the third round can run thirty, and so on. That is, in this example, each round of benchmarking increases the number of website backend programs by ten based on the number of website backend programs in the previous round.

[0214] 902: Determine whether the characteristic value of the load operation characteristics has reached the preset threshold.

[0215] It is understandable that if the characteristic value of the load operation feature during a certain load operation reaches the preset threshold, then 903 is executed, the benchmark test ends, and the benchmark test results are generated. If the characteristic value of the load operation feature does not reach the preset threshold, then 901 is executed again, and the load continues to be linearly accumulated in the next round of benchmark testing until the characteristic value of the load operation feature reaches the preset threshold.

[0216] Taking a website backend program as the load balancing mechanism, the load balancing characteristic as the number of concurrent requests processed, and a preset threshold of 200 as an example, if ten website backend programs are run in the first round of benchmark testing, and the number of concurrent requests processed under this condition is 100 (i.e., the characteristic value of the load balancing characteristic is 100), then the characteristic value of the load balancing characteristic has not reached the preset threshold of 200, and the second round of benchmark testing can continue. If twenty website backend programs are run in the second round of benchmark testing, and the number of concurrent requests processed under this condition is 200, then the characteristic value of the load balancing characteristic has reached the preset threshold of 200, a 903 error is executed, the benchmark test ends, and the benchmark test results are generated.

[0217] 903: End the benchmark test and generate benchmark results.

[0218] Continuing with the example from section 902, when the second round of benchmark testing runs twenty website backend programs, the characteristic value of the load operation features reaches a preset threshold. At this point, the benchmark test results can be obtained based on the characteristic values ​​of the load operation features of the twenty website backend programs. For example, the number of query requests that the twenty website backend programs can process per unit of time can be used as the benchmark test result.

[0219] However, if the number of concurrent requests handled by the tested object reaches the preset threshold of 200 when there are 15 backend programs on the website, then the benchmark test results corresponding to the 20 backend programs determined by the above method do not represent the benchmark test results under the actual load capacity limit of the tested object, resulting in low accuracy of the benchmark test results. In addition, the linearly accumulated load method makes the benchmark test time long, resulting in low efficiency of the benchmark test.

[0220] Compared to this approach, the benchmarking method provided in this application does not involve linearly accumulating loads for each round of benchmarking. Instead, the loads are predicted based on a benchmarking model. This method allows the benchmarking results to reflect the actual load capacity of the tested object, thus improving the accuracy of the benchmarking results. Furthermore, the method provided in this application, which predicts loads based on a benchmarking model, can shorten the benchmarking time and improve benchmarking efficiency.

[0221] In some embodiments, this application also provides a benchmark testing apparatus, which includes:

[0222] The first acquisition module is used to acquire the first feature threshold of the first load running characteristics when the load is running on the object under test.

[0223] The input module is used to input the first feature threshold into the first benchmark model during the Nth round of benchmark testing to obtain the first load of the tested object corresponding to the first load operating feature, where N is an integer greater than 1;

[0224] The second acquisition module is used to acquire the benchmark test results based on the first load when the change of the first load relative to the second load meets the test termination condition. The second load is obtained by inputting the first feature threshold into the second benchmark test model during the (N-1)th round of benchmark testing. The first benchmark test model is obtained by adjusting the second benchmark test model.

[0225] In some embodiments, the test termination condition includes: the absolute value of the difference between the first load amount corresponding to the first load and the second load amount corresponding to the second load is less than or equal to the difference threshold.

[0226] In some embodiments, the first benchmark model is obtained by: acquiring a first feature value of the first load running characteristics during the second load operation; adjusting the second benchmark model based on the difference between the first feature value and the first feature threshold to obtain the first benchmark model.

[0227] Therefore, the benchmark testing apparatus involved in this application may further include a runtime feature acquisition module, which is used to: acquire a first feature value of the runtime feature of the first load during the second load operation. The runtime feature acquisition module may, for example, include the service feature acquisition module 401 and / or the resource feature acquisition module 402 mentioned above.

[0228] Furthermore, the benchmarking apparatus may also include a feature modeling module, which is used to adjust a second benchmarking model based on the difference between a first feature value and a first feature threshold to obtain a first benchmarking model. The feature modeling module may, for example, be the feature modeling module 405 mentioned above.

[0229] In some embodiments, the feature modeling module is further configured to: adjust the first benchmark model to obtain a third benchmark model when the change of the first load relative to the second load does not meet the test termination condition.

[0230] In some embodiments, the feature modeling module is used to: adjust the first benchmark model based on the difference between the second feature value and the first feature threshold corresponding to the first load running feature during the first load running, to obtain the third benchmark model.

[0231] In some embodiments, the second acquisition module is configured to: acquire a third feature value of a second load running feature generated when the first load runs on the object under test; and use the third feature value as a benchmark test result, wherein the first load running feature and the second load running feature are the same or different.

[0232] It is understood that the second acquisition module may include the business feature acquisition module 401 and / or resource feature acquisition module 402 mentioned above. For example, the business feature acquisition module 401 and / or resource feature acquisition module 402 may acquire a third feature value (i.e., benchmark test result).

[0233] In some embodiments, the first load operation characteristic includes at least one of load service characteristics and device resource characteristics; the first characteristic threshold includes at least one of load service characteristic threshold and device resource characteristic threshold.

[0234] In some embodiments, the load includes a website backend program for information retrieval, the tested object includes a cluster, and the load business characteristics include at least one of the following: the number of query requests processed by the tested object per unit time, the number of concurrent processing requests processed, and the average response time of query requests; the device resource characteristics include at least one of the following: the memory occupied by the website backend program during runtime and the utilization rate of the central processing unit of the website backend program during runtime.

[0235] In some embodiments, the first benchmark model includes any one of a linear regression model, a decision tree model, a convolutional neural network model, a recurrent neural network model, and a transformer model.

[0236] It is understood that the beneficial effects of the benchmark testing device provided in this application can be found in the relevant description of the beneficial effects of the benchmark testing method above, and will not be repeated here.

[0237] In some embodiments, this application also provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the benchmark testing method described in the above embodiments.

[0238] In some embodiments, this application also provides a computer program product, including: computer instructions, which, when executed on an electronic device, cause the electronic device to perform the benchmark testing method described in the above embodiments.

[0239] It is understood that the beneficial effects of the electronic equipment and computer program products provided in this application can be found in the relevant description of the beneficial effects of the benchmark test method above, and will not be repeated here.

[0240] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Figure 10 This is a schematic diagram of an electronic device 1010 provided in an embodiment of this application. Figure 10 As shown, the electronic device 1010 includes a processor 102 and memory 104. The processor 102 is connected to the memory 104 via a double data rate (DDR) bus 103. Here, different memory modules 104 may use different data buses to communicate with the processor 102, therefore the DDR bus 103 can also be replaced with other types of data buses. This embodiment does not limit the bus type. In addition, the electronic device 1010 also includes various I / O devices 107, which the processor 102 can access via the peripheral component interconnect express (PCIe) bus 105.

[0241] Processor 102 is the computing and control core of electronic device 1010. Processor 102 includes one or more processor cores 204. Processor 102 runs an operating system and other software programs, thereby enabling processor 102 to access memory 104 and various PCIe devices. It is understood that in the embodiments of this application, processor 102 can be a CPU or other application-specific integrated circuits (ASICs). Processor 102 can also be other general-purpose processors, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In practical applications, electronic device 1010 may include only a single processor or multiple processors.

[0242] The memory controller is a bus circuit controller within the electronic device 1010 that controls the memory 104 and manages and plans the data transfer between the memory 104 and the Core 204. Data can be exchanged between the memory 104 and the Core 204 through the memory controller. The memory controller can be a separate chip connected to the Core 204 via the system bus. The memory controller can also be integrated into the processor 102, built into the northbridge, or be a separate memory controller chip. This application embodiment does not limit the specific location or form of the memory controller. In practical applications, the memory controller controls the necessary logic to write data to or read data from the memory 104.

[0243] Memory 104 is the main memory of electronic device 1010. Memory 104 is typically used to store various running software in the operating system, input and output data, and information exchanged with external storage. To improve the access speed of processor 102, memory 104 needs to have the advantage of high access speed. In traditional computer system architectures, dynamic random access memory (DRAM) is usually used as memory 104. Processor 102 can access memory 104 at high speed through the memory controller, performing read and write operations on any storage unit in memory 104. In addition to DRAM, memory 104 can also be other random access memory, such as static random access memory (SRAM). Alternatively, memory 104 can also be read-only memory (ROM). For example, read-only memory can be programmable read-only memory (PROM) or erasable programmable read-only memory (EPROM). This embodiment does not limit the number or type of memory 104. Furthermore, memory 104 can be configured to have a power-saving function. The power-saving function means that data stored in the memory will not be lost after a power outage and subsequent power restoration. Memory 104 with a power-saving function is called non-volatile memory.

[0244] Input / output (I / O) devices 107 refer to hardware capable of data transmission; they can also be understood as devices that interface with I / O interfaces. Common I / O devices include network cards, printers, keyboards, and mice. All external storage devices can also be used as I / O devices, such as hard drives, floppy disks, and optical discs. The processor 102 can access each I / O device 107 via the PCIe bus 105. It should be noted that the PCIe bus 105 is just one example and can be replaced with other buses, such as the unified bus (UB) bus.

[0245] The baseboard management controller (BMC) 106 is used for firmware upgrades, managing the device's operating status, and troubleshooting. The processor 102 accesses the baseboard management controller 106 via a PCIe bus or a bus such as USB or I2C. The baseboard management controller 106 is also connected to at least one sensor to acquire status data of the computer device. This status data includes temperature data, current data, voltage data, etc. No specific limitations are made on the type of status data in this application. The baseboard management controller 106 communicates with the processor 102 via a PCIe bus or other types of buses, for example, by transmitting the acquired status data to the processor 102 for processing. The baseboard management controller 106 can also maintain the program code in the memory 102, including upgrading or restoring it. The baseboard management controller 106 can also control the power supply circuit or clock circuit within the electronic device 1010. In summary, the baseboard management controller 106 can manage the electronic device 1010 in the above ways. However, the baseboard management controller 106 is only an optional device. In some implementations, the baseboard management controller 106 may be omitted, and the processor 102 may communicate directly with the sensors to directly manage and maintain the electronic equipment.

[0246] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0247] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0248] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0249] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A method of benchmarking, characterized by, The method comprises: obtaining a first feature threshold of a first load running feature when a load runs on a measured object; in an Nth round of benchmark testing, inputting the first feature threshold into a first benchmark testing model to obtain a first load corresponding to the first load running feature of the measured object, wherein N is an integer greater than 1; corresponding to the change of the first load relative to a second load satisfying a test end condition, obtaining a benchmark test result based on the first load, wherein the second load is based on inputting the first feature threshold into a second benchmark testing model in an (N-1)th round of benchmark testing, and the first benchmark testing model is obtained based on adjusting the second benchmark testing model.

2. The method of claim 1, wherein, The test end condition comprises: the absolute value of the difference between the first load amount corresponding to the first load and the second load amount corresponding to the second load is less than or equal to a difference threshold.

3. The method of claim 1, wherein, The first benchmark testing model is obtained based on the following manner: obtaining a first feature value of the first load running feature when the second load runs; adjusting the second benchmark testing model based on the difference between the first feature value and the first feature threshold to obtain the first benchmark testing model.

4. The method of claim 1, wherein, Corresponding to the change of the first load relative to the second load not satisfying the test end condition, adjusting the first benchmark testing model to obtain a third benchmark testing model.

5. The method of claim 4, wherein, The adjusting of the first benchmark testing model to obtain the third benchmark testing model comprises: adjusting the first benchmark testing model based on the difference between the second feature value corresponding to the first load running feature when the first load runs and the first feature threshold to obtain the third benchmark testing model.

6. The method of claim 1, wherein, The corresponding to the change of the first load relative to the second load satisfying the test end condition, obtaining a benchmark test result based on the first load, comprises: obtaining a third feature value of a second load running feature generated when the first load runs on the measured object; taking the third feature value as a benchmark test result, wherein the first load running feature and the second load running feature are the same or different.

7. The method of claim 1, wherein, The first load running feature comprises at least one of a load service feature and a device resource feature; The first feature threshold comprises at least one of a load service feature threshold and a device resource feature threshold.

8. The method of claim 7, wherein, The load comprises a website background program for information query, the measured object comprises a cluster, and The load service feature comprises at least one of the number of query requests processed per unit time, the number of concurrent processing requests processed, and the average response time of query requests of the measured object; The device resource feature comprises at least one of the memory occupied by the website background program when running and the utilization rate of the central processing unit when the website background program runs.

9. The method of claim 1, wherein, The first benchmark testing model comprises any one of a linear regression model, a decision tree model, a convolutional neural network model, a recurrent neural network model, and a transformer model.

10. A benchmarking device characterized by The benchmark testing device comprises: The first obtaining module is configured to obtain a first feature threshold of a first load running feature when a load is running on the to-be-tested object. The input module is configured to input the first feature threshold into a first benchmark test model in an Nth round of benchmark test to obtain a first load corresponding to the first load running feature of the to-be-tested object, where N is an integer greater than 1. The second obtaining module is configured to obtain a benchmark test result based on the first load when a change of the first load relative to a second load satisfies a test end condition, where the second load is obtained by inputting the first feature threshold into a second benchmark test model in an (N-1)th round of benchmark test, and the first benchmark test model is obtained by adjusting the second benchmark test model.

11. The apparatus of claim 10, wherein, The test end condition includes that an absolute value of a difference between a first load amount corresponding to the first load and a second load amount corresponding to the second load is less than or equal to a difference threshold.

12. The apparatus of claim 10, wherein, The first benchmark test model is obtained in the following manner: obtaining a first feature value of the first load running feature when the second load is running; adjusting the second benchmark test model based on a difference between the first feature value and the first feature threshold to obtain the first benchmark test model.

13. An electronic device, comprising: The electronic device comprises: one or more processors; one or more memories storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the benchmark test method in any one of claims 1 to 9.

14. A computer program product, characterised in that, The computer instructions, when executed on the electronic device, cause the electronic device to perform the benchmark test method in any one of claims 1 to 9. ​

Images