A method and device for testing machine performance

Abstract

The present invention discloses a method and device for testing machine performance, and relates to the field of computer technology. A specific implementation of the method includes: repeatedly performing performance tests on a machine to obtain multiple target test results; calculating the multiple target test results based on a performance benchmark value, and judging the performance of the machine according to the calculation results; wherein the performance benchmark value is obtained by the following method: repeatedly performing performance tests on the machine to obtain multiple benchmark test results, thereby calculating the performance benchmark value. This implementation can solve the technical problem of a large number of false alarms.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for testing machine performance. Background Art

[0002] For client-server architecture systems, the performance of the server interface affects the response time of user operations and the system throughput per unit time (QPS, Query Per Second). In order to monitor whether the server interface has performance degradation as the system evolves, performance testing is usually performed regularly. If the tool ecosystem is relatively complete, performance testing will be automated and run every week or even every day.

[0003] This type of performance test usually records the maximum throughput per second of a single machine, that is, the throughput per second of the machine when the machine's CPU, memory and other indicators reach a relatively high critical state. Specifically, the existing method is: first record the maximum throughput per second for a period of time, take an average as a benchmark, and then automatically perform tests regularly. If the maximum throughput per second is found to have decreased, it indicates that the performance has decreased. If the decrease reaches a set threshold value, an alarm will be issued to notify R&D and test personnel. R&D and test personnel will confirm and check whether there is indeed a decrease. If there is indeed a performance decrease, then check the cause and solve it.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] There are a lot of false alarms when judging whether the performance has degraded by setting threshold values, which will waste a lot of time of R&D personnel to investigate. Moreover, when the number of false alarms increases, R&D personnel will pay less attention to the alarms and may miss the real performance degradation. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method and device for testing machine performance to solve the technical problem of a large number of false alarms.

[0007] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for testing machine performance is provided, comprising:

[0008] Repeat the performance test on the machine to obtain multiple target test results;

[0009] Based on the central limit theorem, the multiple target test results are calculated using a performance benchmark value, and the performance of the machine is determined according to the calculation results;

[0010] The performance benchmark value is obtained by the following method:

[0011] The performance test is repeatedly performed on the machine to obtain a plurality of benchmark test results, thereby calculating a performance benchmark value.

[0012] Optionally, the performance test is repeatedly performed on the machine to obtain multiple benchmark test results, thereby calculating a performance benchmark value, including:

[0013] Repeat the performance test on the machine to obtain multiple benchmark results;

[0014] According to the multiple test results, a performance mean and a performance standard deviation are calculated.

[0015] Optionally, the number of the benchmark test results is more than twice the number of the target test results, and the number of the target test results is greater than or equal to 30.

[0016] Optionally, based on the central limit theorem, the multiple target test results are calculated by using a performance benchmark value, and the performance of the machine is judged according to the calculation results, including:

[0017] Calculate the performance threshold according to the preset performance probability threshold, the performance mean, the performance standard deviation and the number of target test results;

[0018] The average of the multiple target test results is calculated, and the performance threshold is compared with the average of the multiple target test results to determine the performance of the machine.

[0019] Optionally, calculating the performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation, and the number of target test results includes:

[0020] Subtract the preset performance probability threshold from 1 to obtain the cumulative probability;

[0021] Mapping the cumulative probability on a normal distribution graph to obtain multiples of the standard deviation;

[0022] The performance threshold is calculated according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of target test results.

[0023] Optionally, calculating the performance threshold according to the multiple of the standard deviation, the performance mean and the performance standard deviation includes:

[0024] Solving the square root of the number of target test results;

[0025] Multiply the performance mean by a multiple of the standard deviation and divide the result by the square root of the target test result to obtain a performance difference;

[0026] The performance difference is subtracted from the performance mean to obtain a performance threshold.

[0027] Optionally, comparing the performance threshold with the average of the plurality of target test results to determine the performance of the machine includes:

[0028] If the average of the multiple target test results is less than the performance threshold, an alarm is issued to indicate that the performance of the machine has deteriorated.

[0029] In addition, according to another aspect of an embodiment of the present invention, there is provided a device for testing machine performance, comprising:

[0030] The test module is used to repeatedly perform performance tests on the machine to obtain multiple target test results;

[0031] A judgment module, configured to calculate the plurality of target test results by using a performance benchmark value based on the central limit theorem, and judge the performance of the machine according to the calculation results;

[0032] The performance benchmark value is obtained by the following method:

[0033] The performance test is repeatedly performed on the machine to obtain a plurality of benchmark test results, thereby calculating a performance benchmark value.

[0034] Optionally, the performance test is repeatedly performed on the machine to obtain multiple benchmark test results, thereby calculating a performance benchmark value, including:

[0035] Repeat the performance test on the machine to obtain multiple benchmark results;

[0036] According to the multiple test results, a performance mean and a performance standard deviation are calculated.

[0037] Optionally, the number of the benchmark test results is more than twice the number of the target test results, and the number of the target test results is greater than or equal to 30.

[0038] Optionally, the judging module is further used for:

[0039] Calculate the performance threshold according to the preset performance probability threshold, the performance mean, the performance standard deviation and the number of target test results;

[0040] The average of the multiple target test results is calculated, and the performance threshold is compared with the average of the multiple target test results to determine the performance of the machine.

[0041] Optionally, the judging module is further used for:

[0042] Subtract the preset performance probability threshold from 1 to obtain the cumulative probability;

[0043] Mapping the cumulative probability on a normal distribution graph to obtain multiples of the standard deviation;

[0044] The performance threshold is calculated according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of target test results.

[0045] Optionally, the judging module is further used for:

[0046] Solving the square root of the number of target test results;

[0047] Multiply the performance mean by a multiple of the standard deviation and divide the result by the square root of the target test result to obtain a performance difference;

[0048] The performance difference is subtracted from the performance mean to obtain a performance threshold.

[0049] Optionally, the judging module is further used for:

[0050] If the average of the multiple target test results is less than the performance threshold, an alarm is issued to indicate that the performance of the machine has deteriorated.

[0051] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including:

[0052] one or more processors;

[0053] a storage device for storing one or more programs,

[0054] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.

[0055] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0056] One embodiment of the above invention has the following advantages or beneficial effects: because a technical means based on the central limit theorem is used to calculate multiple target test results through performance benchmark values, and the performance of the machine is judged according to the calculation results, the technical problem of a large number of false alarms in the prior art is overcome. The embodiment of the present invention accurately judges the performance of the machine based on the central limit theorem, which not only improves the alarm accuracy of the automated performance test, but also improves the sensitivity of performance monitoring, and can effectively monitor a small degree of performance degradation.

[0057] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0059] Figure 1 is a schematic diagram of the main process of the method for testing machine performance according to an embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of the main process of a method for testing machine performance according to a reference embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of the main process of a method for testing machine performance according to another reference embodiment of the present invention;

[0062] Figure 4 is a schematic diagram of main modules of an apparatus for testing machine performance according to an embodiment of the present invention;

[0063] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0064] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0066] In order to determine whether there is performance degradation, existing automated performance tests usually set a threshold value. If a certain run does not reach the threshold value, an alarm will be triggered, indicating that performance may have degraded. However, the machine condition is different each time the test is run. When the machine load is light, the performance test results will be better, while when the machine load is heavy, the performance test results will be worse. It is very likely that the test result did not reach the threshold value because of the machine condition. If an alarm is triggered every time the threshold value is not reached, there will be a large number of false alarms, which will waste a lot of time for R&D personnel to investigate. Moreover, after a large number of false alarms, R&D personnel will pay less attention to the alarms and may miss the real performance degradation.

[0067] In order to reduce the problem of false alarms, some systems will adjust the threshold value so that an alarm will only be triggered if a single test result shows a significant performance degradation. However, this method can only reduce false alarms to a small extent and cannot truly solve the problem of false alarms. Moreover, it can only detect a significant degree of performance degradation, so it cannot solve the problem, but the result will be worse.

[0068] Figure 1 Schematic diagram of the main process of the method for testing machine performance according to an embodiment of the present invention. As an embodiment of the present invention, Figure 1 As shown, the method for testing machine performance may include:

[0069] Step 101, repeatedly perform performance testing on the machine to obtain multiple benchmark test results, thereby calculating a performance benchmark value.

[0070] First, the performance test is repeated N times on the machine, and the test results of each time are recorded (such as performance parameters such as throughput per second QPS), and then the performance benchmark value of the machine is calculated based on these test results. In the embodiment of the present invention, different virtual machines with the same configuration can be selected to run the test, so that the distribution of the test results conforms to the actual situation as much as possible.

[0071] Optionally, step 101 may include: repeatedly performing performance tests on the machine to obtain multiple benchmark test results; and calculating the performance mean and performance standard deviation based on the multiple test results. For example, the performance test may be repeatedly performed on the machine 1000 times, and each test result (i.e., benchmark test result) may be recorded, and then the test results may be fitted using Apache's Common math library (an Apache library that can perform statistical and probability-related calculations), and it may be found that it is relatively well in line with the normal distribution, and the performance mean m and performance standard deviation μ of the machine may be calculated.

[0072] The running code is as follows:

[0073]

[0074] It should be pointed out that the embodiment of the present invention has no limit on the number of executions N of the performance test, which can be executed 500 times, 2000 times or 3200 times, etc. The tester can determine the number of executions according to actual needs, and then calculate the performance mean m and performance standard deviation μ based on the test results of each time.

[0075] Step 102, repeatedly perform performance testing on the machine to obtain multiple target test results.

[0076] After obtaining the performance benchmark value (including the performance mean m and the performance standard deviation μ), the performance test is repeated on the machine with the same configuration to obtain multiple target test results. Optionally, the number of benchmark test results (N) is more than twice the number of target test results (n), and the number of target test results is greater than or equal to 30.

[0077] From a statistical point of view, when the sample size n is greater than 30, the sample mean is more perfect (m, ) is normally distributed. If conditions do not allow execution 30 times a day, the results of several days can also be accumulated and calculated after reaching 30 times. Optionally, each time the test is executed, the test results are written into a text file in an append manner. When the file reaches n lines, it indicates that n performance tests have been executed. At this time, step 103 is entered to determine whether the performance has declined and clear the file. Optionally, n can be 30, 35, 40, 48 or 50, etc., and the embodiment of the present invention does not limit this.

[0078] Step 103, based on the central limit theorem, the plurality of target test results are calculated using a performance benchmark value, and the performance of the machine is determined according to the calculation results.

[0079] After executing n performance tests on the machine, the n performance tests are calculated based on the performance benchmark value obtained in step 101, and the performance of the machine is judged according to the calculation results, thereby determining whether an alarm is needed.

[0080] Optionally, step 103 may include: calculating a performance threshold based on a preset performance probability threshold, the performance mean, the performance standard deviation and the number of target test results; calculating the mean of the multiple target test results, comparing the performance threshold with the mean of the multiple target test results, thereby judging the performance of the machine.

[0081] According to the central limit theorem, the sampling distribution of the sample mean is the distribution formed by all sample means. The sampling distribution of the sample mean is symmetrical in shape. As the sample size n increases, regardless of whether the original population obeys the normal distribution, the sampling distribution of the sample mean will tend to the normal distribution. The expectation of its distribution is the population mean m, the variance is 1 / n of the population variance, and the standard deviation is the population standard deviation.

[0082] Optionally, the performance threshold is compared with the mean of the multiple target test results to determine the performance of the machine, including: if the mean of the multiple target test results is less than the performance threshold, an alarm is issued to indicate that the performance of the machine has deteriorated. Assume that the same performance test is run on a machine with the same configuration at different times, and the test results conform to the normal distribution with an expectation of m and a standard deviation of μ. On this basis, according to the central limit theorem, if the same test is performed n times a day, the mean of these n test results obeys the expectation of m and the standard deviation of μ. Normal distribution. After executing n times a day, the mean of these n test results is calculated, and the cumulative probability of this mean in this normal distribution can be calculated. If this probability is less than a certain set value, an alarm is issued.

[0083] Optionally, the performance threshold is calculated based on a preset performance probability threshold, the performance mean, the performance standard deviation, and the number of target test results, including: subtracting the preset performance probability threshold from 1 to obtain the cumulative probability; mapping the cumulative probability on a normal distribution graph to obtain multiples of the standard deviation; and calculating the performance threshold based on the multiples of the standard deviation, the performance mean, the performance standard deviation, and the number of target test results. Optionally, the performance threshold is calculated based on the multiples of the standard deviation, the performance mean, and the performance standard deviation, including: solving the square root of the number of target test results; multiplying the performance mean by the multiple of the standard deviation and dividing the result by the square root of the target test result to obtain a performance difference; and subtracting the performance difference from the performance mean to obtain a performance threshold.

[0084] First, read the text file in step 102 to obtain n test results, and then calculate the mean m2 of the n results. Since the mean meets (m, ), we can calculate the cumulative probability of m2. The running code is as follows:

[0085] NormalDistribution normalDistribution=new NormalDistribution(m,miu / Math.sqrt(30));

[0086] double cumn=normalDistribution.cumulativeProbability(m2);

[0087] The performance threshold can be calculated using the following formula:

[0088]

[0089] Where gateNum is the performance threshold, m is the performance mean, x is the multiple of the standard deviation obtained by mapping the cumulative probability on the normal distribution graph, μ is the performance standard deviation, and n is the number of target test results.

[0090] Therefore, a new probability distribution is constructed with an expectation of m and a standard deviation of Next, you need to set a performance probability threshold. An alarm will be triggered only when the probability of performance degradation reaches this value.

[0091] Assuming that the set performance probability threshold is gate, the performance threshold gateNum can be calculated by the following program: normalDistribution.inverseCumulativeProbability(1-gate). If the actual mean (m2) is less than this value, an alarm is issued.

[0092] Assume that m is 1000, μ is 100, and the performance probability threshold is set to 84.1%, which means that an alarm will be triggered if there is an 84.1% probability of performance degradation. The corresponding cumulative probability is 1-0.841=0.159. Since the value corresponding to the cumulative probability of 0.159 on the normal distribution graph is 1 times the standard deviation, if the mean of n times is less than This means that there is an 84.1% probability of performance degradation, and an alarm should be sounded at this time. It should be pointed out that 84.1% as the performance probability threshold is only an example, and the actual system does not necessarily use this number, and it can be set flexibly.

[0093] Subsequently, step 102 and step 103 can be executed periodically, that is, the performance test is performed periodically and the test results of each time are recorded. When the recorded results reach n times, the average of the n test results is calculated. If this number is less than the performance threshold gateNum calculated previously, it indicates that the probability of performance degradation exceeds the gate and an alarm is required.

[0094] The embodiment of the present invention uses the central limit theorem to determine whether the result of the automated performance test indicates performance degradation, which has the following advantages:

[0095] 1) The alarm accuracy is greatly improved. Assume that the mean of 30 data is The corresponding probability of performance degradation is 84.1%. If the existing method is used to calculate the result of each performance test, the probability of performance degradation corresponding to this number is 57.2%, which shows that the embodiment of the present invention significantly improves the alarm accuracy.

[0096] 2) The sensitivity of monitoring performance degradation is greatly improved. With the existing method, if you want to alarm when the probability of performance degradation reaches 90%, the alarm can only be issued when the test result is less than 1.28 standard deviations. However, with the method provided in the embodiment of the present invention, the alarm can be issued only when the test result is less than 0.23 standard deviations, and the sensitivity is greatly improved.

[0097] According to the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problem of a large number of false alarms in the prior art by calculating multiple target test results based on the central limit theorem and the performance benchmark value, and judging the performance of the machine based on the calculation results. The embodiments of the present invention accurately judge the performance of the machine based on the central limit theorem, which not only improves the alarm accuracy of the automated performance test, but also improves the sensitivity of the performance monitoring, and can effectively monitor a small degree of performance degradation.

[0098] Figure 2 Schematic diagram of the main process of a method for testing machine performance according to a reference embodiment of the present invention. As another embodiment of the present invention, Figure 2 As shown, the method for testing machine performance may include:

[0099] Step 201, repeatedly perform N performance tests on the machine to obtain N benchmark test results.

[0100] First, repeat the performance test N times on the machine and record the test results each time. You can choose different virtual machines with the same configuration to run the test, so that the distribution of the test results is as consistent as possible with the actual situation.

[0101] Step 202: Calculate the performance mean and performance standard deviation based on the N test results.

[0102] Apache's Common math library can be used to fit the test results to calculate the machine's performance mean m and performance standard deviation μ.

[0103] Step 203, calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation and the number n of target test results.

[0104] Specifically, 1 is subtracted from a preset performance probability threshold to obtain a cumulative probability; the cumulative probability is mapped on a normal distribution graph to obtain a multiple of the standard deviation; the performance threshold is calculated based on the multiple of the standard deviation, the performance mean, the performance standard deviation, and the number of target test results. Optionally, the performance threshold can be calculated using the following formula:

[0105]

[0106] Where gateNum is the performance threshold, m is the performance mean, x is the multiple of the standard deviation obtained by mapping the cumulative probability on the normal distribution graph, μ is the performance standard deviation, and n is the number of target test results. Assume that the same performance test is run on the same machine at different times, and the test results conform to the normal distribution, with an expectation of m and a standard deviation of μ. On this basis, according to the central limit theorem, if the same test is performed n times a day, the mean of these n test results follows the expectation of m and the standard deviation of Normal distribution. After executing n times a day, the mean of these n test results is calculated, and the cumulative probability of this mean in this normal distribution can be calculated. If this probability is less than a certain set value, an alarm is issued.

[0107] Step 204, repeatedly perform the performance test on the machine n times to obtain n target test results.

[0108] Repeat the performance test on the machine n times at a regular interval. Each time the test is performed, the test results are written into a text file in an append manner. When the file reaches n lines, it indicates that n performance tests have been performed. At this time, the next step is entered to determine whether the performance has degraded and clear the file. The number of benchmark test results (N) is more than twice the number of target test results (n), and the number of target test results is greater than or equal to 30.

[0109] Step 205, calculating the mean m2 of the n target test results.

[0110] Step 206, comparing the performance threshold gateNum with the mean m2 of the plurality of target test results, so as to determine the performance of the machine.

[0111] Subsequently, step 204, step 205 and step 206 can be executed periodically, that is, the performance test is performed periodically and the test results of each time are recorded. When the recorded results reach n times, the average of these n test results is calculated. If this number is less than the performance threshold gateNum calculated previously, it indicates that the probability of performance degradation exceeds the gate and an alarm is required.

[0112] In addition, the specific implementation content of the method for testing machine performance in a reference embodiment of the present invention has been described in detail in the method for testing machine performance described above, so the repeated content will not be described here.

[0113] Figure 3 Schematic diagram of the main process of a method for testing machine performance according to another reference embodiment of the present invention. As another embodiment of the present invention, Figure 3 As shown, the method for testing machine performance may include:

[0114] Step 301, preset a performance probability threshold and the number of times n to repeat the performance test.

[0115] Step 302: Subtract a preset performance probability threshold from 1 to obtain a cumulative probability.

[0116] Step 303: Mapping the cumulative probability on a normal distribution graph to obtain multiples of the standard deviation.

[0117] Step 304, calculating a performance threshold according to the multiple of the standard deviation, the performance mean of the machine, the performance standard deviation of the machine, and the number of times n that the performance test is repeated.

[0118] The mean performance value and standard deviation performance value of a machine can be obtained by the following method: First, repeat the performance test on the machine N times and record the test results each time. You can choose different virtual machines with the same configuration to run the test, so that the distribution of the test results is as consistent as possible with the actual situation. Then, you can use Apache's Common math library to fit the test results to calculate the mean performance value m and standard deviation performance value μ of the machine.

[0119] Finally, the performance threshold is calculated using the following formula:

[0120]

[0121] Where gateNum is the performance threshold, m is the performance mean, x is the multiple of the standard deviation obtained by mapping the cumulative probability on the normal distribution graph, μ is the performance standard deviation, and n is the number of target test results.

[0122] Assuming that the set performance probability threshold is gate, the performance threshold gateNum can be calculated by the following program: normalDistribution.inverseCumulativeProbability(1-gate).

[0123] Optionally, the number of the benchmark test results (N) is more than twice the number of the target test results (n).

[0124] Step 305, repeatedly perform the performance test on the machine n times to obtain n target test results.

[0125] Optionally, the number of target test results is greater than or equal to 30. Statistically, when the sample size n is greater than 30, the sample mean is perfectly consistent with (m, ) is normally distributed. If conditions do not allow execution 30 times a day, the results of several days can also be accumulated and calculated after reaching 30 times. Optionally, each time the test is executed, the test results are written into a text file in an append manner. When the file reaches n lines, it indicates that n performance tests have been executed. At this time, step 103 is entered to determine whether the performance has declined and clear the file. Optionally, n can be 30, 38, 42 or 55, etc., and the embodiment of the present invention does not limit this.

[0126] Step 306, calculating the mean of the n target test results.

[0127] Step 307 , determining whether the mean of the n target test results is less than the performance threshold; if so, executing step 308 ; if not, executing step 305 .

[0128] Step 308, an alarm is issued to indicate that the performance of the machine has degraded.

[0129] In addition, the specific implementation content of the method for testing machine performance in another reference embodiment of the present invention has been described in detail in the method for testing machine performance described above, so the repeated content will not be described here.

[0130] Figure 4 is a schematic diagram of the main modules of the device for testing machine performance according to an embodiment of the present invention, such as Figure 4 As shown, the device 400 for testing machine performance includes a test module 401 and a judgment module 402; wherein the test module 401 is used to repeatedly perform performance tests on the machine to obtain multiple target test results; the judgment module 402 is used to calculate the multiple target test results through performance benchmark values ​​based on the central limit theorem, and judge the performance of the machine according to the calculation results; wherein the performance benchmark value is obtained by the following method: repeatedly perform performance tests on the machine to obtain multiple benchmark test results, thereby calculating the performance benchmark value.

[0131] Optionally, the performance test is repeatedly performed on the machine to obtain multiple benchmark test results, thereby calculating a performance benchmark value, including:

[0132] Repeat the performance test on the machine to obtain multiple benchmark results;

[0133] According to the multiple test results, a performance mean and a performance standard deviation are calculated.

[0134] Optionally, the number of the benchmark test results is more than twice the number of the target test results, and the number of the target test results is greater than or equal to 30.

[0135] Optionally, the judging module 402 is further configured to:

[0136] Calculate the performance threshold according to the preset performance probability threshold, the performance mean, the performance standard deviation and the number of target test results;

[0137] The average of the multiple target test results is calculated, and the performance threshold is compared with the average of the multiple target test results to determine the performance of the machine.

[0138] Optionally, the judging module 402 is further configured to:

[0139] Subtract the preset performance probability threshold from 1 to obtain the cumulative probability;

[0140] Mapping the cumulative probability on a normal distribution graph to obtain multiples of the standard deviation;

[0141] The performance threshold is calculated according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of target test results.

[0142] Optionally, the judging module 402 is further configured to:

[0143] Solving the square root of the number of target test results;

[0144] Multiply the performance mean by a multiple of the standard deviation and divide the result by the square root of the target test result to obtain a performance difference;

[0145] The performance difference is subtracted from the performance mean to obtain a performance threshold.

[0146] Optionally, the judging module 402 is further configured to:

[0147] If the average of the multiple target test results is less than the performance threshold, an alarm is issued to indicate that the performance of the machine has deteriorated.

[0148] According to the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problem of a large number of false alarms in the prior art by calculating multiple target test results based on the central limit theorem and the performance benchmark value, and judging the performance of the machine based on the calculation results. The embodiments of the present invention accurately judge the performance of the machine based on the central limit theorem, which not only improves the alarm accuracy of the automated performance test, but also improves the sensitivity of the performance monitoring, and can effectively monitor a small degree of performance degradation.

[0149] It should be noted that the specific implementation content of the device for testing machine performance in the present invention has been described in detail in the method for testing machine performance described above, so the repeated content will not be described again here.

[0150] Figure 5 An exemplary system architecture 500 is shown to which the method for testing machine performance or the apparatus for testing machine performance according to the embodiments of the present invention can be applied.

[0151] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0152] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0153] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0154] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as the item information query request, and feed back the processing results (such as target push information, item information - only an example) to the terminal device.

[0155] It should be noted that the method for testing machine performance provided in the embodiment of the present invention is generally executed by the server 505, and accordingly, the apparatus for testing machine performance is generally set in the server 505. The method for testing machine performance provided in the embodiment of the present invention can also be executed by the terminal devices 501, 502, 503, and accordingly, the apparatus for testing machine performance can be set in the terminal devices 501, 502, 503.

[0156] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0157] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0158] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0159] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0160] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0161] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0162] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a test module and a judgment module, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases.

[0164] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device implements the following method: repeatedly performing performance tests on the machine to obtain multiple target test results; based on the central limit theorem, the multiple target test results are calculated by the performance benchmark value, and the performance of the machine is judged according to the calculation results; wherein the performance benchmark value is obtained by the following method: repeatedly performing performance tests on the machine to obtain multiple benchmark test results, thereby calculating the performance benchmark value.

[0165] According to the technical solution of the embodiment of the present invention, because a technical means based on the central limit theorem is adopted to calculate multiple target test results through performance benchmark values, and the performance of the machine is judged according to the calculation results, the technical problem of a large number of false alarms in the prior art is overcome. The embodiment of the present invention accurately judges the performance of the machine based on the central limit theorem, which not only improves the alarm accuracy of the automated performance test, but also improves the sensitivity of performance monitoring, and can effectively monitor a small degree of performance degradation.

[0166] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for testing machine performance, characterized in that: include: Repeat the performance test on the machine to obtain multiple target test results; Based on the central limit theorem, the multiple target test results are calculated using a performance benchmark value, and the performance of the machine is determined according to the calculation results; The performance benchmark value is obtained by the following method: Repeating the performance test on the machine to obtain multiple benchmark test results, thereby calculating a performance benchmark value; the performance benchmark value includes a performance mean and a performance standard deviation; Based on the central limit theorem, the multiple target test results are calculated by using the performance benchmark value, and the performance of the machine is judged according to the calculation results, including: Calculate the performance threshold according to the preset performance probability threshold, the performance mean, the performance standard deviation and the number of target test results; Calculating the average of the multiple target test results, comparing the performance threshold with the average of the multiple target test results, thereby determining the performance of the machine; Calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation, and the number of target test results includes: Subtract the preset performance probability threshold from 1 to obtain the cumulative probability; Mapping the cumulative probability on a normal distribution graph to obtain multiples of the standard deviation; The performance threshold is calculated according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of target test results.

2. The method according to claim 1, characterized in that Repeat the performance test on the machine to obtain multiple benchmark test results, and then calculate the performance benchmark value, including: Repeat the performance test on the machine to obtain multiple benchmark results; According to the multiple test results, a performance mean and a performance standard deviation are calculated.

3. The method according to claim 2, characterized in that The number of the benchmark test results is more than twice the number of the target test results, and the number of the target test results is greater than or equal to 30.

4. The method according to claim 1, characterized in that: Calculating a performance threshold according to the multiple of the standard deviation, the performance mean, and the performance standard deviation includes: Solving the square root of the number of target test results; The performance mean is multiplied by a multiple of the standard deviation and the result is divided by the square root of the target test result to obtain a performance difference; The performance difference is subtracted from the performance mean to obtain a performance threshold.

5. The method according to claim 1, characterized in that: Comparing the performance threshold with the average of the plurality of target test results to determine the performance of the machine includes: If the average of the multiple target test results is less than the performance threshold, an alarm is issued to indicate that the performance of the machine has deteriorated.

6. A device for testing machine performance, characterized in that: include: The test module is used to repeatedly perform performance tests on the machine to obtain multiple target test results; A judgment module, configured to calculate the plurality of target test results by using a performance benchmark value based on the central limit theorem, and judge the performance of the machine according to the calculation results; The performance benchmark value is obtained by the following method: Repeating the performance test on the machine to obtain multiple benchmark test results, thereby calculating a performance benchmark value; the performance benchmark value includes a performance mean and a performance standard deviation; The judging module is also used for: Calculate the performance threshold according to the preset performance probability threshold, the performance mean, the performance standard deviation and the number of target test results; Calculating the average of the multiple target test results, comparing the performance threshold with the average of the multiple target test results, thereby determining the performance of the machine; The judging module is also used for: Subtract the preset performance probability threshold from 1 to obtain the cumulative probability; Mapping the cumulative probability on a normal distribution graph to obtain multiples of the standard deviation; The performance threshold is calculated according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of target test results.

7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images