A method, device and equipment for evaluating system performance based on a test platform

Abstract

The present invention discloses a method, device and equipment for evaluating system performance based on a test platform, the method comprising: executing a use case set through an automated test platform to trigger the operation of a system program; monitoring the operation process of the system program through a performance monitoring tool to obtain the execution efficiency of the relevant structured query language SQL in the use case set; after the execution of a single use case in the use case set is completed, judging whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline; if not, judging whether the SQL performance of the single use case meets the performance index; determining the SQL performance result according to the judgment result; and generating a system performance report according to the SQL performance result. The technical problem that the system test result is only valid once and there are false reports is solved.

Description

Technical Field

[0001] The present invention relates to the field of system testing, and in particular to a method, device and equipment for evaluating system performance based on a testing platform. Background Art

[0002] In order to ensure that the system functions meet the performance requirements and discover performance bottlenecks in advance, it is necessary to analyze and evaluate the system information performance during the system development phase. At the same time, testers also need to accept the performance of the functions. In addition, in the face of frequent changes in system functions, in order to more accurately analyze the performance changes of the system, it is necessary to establish a system performance indicator baseline to support developers and testers to accurately obtain the system performance results during the verification process.

[0003] Currently, common efficiency monitoring tools deploy tools to system services. When the system is running, they analyze and present the system code call chain in real time and display performance problem points based on the running time.

[0004] Common application scenarios include performance analysis in the development phase and performance monitoring of production operations. Performance reports need to be analyzed and interpreted by developers or operation and maintenance personnel for targeted troubleshooting.

[0005] Disadvantages of existing solutions:

[0006] 1. The results are only valid for the current time, and there is no comparison with previous versions. When the current tool is running, it collects and analyzes the operating efficiency in real time, and the final presentation is the performance of the current time. For historical function modifications, it is impossible to conduct further evaluation based on the previous performance of the function;

[0007] 2. There are false positives. As mentioned above, existing tools display real-time analysis results, and the system will only run once. However, there are many factors that may affect poor performance. It is inaccurate to judge performance abnormality based on only one performance. It takes multiple analyses to determine that the performance indicators have not been met before it can be determined as a performance problem. Summary of the invention

[0008] The present invention provides a method, device and equipment for evaluating system performance based on a test platform, which solves the technical problem that the system test result is only valid once and there are false alarms.

[0009] A method for evaluating system performance based on a test platform, comprising:

[0010] Execute the use case set through the automated testing platform to trigger the running of the system program;

[0011] Monitor the running process of the system program through a performance monitoring tool to obtain the execution efficiency of the relevant structured query language SQL in the use case set;

[0012] After the execution of a single use case in the use case set is completed, determining whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline;

[0013] If not, determine whether the SQL performance of the single use case meets the performance indicator;

[0014] Determine the SQL performance result based on the judgment result;

[0015] Generate a system performance report based on the SQL performance results.

[0016] In one embodiment of the present invention, the monitoring of the running process of the system program specifically includes: tracking the stack information of the system program during running; monitoring time and memory information; and associating and recording the running efficiency of SQL scripts related to the use case.

[0017] In one embodiment of the present invention, the determination of whether the execution efficiency of the relevant SQL within the single use case meets the preset performance baseline specifically includes: determining that the SQL script related to the single use case is executed for the first time, then adding the SQL performance baseline of the single use case according to the single use case and the SQL identifier; determining that the SQL related to the single use case already has a performance baseline, then comparing the SQL performance of the single use case with the preset performance baseline, and performing subsequent actions according to the comparison result.

[0018] In one embodiment of the present invention, the SQL performance of the single use case is compared with a preset performance baseline, and subsequent actions are performed according to the comparison result, specifically including: determining that the SQL performance of the single use case is lower than the preset performance baseline, then judging whether the single use case is a performance problem, and if not, recording it in the performance risk use case set; determining that the SQL performance of the single use case is not lower than the preset performance baseline, then judging whether the performance of the SQL meets the performance indicators.

[0019] In one embodiment of the present invention, the determining whether the single use case SQL performance meets the performance indicator specifically includes: if it is determined that the SQL performance meets the performance indicator, the performance indicator is entered into the performance report to determine that the performance evaluation passes; if it is determined that the SQL performance does not meet the performance indicator, the SQL is recorded in the performance risk use case set.

[0020] In one embodiment of the present invention, the method further includes: after the automated testing platform has completed running all use cases, scheduling the performance risk use case set to be run again for evaluation according to evaluation requirements; the evaluation requirement is that if the use case is executed for a preset number of times and still does not meet preset conditions, it is determined to be a performance problem.

[0021] In one embodiment of the present invention, the method further includes: after the number of use case executions reaches a preset number, excluding use cases that meet the performance requirements; determining whether the performance risk has turned into a performance problem through a performance monitoring tool, and recording the performance problem in a performance report.

[0022] A device for evaluating system performance based on a test platform, comprising:

[0023] Automated testing platform, used to execute use case sets and trigger the running of system programs;

[0024] A performance monitoring tool is used to monitor the running process of the system program and obtain the execution efficiency of the relevant structured query language SQL in the use case set; after the execution of a single use case in the use case set is completed, determine whether the relevant SQL execution efficiency in the single use case meets the preset performance baseline; if not, determine whether the SQL performance of the single use case meets the performance indicator; determine the SQL performance result based on the judgment result; and generate a system performance report based on the SQL performance result.

[0025] A device for evaluating system performance based on a test platform, comprising:

[0026] at least one processor; and,

[0027] A memory connected to the at least one processor via a bus; wherein,

[0028] The memory stores instructions executable by the at least one processor, and the instructions are executed to implement:

[0029] Execute the use case set through the automated testing platform to trigger the running of the system program;

[0030] Monitor the running process of the system program through a performance monitoring tool to obtain the execution efficiency of the relevant structured query language SQL in the use case set;

[0031] After the execution of a single use case in the use case set is completed, determining whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline;

[0032] If not, determine whether the SQL performance of the single use case meets the performance indicator;

[0033] Determine the SQL performance result based on the judgment result;

[0034] Generate a system performance report based on the SQL performance results.

[0035] A non-volatile storage medium stores computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the following steps:

[0036] Execute the use case set through the automated testing platform to trigger the running of the system program;

[0037] Monitor the running process of the system program through a performance monitoring tool to obtain the execution efficiency of the relevant structured query language SQL in the use case set;

[0038] After the execution of a single use case in the use case set is completed, determining whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline;

[0039] If not, determine whether the SQL performance of the single use case meets the performance indicator;

[0040] Determine the SQL performance result based on the judgment result;

[0041] Generate a system performance report based on the SQL performance results.

[0042] The present invention provides a method, device and equipment for evaluating system performance based on a test platform, which include at least the following beneficial effects: by combining an automated test platform and existing performance monitoring tools, continuous evaluation of system performance is achieved, showing the performance changes that need the most attention and the most accurate performance anomalies of the system; by executing multiple use cases and executing performance risk use cases multiple times, system performance misjudgments can be reduced; by recording the performance results of each use case, the performance results of executing multiple use cases can be compared, and the system performance can be evaluated more clearly. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 A schematic diagram of the steps of a method for evaluating system performance based on a test platform provided by an embodiment of the present invention;

[0045] Figure 2 A flow chart of a method for evaluating system performance based on a test platform provided in an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a device for evaluating system performance based on a test platform provided in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a device for evaluating system performance based on a test platform provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be described clearly and completely in conjunction with the specific embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0050] The present invention proposes a method, device and equipment for evaluating system performance based on a test platform. By combining an automated test platform and existing performance monitoring tools, continuous evaluation of system performance is achieved, showing the performance changes that need the most attention and the most accurate performance anomalies of the system. The following is a detailed description.

[0051] Figure 1 A schematic diagram of a method for evaluating system performance based on a test platform provided in an embodiment of the present invention may include the following steps:

[0052] S110: Execute the use case set through the automated testing platform to trigger the running of the system program.

[0053] Specifically, Figure 2 As shown in Figure 1, the performance monitoring tool is associated with the system engineering and automated testing platform. The overall process is triggered by the automated testing platform scheduling. When the automated test is running, the performance monitoring tool monitors the execution process of the system program in real time.

[0054] S120: Monitor the running process of the system program through a performance monitoring tool to obtain the execution efficiency of the relevant structured query language SQL in the use case set.

[0055] In one embodiment of the present invention, the running process of the system program is monitored, specifically including: tracking the stack information of the system program during running; monitoring time and memory information; and associating and recording the running efficiency of the SQL script related to the use case.

[0056] Specifically, the performance monitoring tool monitors the execution process of the system program in real time, including tracking the stack information, monitoring time, memory and other information, and associating and recording the running efficiency of the SQL script related to the use case.

[0057] S130: After the execution of a single use case in the use case set is completed, determine whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline. If not, determine whether the SQL performance of the single use case meets the performance index.

[0058] In one embodiment of the present invention, whether the execution efficiency of relevant SQL in a single use case meets the preset performance baseline is judged, specifically including: determining that the SQL script related to the single use case is executed for the first time, then adding a single use case SQL performance baseline according to the single use case and SQL identifier; determining that the SQL related to the single use case already has a performance baseline, then comparing the SQL performance of the single use case with the preset performance baseline, and executing subsequent actions according to the comparison result.

[0059] In one embodiment of the present invention, the SQL performance of a single use case is compared with a preset performance baseline, and subsequent actions are performed according to the comparison result, specifically including: if it is determined that the SQL performance of a single use case is lower than the preset performance baseline, then it is determined whether the single use case is a performance problem, and if not, it is recorded in the performance risk use case set; if it is determined that the SQL performance of a single use case is not lower than the preset performance baseline, then it is determined whether the performance of this SQL meets the performance indicators.

[0060] In one embodiment of the present invention, judging whether the SQL performance of a single use case meets the performance indicator specifically includes: if it is determined that the SQL performance meets the performance indicator, the performance indicator is entered into the performance report, and the performance evaluation is determined to be passed; if it is determined that the SQL performance does not meet the performance indicator, the SQL is recorded in the performance risk use case set.

[0061] Specifically, after a single use case is executed, start to compare whether the execution efficiency of related SQL in the use case meets expectations:

[0062] 1) SQL is executed for the first time: the running baseline is recorded with the use case and SQL identifier;

[0063] 2) SQL already has performance baselines:

[0064] Determine whether the performance is worse than the performance baseline:

[0065] If so, record the performance risk for subsequent evaluation and processing;

[0066] If not, further evaluate whether the performance meets the performance indicators.

[0067] Whether the performance indicators are met:

[0068] If yes, it is recorded in the performance report and the performance evaluation is passed;

[0069] If not, record it as a performance risk and wait for subsequent evaluation and processing.

[0070] In one embodiment of the present invention, after the automated testing platform has completed running all use cases, the performance risk use case set is scheduled to run again for evaluation according to the evaluation requirements; the evaluation requirement is that if the use case is executed for a preset number of times and still does not meet the preset conditions, it is determined to be a performance problem.

[0071] Specifically, after the automated testing platform has completed running all use cases, it will schedule the execution of the performance risk-related use case set again according to the evaluation requirements (if the number of evaluations reaches the preset number but does not meet the preset conditions, it will be listed as a performance problem), and re-run the use case set with performance risks. After the re-run is completed, it will determine whether the performance risk use case set needs to be run again. If so, it will be re-run, otherwise it will end.

[0072] S140: Determine the SQL performance result according to the judgment result, and generate a system performance report according to the SQL performance result.

[0073] In one embodiment of the present invention, after the number of use case executions reaches a preset number, use cases that meet the performance requirements are excluded; a performance monitoring tool is used to determine whether the performance risk turns into a performance problem, and the performance problem is recorded in a performance report.

[0074] Specifically, by eliminating false positives through multiple use case executions, the performance monitoring tool ultimately determines whether the performance risk has turned into a performance problem. If it is determined to be a performance problem, it will be recorded in the performance report; if it cannot be determined to be a performance problem, it will be recorded in the performance risk record.

[0075] The above is a method for evaluating system performance based on a test platform provided by an embodiment of the present invention. Based on the same inventive idea, an embodiment of the present invention also provides a corresponding device for evaluating system performance based on a test platform, such as Figure 2 shown.

[0076] The automated testing platform 201 is used to execute the use case set and trigger the running of the system program; the performance monitoring tool 202 monitors the running process of the system program and obtains the execution efficiency of the relevant structured query language SQL in the use case set; after the execution of a single use case in the use case set is completed, it is determined whether the relevant SQL execution efficiency in the single use case meets the preset performance baseline; if not, it is determined whether the SQL performance of the single use case meets the performance indicator; the SQL performance result is determined based on the judgment result; and a system performance report is generated based on the SQL performance result.

[0077] The embodiment of the present invention also provides a corresponding device for evaluating system performance based on a test platform, such as Figure 3 shown.

[0078] This embodiment provides a device for evaluating system performance based on a test platform, including:

[0079] At least one processor 301; and a memory 302 communicatively connected to the at least one processor 301 via a bus 303; wherein the memory 302 stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute:

[0080] Execute the use case set through the automated testing platform to trigger the running of the system program; monitor the running process of the system program through the performance monitoring tool to obtain the execution efficiency of the relevant structured query language SQL in the use case set; after the execution of a single use case in the use case set is completed, determine whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline; if not, determine whether the SQL performance of the single use case meets the performance indicators; determine the SQL performance results based on the judgment results; generate a system performance report based on the SQL performance results.

[0081] Based on the same idea, some embodiments of the present invention also provide a medium corresponding to the above method.

[0082] Some embodiments of the present invention provide a storage medium storing computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the following steps:

[0083] Execute the use case set through the automated testing platform to trigger the running of the system program; monitor the running process of the system program through the performance monitoring tool to obtain the execution efficiency of the relevant structured query language SQL in the use case set; after the execution of a single use case in the use case set is completed, determine whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline; if not, determine whether the SQL performance of the single use case meets the performance indicators; determine the SQL performance results based on the judgment results; generate a system performance report based on the SQL performance results.

[0084] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0085] The devices and media provided in the embodiments of the present invention correspond one-to-one to the methods, and therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0086] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or method. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the presence of other identical elements in the process, method, commodity or method including the elements.

[0087] The above are only embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

Claims

1. A method for evaluating system performance based on a test platform, characterized in that: include: Execute the use case set through the automated testing platform to trigger the running of the system program; Use performance monitoring tools to track the stack information of system programs during operation; Monitor the time and memory information of system programs during operation; associate and record the operation efficiency of SQL scripts related to use cases, and obtain the execution efficiency of relevant structured query language SQL in the use case set; After the execution of a single use case in the use case set is completed, determine whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline, including: if the SQL script related to the single use case is executed for the first time, then add the performance baseline of the SQL of the single use case according to the single use case and the SQL identifier; if the SQL related to the single use case already has a performance baseline, and the performance of the SQL of the single use case is lower than the preset performance baseline, determine whether the single use case is a performance problem, if not, record it in the performance risk use case set; if the performance of the SQL of the single use case is not lower than the preset performance baseline, determine whether the performance of the SQL meets the performance index; If the performance of this SQL meets the performance index, the performance index is recorded in the performance report to determine that the performance evaluation has passed; if the performance of this SQL does not meet the performance index, this SQL is recorded in the performance risk use case set; Determine the performance result of this SQL according to the judgment result of whether the performance of this SQL meets the performance index; Generate a system performance report based on the performance results of this SQL.

2. The method according to claim 1, characterized in that The method further comprises: When the automated testing platform has completed running all the use cases, the performance risk use case set is scheduled to run again for evaluation according to the evaluation requirements; The evaluation requirement is that if the use case is executed a preset number of times and still does not meet the preset conditions, it is determined to be a performance problem.

3. The method according to claim 2, characterized in that The method further comprises: After the number of use case executions reaches the preset number, the use cases that meet the performance requirements are excluded; The performance monitoring tool is used to determine whether the performance risk has turned into a performance problem, and the performance problem is recorded in the performance report.

4. A device for evaluating system performance based on a test platform, characterized in that: include: Automated testing platform, used to execute use case sets and trigger the running of system programs; Performance monitoring tool, used to track the stack information of system programs during operation; Monitor the time and memory information of system programs during operation; Associate and record the running efficiency of SQL scripts related to the use case, and obtain the execution efficiency of relevant structured query language SQL in the use case set; After the execution of a single use case in the use case set is completed, it is determined whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline, including: if the SQL script related to the single use case is executed for the first time, a performance baseline of the SQL of the single use case is added according to the single use case and the SQL identifier; if the SQL related to the single use case already has a performance baseline, and the performance of the SQL of the single use case is lower than the preset performance baseline, it is determined whether the single use case is a performance problem, and if not, it is recorded in the performance risk use case set; if the performance of the SQL of the single use case is not lower than the preset performance baseline, it is determined whether the performance of the SQL meets the performance index; if the performance of the SQL meets the performance index, the performance index is recorded in the performance report to determine that the performance evaluation is passed; if the performance of the SQL does not meet the performance index, the SQL is recorded in the performance risk use case set; the performance result of the SQL meets the performance index according to the judgment result; and a system performance report is generated according to the performance result of the SQL.

5. A device for evaluating system performance based on a test platform, characterized in that: include: at least one processor; as well as, A memory connected to the at least one processor via a bus; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed to implement: Execute the use case set through the automated testing platform to trigger the running of the system program; Use performance monitoring tools to track the stack information of system programs during operation; monitor the time information and memory information of system programs during operation; associate and record the operation efficiency of SQL scripts related to use cases, and obtain the execution efficiency of relevant structured query language SQL in the use case set; After the execution of a single use case in the use case set is completed, determine whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline, including: if the SQL script related to the single use case is executed for the first time, then add the performance baseline of the SQL of the single use case according to the single use case and the SQL identifier; if the SQL related to the single use case already has a performance baseline, and the performance of the SQL of the single use case is lower than the preset performance baseline, determine whether the single use case is a performance problem, if not, record it in the performance risk use case set; if the performance of the SQL of the single use case is not lower than the preset performance baseline, determine whether the performance of the SQL meets the performance index; If the performance of this SQL meets the performance index, the performance index is recorded in the performance report to determine that the performance evaluation has passed; if the performance of this SQL does not meet the performance index, this SQL is recorded in the performance risk use case set; Determine the performance result of this SQL according to the judgment result of whether the performance of this SQL meets the performance index; Generate a system performance report based on the performance results of this SQL.

6. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are executed by a processor to implement the following steps: Execute the use case set through the automated testing platform to trigger the running of the system program; Use performance monitoring tools to track the stack information of system programs during operation; Monitor the time and memory information of system programs during operation; associate and record the operation efficiency of SQL scripts related to use cases, and obtain the execution efficiency of relevant structured query language SQL in the use case set; After the execution of a single use case in the use case set is completed, determine whether the execution efficiency of the relevant SQL in the single use case meets the preset performance baseline, including: if the SQL script related to the single use case is executed for the first time, then add the performance baseline of the SQL of the single use case according to the single use case and the SQL identifier; if the SQL related to the single use case already has a performance baseline, and the performance of the SQL of the single use case is lower than the preset performance baseline, determine whether the single use case is a performance problem, if not, record it in the performance risk use case set; if the performance of the SQL of the single use case is not lower than the preset performance baseline, determine whether the performance of the SQL meets the performance index; If the performance of this SQL meets the performance index, the performance index is recorded in the performance report to determine that the performance evaluation has passed; if the performance of this SQL does not meet the performance index, this SQL is recorded in the performance risk use case set; Determine the performance result of this SQL according to the judgment result of whether the performance of this SQL meets the performance index; Generate a system performance report based on the performance results of this SQL.

Images