Automatic evaluation method, system and equipment for database performance test and medium

Abstract

The invention discloses an automatic evaluation method, system and equipment for database performance test and a medium, belongs to the technical field of database performance test, and aims to solve the technical problem of how to overcome the defects of low manual evaluation efficiency, high misjudgment rate, feedback lagging and non-standard flow in the database performance evaluation process in the prior art. According to the technical scheme, the method comprises the steps of data collection, wherein TSBS performance test historical data and a current test result are obtained; constructing a dynamic threshold: constructing the dynamic threshold based on the TSBS performance test historical data; merging request automatic judgment: taking a TSBS performance test as a merging standard which must be passed through a Gitee access control assembly line technology; when the PR is combined, differential performance verification is carried out; automatic comparison and judgment of daily performance test results: executing a TSBS daily performance test, obtaining a current-day test result, and comparing the current-day test result with the dynamic threshold value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database performance testing, in particular to a database performance test automatic evaluation method, system, device and medium. BACKGROUND

[0002] The core goal of database performance testing is to verify the stability, response speed and resource utilization of the system under high load, to ensure that it meets the business requirements. In the existing field of database performance testing, there are the following technical defects in the performance evaluation during the database development process:

[0003] ① Low efficiency of manual evaluation: The traditional scheme relies on manual sorting of performance results and individual value comparison, resulting in wasted labor due to repetitive labor, which cannot meet the rapid iteration requirements.

[0004] ② Lack of quantitative standards: There is a lack of clear performance compliance quantitative indicators, and developers cannot accurately judge the performance impact when submitting PR (Pull Request), resulting in blind integration decisions.

[0005] ③ Significant feedback lag: Due to the influence of test result fluctuations, it takes several days of data to determine performance decline, delaying problem discovery.

[0006] ④ High risk of misjudgment: When manually evaluating performance based on fluctuating data, subjective factors can easily lead to misjudgment, affecting version quality.

[0007] ⑤ Non-standard response process: When performance issues involve multiple PRs and multiple developers, there is a lack of standardized response process, resulting in low positioning efficiency.

[0008] In the prior art, some schemes attempt to judge performance through single test results or simple threshold values, but do not solve the problem of dynamic threshold setting and automatic evaluation under fluctuating data, and cannot meet the needs of high-frequency iteration in the development scene.

[0009] Therefore, how to overcome the defects of low efficiency, high misjudgment rate, feedback lag, and non-standard process in the manual evaluation process of database performance evaluation in the prior art, and improve the accuracy and timeliness of database performance management is a technical problem that needs to be solved at present. SUMMARY

[0010] The technical task of the present application is to provide a database performance test automatic evaluation method, system, device and medium to solve the problem of how to overcome the defects of low efficiency, high misjudgment rate, feedback lag, and non-standard process in the manual evaluation process of database performance evaluation in the prior art, and improve the accuracy and timeliness of database performance management.

[0011] The technical task of the present application is achieved in the following manner, a database performance test automatic evaluation method, which is specifically as follows:

[0012] Data collection: obtain TSBS performance test historical data and current test results;

[0013] Dynamic threshold construction: dynamic threshold is constructed based on TSBS performance test historical data; wherein, the dynamic threshold is determined by calculating the average value, standard deviation and adjustable coefficient k of the test case;

[0014] Pull request (PR) automatic evaluation: TSBS performance test is taken as a necessary passing standard through Gitee access pipeline technology; and when PR is merged, differential performance verification is carried out;

[0015] Automatic comparison and evaluation of daily performance test results: daily performance test of TSBS is performed, daily test results are obtained, and daily test results are compared with dynamic threshold: if all case values are less than the upper threshold, it is determined to pass; if any case value is not less than the upper threshold, it enters the manual confirmation link; wherein, the manual confirmation link is specifically: re-running TSBS performance test of master branch of the day, if it still does not meet the standard, the standardized response process is triggered.

[0016] As a preferred embodiment, the dynamic threshold is constructed based on historical data, which is specifically as follows:

[0017] Collect 7 days of TSBS performance test historical data of the target database in units of weeks;

[0018] Calculate the key statistics for each test case; wherein, the key statistics include the next minimum value, the next maximum value, the average value (μ), the standard deviation (α) and the coefficient of variation (CV);

[0019] Determine the upper threshold, the formula is: upper threshold = historical mean + k x historical standard deviation; wherein, k represents the adjustable coefficient (default k = 2, covering about 95% of historical data).

[0020] As a preferred embodiment, the merging standard of TSBS performance test is specifically as follows:

[0021] If all case values of TSBS performance of the merged branch are less than the upper threshold, it is determined to pass;

[0022] If any case value of TSBS performance of the merged branch is not less than the upper threshold, it is determined not to pass, and after the performance problem is repaired by the R&D personnel, TSBS performance test is re-performed.

[0023] As a preferred embodiment, the standardized response process is specifically as follows:

[0024] Create an ISSUE single: When the daily performance test fails and is confirmed by manual, the tester creates an ISSUE single, specifies the Release Team as the responsible person;

[0025] Trace the PR list: The Release Team provides the PR list integrated the day before and notifies the relevant R&D, and transfers it to the performance team for responsibility;

[0026] R&D retest and rectification: R&D personnel re-execute TSBS performance test and submit the results, and the performance team confirms the impact PR and notifies the rectification, which is solved by the corresponding R&D responsible person.

[0027] More preferably, the PR integration verification includes daily dynamic threshold for open source branch PR.

[0028] A database performance test automatic evaluation system, the system comprises:

[0029] A data acquisition module for acquiring TSBS performance test historical data and current test results;

[0030] A dynamic threshold construction module for constructing a dynamic threshold based on TSBS performance test historical data;

[0031] A merge request performance evaluation module for evaluating the performance of PR integration through Gitee access control pipeline technology;

[0032] A daily performance evaluation module for performing daily TSBS performance tests and comparing the daily test results with the dynamic threshold to obtain an evaluation conclusion.

[0033] As a preferred, the working process of the dynamic threshold construction module is as follows:

[0034] ① Collect 7 days of TSBS performance test historical data of the target database in a week;

[0035] ② Calculate the key statistics for each test case; wherein, the key statistics include the second minimum value, the second maximum value, the average value (μ), the standard deviation (α) and the coefficient of variation (CV);

[0036] ③ Determine the upper threshold, the formula is: upper threshold = historical mean + k x historical standard deviation; wherein, k represents the adjustable coefficient (default k = 2, covering about 95% of historical data).

[0037] More preferably, the daily performance evaluation module performs the TSBS daily performance test, obtains the daily test result, and compares the daily test result with the dynamic threshold value: if all use case values are less than the upper threshold value, it is determined to pass; if any use case value is not less than the upper threshold value, it enters the manual confirmation link; wherein the manual confirmation link is specifically: re-running the daily master branch TSBS performance test, if it still does not meet the standard, the standardized response process is triggered;

[0038] The standardized response process is specifically as follows:

[0039] Create an ISSUE single: when the daily performance test fails and after manual confirmation, the tester creates an ISSUE single, and specifies the Release Team as the responsible person;

[0040] Trace the PR list: the Release Team provides the previous day's merged PR list and notifies the relevant research and development, and transfers the performance team responsible;

[0041] R&D retest and rectification: the R&D personnel re-execute the TSBS performance test and submit the results, and the performance team confirms the impact PR and notifies the rectification, and the corresponding R&D responsible person solves the problem.

[0042] An electronic device, comprising: a memory and at least one processor;

[0043] The memory has stored thereon a computer program;

[0044] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the database performance test automatic evaluation method as described above.

[0045] A computer readable storage medium, the computer readable storage medium has stored therein a computer program, the computer program can be executed by a processor to implement the database performance test automatic evaluation method as described above.

[0046] The database performance test automatic evaluation method, system, device and medium of the application have the following advantages:

[0047] (I) The application realizes the automatic evaluation of the database performance through the historical data driven dynamic threshold model (upper threshold value = historical mean + k x standard deviation), and solves the problems of low manual evaluation efficiency, high misjudgment rate and feedback lag in the prior art through daily test automation comparison, PR merging differential verification and standardized response process, improves the accuracy and timeliness of database performance management, and is suitable for performance problem rapid positioning and quantitative judgment in the TSBS (Time Series Benchmark Suite) scene;

[0048] (II) The application overcomes the defects of low efficiency, high misjudgment rate, feedback lag and non-standard process in the existing art by establishing a dynamic threshold model driven by historical data, a standardized evaluation process and an automatic program, and improves the accuracy and timeliness of database performance management;

[0049] (III) The application replaces manual comparison with automatic evaluation, reduces labor consumption by more than 80%, shortens the feedback time of daily performance problems from 48 hours to 8 hours, and effectively improves the evaluation efficiency;

[0050] (IV) The application covers 95% of normal fluctuation scenarios based on dynamic threshold of historical data, and the misjudgment rate is reduced from 30% to less than 5% by combining with automatic program determination, effectively improving the accuracy of evaluation;

[0051] (V) The application realizes standardized landing by clearly defining the PR quantitative standard, and all PRs (including small changes) need to pass performance verification to avoid performance degradation caused by single code modification;

[0052] (VI) The application realizes closed-loop management of problem positioning, rectification and verification through standardized response process, and the average solution time of performance problems is shortened from 72 hours to 24 hours, realizing process standardization. BRIEF DESCRIPTION OF DRAWINGS

[0053] The application will be further described below in conjunction with the accompanying drawings.

[0054] FIG. 1 is a flow chart of a database performance test automatic evaluation method according to the application; Figure 1 FIG. 2 is a structure diagram of a database performance test automatic evaluation system according to the application; FIG. 3 is a PR access test flow chart according to the application;

[0055] FIG. 4 is a daily performance test flow chart according to the application. Figure 2 FIG. 5 is a structure diagram of a database performance test automatic evaluation system according to the application;

[0056] FIG. 6 is a PR access test flow chart according to the application; Figure 3 FIG. 7 is a daily performance test flow chart according to the application.

[0057] FIG. 8 is a structure diagram of a database performance test automatic evaluation system according to the application; Figure 4 FIG. 9 is a PR access test flow chart according to the application; DETAILED DESCRIPTION

[0058] The database performance test automatic evaluation method, system, device and medium of the application will be described in detail below with reference to the accompanying drawings and specific examples.

[0059] Example 1:

[0060] As shown in FIG. 1, the present embodiment provides a database performance test automatic evaluation method, which specifically includes the following steps: Figure 1

[0061] S1, data collection: obtain TSBS performance test historical data and current test results;

[0062] S2, dynamic threshold construction: based on TSBS performance test historical data to construct a dynamic threshold; wherein, the dynamic threshold is determined by calculating the average value, standard deviation and adjustable coefficient k of the test case;

[0063] S3, merge request (Pull Request) automatic evaluation: through the Gitee access control pipeline technology, the TSBS performance test is taken as a necessary passing standard for merging; and when the PR is merged, differential performance verification is carried out;

[0064] S4, daily performance test result automatic comparison and evaluation: execute TSBS daily performance test, obtain the daily test result, and compare the daily test result with the dynamic threshold: if all the values of the test cases are less than the upper threshold, it is determined to pass; if any value of the test case is not less than the upper threshold, it enters the manual confirmation link; wherein, the manual confirmation link is specifically: re-running the TSBS performance test of the master branch of the day, if it still does not meet the standard, the standardized response process is triggered.

[0065] The dynamic threshold based on historical data in step S2 of the embodiment is specifically as follows:

[0066] S201, collect the TSBS performance test historical data of the target database for 7 days in a week;

[0067] S202, calculate the key statistics for each test case; wherein, the key statistics include the next minimum value, the next maximum value, the average value (μ), the standard deviation (α) and the coefficient of variation (CV);

[0068] S203, determine the upper threshold, the formula is: upper threshold = historical mean + k x historical standard deviation; wherein, k represents the adjustable coefficient (default k = 2, covering about 95% of the historical data).

[0069] The merging standard of TSBS performance test in step S3 of the embodiment is specifically as follows:

[0070] If all the values of the TSBS performance of the merged branch are less than the upper threshold, it is determined to pass;

[0071] If any value of the TSBS performance of the merged branch is not less than the upper threshold, it is determined not to pass, and after the performance problem is repaired by the R&D personnel, the TSBS performance test is re-performed.

[0072] The standardized response process in step S3 of the embodiment is specifically as follows:

[0073] S401, create an ISSUE single: when the daily performance test fails and after manual confirmation, the tester creates an ISSUE single, specifies the Release Team as the responsible person;

[0074] S402, trace the PR list: the Release Team provides the PR list merged the previous day and notifies the relevant R&D, and transfers the performance team for responsibility;

[0075] S403, R&D retest and rectification: the R&D personnel re-execute the TSBS performance test and submit the results, and the performance team confirms the impact PR and notifies the rectification, which is solved by the corresponding R&D person in charge.

[0076] The PR merge check in step S3 of the embodiment includes daily dynamic threshold for open source branch PR.

[0077] Embodiment 2:

[0078] As shown in the accompanying Figure 2 , the embodiment provides a database performance test automatic evaluation system, which comprises:

[0079] A data acquisition module for acquiring TSBS performance test historical data and current test results;

[0080] A dynamic threshold construction module for constructing a dynamic threshold based on TSBS performance test historical data;

[0081] A merge request performance evaluation module for evaluating the performance when PR is merged through Gitee access pipeline technology;

[0082] A daily performance evaluation module for performing daily TSBS performance test and comparing the daily test results with the dynamic threshold to obtain an evaluation conclusion.

[0083] The working process of the dynamic threshold construction module in the embodiment is as follows:

[0084] ① Collect 7 days of TSBS performance test historical data of the target database in units of weeks;

[0085] ② Calculate the key statistics for each test case; wherein the key statistics include the second minimum value, the second maximum value, the average value (μ), the standard deviation (α) and the coefficient of variation (CV);

[0086] ③ Determine the upper threshold, the formula is: upper threshold = historical mean + k x historical standard deviation; wherein k represents an adjustable coefficient (default k = 2, covering about 95% of historical data).

[0087] In the embodiment, as shown in the accompanying Figure 3As shown, the performance verification process when PR (Pull Request) is merged is: after the PR test is completed, the result is obtained, and the judgment is made through automatic quantitative analysis. If it fails, the developer needs to perform self-checking to determine whether it has an impact on performance. If it is determined that there is no impact, the TSBS test is re-executed; if there is an impact, the developer needs to repair the performance problem, and then test again until it passes. This process ensures that the performance before the PR is merged meets the standards, avoiding the impact of performance problems on the quality of the version.

[0088] As shown in the accompanying Figure 4 In this embodiment, the daily performance judgment module performs TSBS daily performance test, obtains the test result of the day, and compares the test result of the day with the dynamic threshold: if all use case values are less than the upper threshold, it is determined to pass; if any use case value is not less than the upper threshold, it enters the manual confirmation link; wherein the manual confirmation link is specifically: re-running the TSBS performance test of the master branch of the day, if it still does not meet the standard, triggering the standardized response process to ensure timely discovery and processing of performance anomalies;

[0089] The standardized response process is specifically as follows:

[0090] 1. Create an ISSUE single: after the daily performance test fails and manual confirmation, the tester creates an ISSUE single, and specifies the Release Team as the responsible person;

[0091] 2. Trace the PR list: the Release Team provides the merged PR list of the previous day and notifies the relevant research and development, and transfers the performance team responsible;

[0092] 3. Research and development retest and rectification: the developer re-executes the TSBS performance test and submits the result, and the performance team confirms the impact PR and notifies the rectification, which is solved by the corresponding developer responsible person.

[0093] Embodiment 3:

[0094] The embodiment also provides an electronic device, comprising: a memory and a processor;

[0095] The memory stores computer execution instructions;

[0096] The processor executes the computer execution instructions stored in the memory, so that the processor executes the database performance test automatic evaluation method in any embodiment of the application.

[0097] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The processor can be a microprocessor or the processor can be any conventional processor.

[0098] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function, etc.; and the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0099] Embodiment 4:

[0100] The embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, the instructions are loaded by a processor to make the processor execute the database performance test automatic evaluation method in any embodiment of the application. Specifically, a system or device provided with a storage medium can be provided, and the storage medium stores software program codes for realizing the functions of any embodiment of the above-mentioned embodiments, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0101] In this case, the program codes read from the storage medium can realize the functions of any one of the above-mentioned embodiments, and therefore the program codes and the storage medium storing the program codes constitute a part of the application.

[0102] The storage medium for providing the program codes includes floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program codes can be downloaded from a server computer through a communication network.

[0103] Furthermore, it should be understood that, not only the program codes read by the computer can be executed to complete the actual operation, but also the operating system and the like operating on the computer can be instructed based on the program codes to complete part or all of the actual operation, thereby realizing the function of any one of the above embodiments.

[0104] Furthermore, it should be understood that, not only the program codes read by the computer can be executed to complete the actual operation, but also the operating system and the like operating on the computer can be instructed based on the program codes to complete part or all of the actual operation, thereby realizing the function of any one of the above embodiments.

[0105] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A database performance test automatic evaluation method, characterized in that, The method is specifically as follows: Data collection: obtain TSBS performance test historical data and current test results; Dynamic threshold construction: based on TSBS performance test historical data, a dynamic threshold is constructed; wherein the dynamic threshold is determined by calculating the average value, standard deviation and adjustable coefficient k of the test cases; Merging request automatic evaluation: TSBS performance test is taken as a necessary passing standard through Gitee access pipeline technology; and when PR is merged, differential performance verification is carried out; Daily performance test result automatic comparison and evaluation: daily TSBS performance test is performed, the daily test result is obtained, and the daily test result is compared with the dynamic threshold: if all the values of the test cases are less than the upper threshold, it is determined to pass; if any value of the test cases is not less than the upper threshold, it enters the manual confirmation link; wherein the manual confirmation link is specifically: the TSBS performance test of the master branch on the same day is re-run, and if it still does not meet the standard, a standardized response process is triggered.

2. The method of claim 1, wherein, The dynamic threshold is constructed based on historical data as follows: Collect 7 days of TSBS performance test historical data of the target database in units of weeks; Calculate the key statistics for each test case; wherein the key statistics include the next minimum value, the next maximum value, the average value, the standard deviation and the coefficient of variation; Determine the upper threshold, the formula is: upper threshold = historical average + k * historical standard deviation; wherein k represents the adjustable coefficient.

3. The method of claim 1, wherein, The merging standard of TSBS performance test is as follows: If all the values of TSBS performance of the merged branch are less than the upper threshold, it is determined to pass; If any value of TSBS performance of the merged branch is not less than the upper threshold, it is determined not to pass, and after the performance problem is repaired by the R&D personnel, TSBS performance test is re-performed.

4. The method of claim 1, wherein, The standardized response process is as follows: Create an ISSUE single: when the daily performance test fails and after manual confirmation, the tester creates an ISSUE single and specifies the Release Team as the responsible person; Trace the PR list: the Release Team provides the PR list merged the previous day and notifies the relevant R&D, and transfers it to the performance team; R&D retest and rectification: the R&D personnel re-perform TSBS performance test and submit the results, and the performance team confirms the impact PR and notifies the rectification, which is solved by the corresponding R&D person in charge.

5. The method of claim 1-4, wherein, PR merging verification includes daily dynamic threshold for open source branch PR.

6. A database performance test automation evaluation system, characterized by, The system comprises: A data collection module for obtaining TSBS performance test historical data and current test results; A dynamic threshold construction module for constructing a dynamic threshold based on TSBS performance test historical data; A merging request performance evaluation module for evaluating the performance when PR is merged through Gitee access pipeline technology; A daily performance evaluation module for performing daily TSBS performance test, comparing the daily test result with the dynamic threshold, and obtaining the evaluation conclusion.

7. The database performance test automation grading system of claim 6, wherein, The working process of the dynamic threshold construction module is as follows: ①Collect 7 days of TSBS performance test historical data of the target database in units of weeks; ②Calculate key statistics for each test case; wherein, the key statistics include the next minimum value, the next maximum value, the average value, the standard deviation and the coefficient of variation; ③Determine the upper threshold, the formula is: upper threshold = historical average + k x historical standard deviation; wherein, k represents the adjustable coefficient.

8. The database performance test automation evaluation system of claim 6 or 7, wherein, The daily performance evaluation module executes TSBS daily performance test, obtains the daily test results, and compares the daily test results with the dynamic threshold: if all the values of the test cases are less than the upper threshold, it is determined to pass; if any test case value is not less than the upper threshold, it enters the manual confirmation link; wherein, the manual confirmation link is specifically: re-running the TSBS performance test of the master branch of the day, if it still does not meet the standard, the standardized response process is triggered; Wherein, the standardized response process is as follows: Create an ISSUE single: when the daily performance test fails and after manual confirmation, the tester creates an ISSUE single and specifies the Release Team as the responsible person; Trace the PR list: the Release Team provides the PR list integrated the day before and notifies the relevant research and development, and transfers the performance team responsible; R&D retest and rectification: the R&D personnel re-execute the TSBS performance test and submit the results, and the performance team confirms the impact PR and notifies the rectification, and the corresponding R&D person in charge solves the problem.

9. An electronic device, comprising: It includes: a memory and at least one processor; wherein the memory has stored thereon a computer program; the at least one processor executes the computer program stored in the memory, so that the at least one processor executes the database performance test automatic evaluation method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored therein a computer program, which can be executed by a processor to implement the database performance test automatic evaluation method according to any one of claims 1 to 5.

Images