Test case selection methods, electronic devices, storage media, and program products

Abstract

This application discloses a test case selection method, electronic device, storage medium, and program product, relating to the field of performance testing technology. Through this application, sample test cases are executed under different operating parameters. By calculating the impact evaluation value, representative test cases that are more sensitive to changes in operating parameters are selected. This ensures that the selected target test cases can still accurately reflect the performance of the device under test, reducing the number of test cases and the time required for testing while ensuring the test effect and improving the test efficiency.

Description

Technical Field

[0001] This application relates to the field of performance testing technology, and in particular to a test case selection method, electronic device, storage medium, and program product. Background Technology

[0002] Testing and evaluating processors is an important means of obtaining processor performance data, providing reliable quantitative data support for processor research and development improvements.

[0003] SPEC CPU2017, a Central Processing Unit (CPU) performance benchmark published by the Standard Performance Evaluation Corporation, is an industry-standard benchmark for processor performance and includes 43 integer and floating-point test cases. Currently, testing a processor typically requires running all of these test cases completely, and then analyzing the data from those test cases to obtain the processor's performance data.

[0004] However, the time required to run all test cases is long, resulting in low testing efficiency and difficulty in obtaining processor performance data in a timely manner. Summary of the Invention

[0005] This application provides a test case selection method, electronic device, storage medium, and program product to at least solve the problem of low testing efficiency in related technologies.

[0006] This application provides a test case selection method, including:

[0007] Multiple rounds of testing were performed on the first device based on multiple sample test cases. The test score of each sample test case in each round of testing was obtained. The operating parameters of the first device were different in each round of testing. The test score was used to characterize the speed at which the first device processed tasks when running the sample test cases.

[0008] Based on the test scores and operating parameters, the impact evaluation value corresponding to each sample test case is calculated. The impact evaluation value corresponding to the sample test case is used to characterize the correlation between the sample test case and the operating parameters. The correlation refers to the degree of influence of the change of operating parameters on the test score of the sample test case.

[0009] Identify at least one sample test case whose impact assessment value is greater than the assessment value threshold as the target test case.

[0010] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described test case selection methods when executing the computer program.

[0011] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described test case selection methods.

[0012] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described test case selection methods.

[0013] This application executes sample test cases under different operating parameters. By calculating the impact evaluation value, it selects representative test cases that are more sensitive to changes in operating parameters. This ensures that the selected target test cases can still accurately reflect the performance of the device under test. While ensuring the test effect, it reduces the number of test cases, reduces the test time, and improves the test efficiency. Attached Figure Description

[0014] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 Flowchart of the test case selection method provided for this application;

[0016] Figure 2 This is a schematic diagram of the test case selection device provided in the embodiments of this application;

[0017] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The embodiments of this application provide a test case selection method. The method will be described in detail below with reference to the execution flow of the test case selection method.

[0022] Figure 1 A flowchart illustrating the method for selecting test cases provided in this application. Figure 1 As shown, the method includes the following steps:

[0023] S101. Perform multiple rounds of testing on the first device based on multiple sample test cases, and obtain the test score of each sample test case in each round of testing.

[0024] In each round of testing, the operating parameters of the first device are different, and the test score is used to characterize the speed at which the first device processes tasks when running sample test cases.

[0025] Multiple sample test cases refer to a set of benchmark test cases covering different computing scenarios, such as the 43 test cases in SPEC CPU2017. The performance data obtained after performing tests on the device based on multiple sample test cases can comprehensively characterize the processing capabilities of the Central Processing Unit (CPU) in computing, memory access, and other scenarios.

[0026] The test score is used to quantify the task processing speed of the first device running sample test cases. A higher test score indicates a faster speed, and a lower test score indicates a slower speed.

[0027] During multiple rounds of testing on the first device, its operating parameters are adjusted. Specifically, in each round of testing, only one operating parameter of the first device is adjusted, and based on the adjusted operating parameter, the first device is controlled to run multiple sample test cases sequentially to obtain the test score for each sample test case under the current operating parameters.

[0028] S102. Based on the test scores and running parameters, calculate the impact assessment value corresponding to each sample test case.

[0029] Among them, the impact evaluation value corresponding to the sample test case is used to characterize the correlation between the sample test case and the running parameters. The correlation refers to the degree of influence of the change of the running parameters on the test score of the sample test case.

[0030] Specifically, for the same sample test case, if the difference in the running parameters is small in two tests, but the difference in the test scores is large, it means that the sample test case is more sensitive to the running parameters, has a high degree of correlation, and has a higher corresponding impact assessment value.

[0031] If the operating parameters differ significantly between the two tests, but the test scores differ only slightly, it indicates that the sample test cases are not sensitive to the operating parameters, have low correlation, and therefore have a low impact assessment value.

[0032] S103. Determine at least one sample test case whose impact assessment value is greater than the assessment value threshold as the target test case.

[0033] The evaluation threshold is a preset value that can exclude sample test cases with low correlation. It is set according to the test accuracy requirements and / or expert experience.

[0034] The target test cases are a subset of test cases that are sensitive to changes in operating parameters and can accurately characterize the device's performance. These test cases are used to replace the full set of sample test cases when testing the first device.

[0035] Specifically, different evaluation thresholds are set for different operating parameters, and the target test cases corresponding to the operating parameters are determined.

[0036] For example, in step S101, if the operating parameter changed in each round is the CPU frequency, the impact evaluation value of the sample test case reflects the correlation between the test score of the sample test case and the change in CPU frequency. The target test case corresponding to the CPU frequency is selected based on the evaluation value threshold corresponding to the CPU frequency.

[0037] For example, in step S101, if the running parameter that is changed in each round is the memory frequency, the impact evaluation value of the sample test case reflects the correlation between the test score of the sample test case and the change in memory frequency. The target test case corresponding to the memory frequency is selected based on the evaluation value threshold corresponding to the memory frequency.

[0038] Once the target test cases are obtained, subsequent testing of the first device can be performed by executing only the target test cases, without having to execute all sample test cases.

[0039] This application calculates the impact assessment value by executing sample test cases under different operating parameters, and selects representative test cases that are more sensitive to changes in operating parameters. This ensures that the selected target test cases can still accurately reflect the performance of the device under test, reducing the number of test cases and the time required for testing while ensuring the test effect and improving the test efficiency.

[0040] Based on the above embodiments, multiple rounds of testing are performed on the first device according to multiple sample test cases to obtain the test score of each sample test case in each round of testing. This includes: for each sample test case, determining the number of tasks completed per second by the first device when running the sample test case, and the time required to complete a single task; calculating the test score of the sample test case in the current test based on the reciprocal of the number of tasks completed per second and / or the time required to complete a single task by the first device, wherein the test score is positively correlated with the reciprocal of the number of tasks completed per second and / or the time required to complete a single task by the first device.

[0041] Tasks completed per second (SPS) refers to the number of tasks successfully executed by the first device per unit of time in a multi-tasking concurrent scenario, such as the number of successfully run sample test cases. It is used to evaluate the computing performance of the first device in a multi-tasking scenario. The more tasks completed per second, the better the computing performance of the first device in a multi-tasking scenario; the fewer tasks completed per second, the worse the computing performance of the first device in a multi-tasking scenario.

[0042] The time required to complete a single task refers to the time it takes for the first device to successfully execute a single sample test case in a single-task serial scenario. It is used to evaluate the computing performance of the first device in a single-task scenario. The shorter the time required to complete a single task, the better the computing performance of the first device in a multi-task scenario; the longer the time required to complete a single task, the worse the computing performance of the first device in a multi-task scenario.

[0043] Specifically, the sample test cases are divided into speed test cases and rate test cases. For speed test cases, the number of tasks completed per second is used as the test score of the sample test case. For rate test cases, the reciprocal of the time required to complete a single task is used as the test score of the sample test case.

[0044] This application quantifies the computing power of the first device in parallel or serial task scenarios by utilizing the number of tasks completed per second and the time required to complete a single task. This provides a clear basis for subsequently calculating the impact evaluation value of sample test cases, ensuring the accuracy of the evaluation of sample test cases, and thus ensuring the accuracy of the selection of target test cases.

[0045] In some embodiments, the impact assessment value corresponding to each sample test case is calculated based on the test score and the running parameters, including: for each sample test case, performing linear fitting with the running parameters as independent variables and the test score as dependent variables to obtain the regression equation corresponding to the sample test case; and determining the absolute value of the slope of the regression equation as the impact assessment value corresponding to the test case.

[0046] After performing multiple rounds of testing on the first device using multiple sample test cases, test scores for each sample test case under different operating parameters can be obtained. For example, after performing five rounds of testing using 43 sample test cases, five test scores can be obtained for each sample test case.

[0047] Optionally, after each sample test case is executed, the "running parameters - test score" data pair is saved.

[0048] For each sample test case, a univariate linear regression equation is constructed based on multiple "running parameter-test score" data pairs, such that the sum of the squares of the distances between the coordinate points corresponding to each data pair and the regression equation is minimized.

[0049] The absolute value of the slope of the regression equation represents the change in test score corresponding to a unit change in operating parameters. It can reflect the degree of influence of changes in operating parameters on the test scores of sample test cases, i.e., the impact assessment value.

[0050] After obtaining the regression equation corresponding to the sample test cases, the theoretical score corresponding to each running parameter of the sample test cases in multiple rounds of testing is determined according to the regression equation. For each running parameter, the square of the difference between the theoretical score and the test score corresponding to the running parameter is calculated to obtain the first reference value corresponding to the running parameter. The square of the difference between the test score corresponding to the running parameter and the average of the test scores in multiple rounds of testing is calculated to obtain the second reference value corresponding to the running parameter. The ratio of the sum of multiple first reference values ​​corresponding to the sample test cases to the sum of multiple second reference values ​​is calculated to obtain the variation evaluation value of the sample test cases. If the variation evaluation value is greater than the preset variation threshold, the sample test cases are determined not to be target test cases.

[0051] The theoretical score refers to the dependent variable value in the regression equation corresponding to the actual operating parameters when the first device is tested.

[0052] The first reference value is the square of the difference between the theoretical score and the test score corresponding to the running parameters, reflecting the fitting bias of a single point of data. The second reference value is the square of the difference between the test score and the average score of multiple rounds of testing, reflecting the degree of variation of a single point of data relative to the overall dependent variable level.

[0053] The variance evaluation value is the ratio of the sum of multiple first reference values ​​to the sum of multiple second reference values, reflecting the proportion of test scores that cannot be linearly explained by the runtime parameters. A smaller variance evaluation value indicates that the test scores of the test cases can be more predictable by adjusting the runtime parameters. A larger variance evaluation value indicates that the test scores of the test cases are more affected by factors other than the runtime parameters.

[0054] If the variance assessment value of a sample test case is greater than the preset variance threshold, it means that the linear relationship between the sample test case and the running parameters is unstable. The accuracy of determining the impact assessment value of the sample test case based on the regression equation is insufficient, and such sample test cases need to be excluded when selecting target test cases.

[0055] That is, sample test cases with a variation evaluation value less than or equal to a preset variation threshold and an impact evaluation value greater than the evaluation value threshold are selected as target test cases.

[0056] This application constructs a regression equation between operating parameters and test scores, uses the slope of the regression equation as an evaluation value of the influence of sample test cases, and determines the stability of the linear relationship between the test scores and operating parameters of sample test cases based on the regression equation. This makes the final target test cases sensitive to changes in operating parameters, while the linear relationship between the changes in their test scores and changes in operating parameters is stable, reducing performance evaluation errors caused by sudden changes in test scores and improving the accuracy and stability of testing.

[0057] In some embodiments, the operating parameters include the central processing unit frequency and the memory frequency; the test scores include a first test score and a second test score.

[0058] Accordingly, multiple rounds of testing are performed on the first device based on multiple sample test cases, and the test score of each sample test case in each round of testing is obtained. This includes: fixing the memory frequency of the first device to a first preset frequency, performing multiple rounds of first performance testing on the first device based on multiple sample test cases, adjusting the CPU frequency according to a first preset step size in each round of first performance testing, and then sequentially executing multiple sample test cases to obtain the first test score of multiple sample test cases at different CPU frequencies; fixing the CPU frequency of the first device to a second preset parameter, performing multiple rounds of second performance testing on the first device based on multiple sample test cases, adjusting the memory frequency according to a preset second step size in each round of second performance testing, and then sequentially executing multiple sample test cases to obtain the second test score of multiple sample test cases at different memory frequencies.

[0059] Optionally, multiple CPU frequencies, such as 100%, 90%, 80%, 70%, and 60%, can be preset according to a first preset step size; and multiple memory frequencies, such as 100%, 87%, 74%, and 61%, can be preset according to a second preset step size. It can be understood that the first preset step size and the second preset step size can be uniform or non-uniform step sizes.

[0060] The central processing unit (CPU) corresponds to the computing power of the first device, which is obtained through multiple rounds of first performance testing; the memory frequency corresponds to the data processing power of the first device, which is obtained through multiple rounds of second performance testing. Both are key performance indicators of the first device.

[0061] The first preset frequency is the nominal frequency of the memory, such as 3200MT / s; the second preset frequency is the nominal frequency of the CPU, such as 2.3GHz. The nominal frequency is the standard operating parameter set at the factory of the first device. Fixing the memory frequency or CPU frequency of the first device at the nominal frequency puts the first device in a standard operating state. The test scores of the sample test cases obtained under this state are more reliable.

[0062] After fixing the memory frequency to a first preset frequency, adjust the CPU frequency, for example, by setting it to 100%, 90%, 80%, 70%, and 60% sequentially. Run multiple sample test cases at each CPU frequency and record the first test score of each sample test case at each CPU frequency.

[0063] After fixing the CPU frequency to the second preset frequency, adjust the memory frequency, for example, setting it to 100%, 87%, 74%, and 61% sequentially. Run multiple sample test cases at each memory frequency and record the second test score for each sample test case at each memory frequency.

[0064] Optionally, under the same running parameters, each sample test case is executed multiple times, and the average or median of the test scores from the multiple runs is taken as the final recorded first or second test score.

[0065] Accordingly, the impact assessment value includes the first impact assessment value and the second impact assessment value.

[0066] Based on the test scores and operating parameters, the impact assessment value corresponding to each sample test case is calculated, including: based on the first test scores of multiple sample test cases at different CPU frequencies, and the CPU frequency of the first device in each round of the first performance test, the first impact assessment value corresponding to each sample test case is calculated, and the first impact assessment value represents the degree of influence of the change in CPU frequency on the first test score of the sample test case; based on the second test scores of multiple sample test cases at different memory frequencies, and the memory frequency of the first device in each round of the second performance test, the second impact assessment value corresponding to each sample test case is calculated, and the second impact assessment value represents the degree of influence of the change in memory frequency on the second test score of the sample test case.

[0067] Specifically, for each sample test case, a linear fit is performed based on multiple different CPU frequencies and corresponding first test scores in the first performance test to obtain the CPU frequency change regression equation corresponding to the sample test case, and the first influence evaluation value corresponding to the sample test case is determined as the slope of the CPU frequency change regression equation.

[0068] For each sample test case, a linear fit is performed based on multiple different memory frequencies and corresponding second test scores in the second performance test to obtain the memory frequency change regression equation corresponding to the sample test case. The second influence evaluation value corresponding to the sample test case is determined as the slope of the memory frequency change regression equation.

[0069] Accordingly, the evaluation thresholds include a first evaluation threshold and a second evaluation threshold.

[0070] Identifying at least one sample test case with an impact assessment value greater than an assessment value threshold as a target test case includes: identifying at least one sample test case with a first impact assessment value greater than a first assessment value threshold as a first target test case, wherein the first impact assessment value characterizes the degree of influence of the change in central processing unit frequency on the first test score of the sample test case; and identifying at least one sample test case with a second impact assessment value greater than a second assessment value threshold as a second target test case, wherein the second impact assessment value characterizes the degree of influence of the change in memory frequency on the second test score of the sample test case.

[0071] The first evaluation threshold and the second evaluation threshold are used to filter the first target test cases that are sensitive to changes in CPU frequency and the second target test cases that are sensitive to changes in memory frequency, respectively.

[0072] Optionally, before selecting the first target test case, the variance evaluation value of the sample test cases is calculated based on the CPU frequency change regression equation, and sample test cases with variance evaluation values ​​greater than a preset variance threshold are excluded. Before selecting the second target test case, the variance evaluation value of the sample test cases is calculated based on the memory frequency change regression equation, and sample test cases with variance evaluation values ​​greater than a preset variance threshold are excluded.

[0073] This application embodiment obtains the correlation between sample test cases and changes in CPU frequency or memory frequency by fixing the CPU frequency or memory frequency respectively, and selects two types of target test cases to conduct tests on compute-intensive or memory-intensive loads respectively, thereby improving the targeting of performance testing and further improving the accuracy and effectiveness of testing.

[0074] Based on the above embodiments, after obtaining the first target test case and the second target test case, the load type of the device under test is obtained, wherein the load type is compute-intensive or memory-intensive; if the load type is compute-intensive, the first target test case is executed on the device under test; or, if the load type is memory-intensive, the second target test case is executed on the device under test.

[0075] The load type refers to the performance characteristics exhibited by the main application running on the device under test, and is divided into two categories: compute-intensive and memory-intensive.

[0076] Optionally, the load type of the device under test is known data, which is obtained along with the attribute data of the device under test.

[0077] Optionally, if the load type of the device under test is not specified, the device under test is controlled to run the test case with the highest first influence evaluation value in the first target test case at the nominal memory frequency, according to the preset maximum CPU frequency and the preset minimum CPU frequency, respectively, to obtain a first score corresponding to the preset maximum CPU frequency and a second score corresponding to the preset minimum CPU frequency. The device under test is then controlled to run the test case with the highest second influence evaluation value in the second target test case at the preset maximum memory frequency and the preset minimum memory frequency, respectively, to obtain a third score corresponding to the preset maximum memory frequency and a fourth score corresponding to the preset maximum memory frequency.

[0078] Calculate the difference between the first and second scores, and the difference between the third and fourth scores. If the difference between the first and second scores is greater than the difference between the third and fourth scores, the load type is determined to be compute-intensive; if the difference between the first and second scores is less than the difference between the third and fourth scores, the load type is determined to be memory-intensive.

[0079] Once the load type is obtained, the corresponding first or second target test case can be determined. The first or second target test case is then executed on the device under test to achieve performance testing of the device under test.

[0080] This application embodiment uses target test cases that match the load type to ensure the correlation between the test results and the actual application performance of the device under test, so that the test results can more clearly reflect the main performance of the device under test, and further improve the accuracy and effectiveness of the test.

[0081] Meanwhile, when the load type is unknown, the target test case with the highest impact evaluation value is used to conduct experimental tests on the device under test, quickly obtaining the load type of the device under test. This allows for targeted testing of the device under test using matching target test cases, ensuring testing efficiency.

[0082] In some embodiments, after obtaining the target test case, the second device is tested according to multiple sample test cases to obtain a first average test score corresponding to the multiple sample test cases; the second device is tested according to the target test case to obtain a second average test score corresponding to the target test case; if the relative error between the first average test score and the second average test score is less than a preset error threshold, the target test case is saved as a test case set corresponding to the multiple sample test cases.

[0083] The first device and the second device can be the same device or different devices.

[0084] The first test average score refers to the average test score of multiple sample test cases on the second device. The second test average score refers to the geometric mean of the test scores of the target test cases on the second device.

[0085] The relative error between the average score of the first test and the average score of the second test represents the fidelity of the target test case's performance relative to the performance of multiple sample test cases. Specifically, the relative error is the ratio of the absolute value of the difference between the average score of the first test and the average score of the second test to the average score of the first test.

[0086] The larger the relative error, the worse the fidelity of the target test case's performance compared to the performance of multiple sample test cases; the smaller the relative error, the better the fidelity of the target test case's performance compared to the performance of multiple sample test cases.

[0087] When the relative error is greater than or equal to the preset error threshold, it means that the fidelity of the currently selected target test case exceeds the tolerance value, and the target test case needs to be changed to improve the fidelity.

[0088] Specifically, if the relative error between the average score of the first test and the average score of the second test is greater than or equal to a preset error threshold, the non-target test case with the largest impact evaluation value among multiple sample test cases will be added as the target test case in sequence, and the average score of the second test corresponding to the target test case will be obtained again until the relative error between the average score of the first test and the average score of the second test is less than the preset error threshold.

[0089] The non-target test case with the highest impact assessment value refers to the test case that has not yet been selected as a target test case, but whose impact assessment value is the highest among non-target test cases. Prioritize adding this test case to maximize the performance representation capability of the target test case, reduce the number of times target test cases need to be added, and achieve higher accuracy compared to randomly adding target test cases.

[0090] Optionally, the non-target test case with the highest impact evaluation value can be used to replace the test case with the lowest impact evaluation value in the target test cases; or, a maximum number threshold for the number of target test cases can be set, and after the number of target test cases reaches the maximum number threshold, the non-target test case with the highest impact evaluation value can be used to replace the test case with the lowest impact evaluation value in the target test cases.

[0091] If the target test cases include multiple types of target test cases, then the corresponding target test cases need to be evaluated according to the type of the second device.

[0092] Specifically, the target test cases include a first target test case and a second target test case. The first target test case is used to test compute-intensive load devices, and the second target test case is used to test memory-intensive load devices.

[0093] The second device is tested according to the target test case, and the second average test score corresponding to the target test case is obtained. This includes: obtaining the load type of the second device under test, which is either compute-intensive or memory-intensive; if the load type is compute-intensive, the first target test case is executed on the device under test to obtain the second average test score; or, if the load type is memory-intensive, the second target test case is executed on the device under test to obtain the second average test score.

[0094] Based on the above embodiments, it is also possible to determine whether the currently selected target test case meets the requirements based on the execution time of the target test case.

[0095] Optionally, a first test time is obtained based on the test time used to perform tests on the second device according to multiple sample test cases; a second test time is obtained by obtaining the test time used to perform tests on the second device according to the target test case; if the proportion of the second test time to the first test time is less than a preset proportion threshold, and the relative error between the average score of the first test and the average score of the second test is less than a preset error threshold, then the target test case is saved as a test case set corresponding to multiple sample test cases.

[0096] The proportion of the second test time to the first test time represents the efficiency improvement of executing the target test case compared to executing the sample test case. The smaller the proportion of the second test time to the first test time, the greater the efficiency improvement of executing the target test case; the larger the proportion of the second test time to the first test time, the smaller the efficiency improvement of executing the target test case.

[0097] If the proportion of the second test time to the first test time is greater than or equal to the preset proportion threshold, then the non-target test case with the largest impact evaluation value among multiple sample test cases will be added as the target test case in sequence, and the second test time corresponding to the target test case will be obtained again until the proportion of the second test time to the first test time is less than the preset proportion threshold.

[0098] Specifically, replace the test case with the smallest impact evaluation value among the target test cases with the non-target test case with the largest impact evaluation value; or, set a maximum number threshold for the number of target test cases, and after the number of target test cases reaches the maximum number threshold, replace the test case with the smallest impact evaluation value among the target test cases with the non-target test case with the largest impact evaluation value.

[0099] This application obtains the fidelity of the test effect of the target test case relative to the test effect of multiple sample test cases by calculating the relative error between the average score of the first test and the average score of the second test. When the relative error does not meet the requirements, the target test case is added or replaced, so that the final set of test cases can improve the test efficiency while ensuring that the test effect is close to that of the full set of sample test cases, thus ensuring the reliability of the test results.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0101] Figure 2 This is a schematic diagram of the test case selection device provided in an embodiment of this application. Figure 2As shown, the test case selection device 20 includes a first test module 21, a first calculation module 22, and a first determination module 23. The first test module 21 is used to perform multiple rounds of testing on the first device based on multiple sample test cases, and obtain the test score of each sample test case in each round of testing. The operating parameters of the first device are different in each round of testing, and the test score is used to characterize the speed at which the first device processes tasks when running sample test cases. The first calculation module 22 is used to calculate the impact evaluation value corresponding to each sample test case based on the test score and the operating parameters. The impact evaluation value corresponding to the sample test case is used to characterize the correlation between the sample test case and the operating parameters. The correlation refers to the degree of influence of the change of operating parameters on the test score of the sample test case. The first determination module 23 is used to determine at least one sample test case with an impact evaluation value greater than the evaluation value threshold as the target test case.

[0102] Optionally, the first test module 21 includes a first determining unit and a first calculating unit; the first determining unit is used to determine, for each sample test case, the number of tasks completed per second by the first device when running the sample test case, and the time required to complete a single task; the first calculating unit is used to calculate the test score of the sample test case in the current test based on the reciprocal of the number of tasks completed per second by the first device and / or the time required to complete a single task, and the test score is positively correlated with the reciprocal of the number of tasks completed per second by the first device and / or the time required to complete a single task.

[0103] Optionally, the first calculation module 22 includes a fitting unit and a second determination unit; the fitting unit is used to perform linear fitting for each sample test case, with the running parameters as independent variables and the test score as dependent variables, to obtain the regression equation corresponding to the sample test case; the second determination unit is used to determine that the absolute value of the slope of the regression equation is the impact evaluation value corresponding to the test case.

[0104] Optionally, the first calculation module 22 further includes a third determining unit, used to determine the theoretical score corresponding to each running parameter of the sample test case in multiple rounds of testing according to the regression equation; for each running parameter, calculate the square of the difference between the theoretical score and the test score corresponding to the running parameter to obtain the first reference value corresponding to the running parameter; calculate the square of the difference between the test score corresponding to the running parameter and the average value of the test scores in multiple rounds of testing to obtain the second reference value corresponding to the running parameter; calculate the ratio of the sum of multiple first reference values ​​corresponding to the sample test case to the sum of multiple second reference values ​​to obtain the variation evaluation value of the sample test case; if the variation evaluation value is greater than a preset variation threshold, then determine that the sample test case is not the target test case.

[0105] Optionally, the operating parameters include the central processing unit frequency and the memory frequency, and the test scores include a first test score and a second test score; the first test module 21 includes a first test unit and a second test unit. The first test unit is used to fix the memory frequency of the first device to a first preset frequency, and perform multiple rounds of first performance testing on the first device according to multiple sample test cases. In each round of first performance testing, the central processing unit frequency is adjusted according to a first preset step size, and multiple sample test cases are executed sequentially to obtain the first test scores of multiple sample test cases at different central processing unit frequencies. The second test unit is used to fix the central processing unit frequency of the first device to a second preset parameter, and perform multiple rounds of second performance testing on the first device according to multiple sample test cases. In each round of second performance testing, the memory frequency is adjusted according to a preset second step size, and multiple sample test cases are executed sequentially to obtain the second test scores of multiple sample test cases at different memory frequencies.

[0106] Optionally, the impact assessment value includes a first impact assessment value and a second impact assessment value; the first calculation module 22 is specifically used to calculate the first impact assessment value corresponding to each sample test case based on the first test scores of multiple sample test cases at different CPU frequencies and the CPU frequency of the first device in each round of the first performance test, wherein the first impact assessment value characterizes the degree of influence of the change in CPU frequency on the first test score of the sample test case; and to calculate the second impact assessment value corresponding to each sample test case based on the second test scores of multiple sample test cases at different memory frequencies and the memory frequency of the first device in each round of the second performance test, wherein the second impact assessment value characterizes the degree of influence of the change in memory frequency on the second test score of the sample test case.

[0107] Optionally, the impact assessment value includes a first impact assessment value and a second impact assessment value; the assessment value threshold includes a first assessment value threshold and a second assessment value threshold. The first determining module 23 is specifically used to determine at least one sample test case whose first impact assessment value is greater than the first assessment value threshold as a first target test case, wherein the first impact assessment value characterizes the degree of influence of the change in central processing unit frequency on the first test score of the sample test case; and to determine at least one sample test case whose second impact assessment value is greater than the second assessment value threshold as a second target test case, wherein the second impact assessment value characterizes the degree of influence of the change in memory frequency on the second test score of the sample test case.

[0108] Optionally, the test case selection device includes a second test module, used to obtain the load type of the device under test, which is either compute-intensive or memory-intensive; if the load type is compute-intensive, then execute the first target test case on the device under test; or, if the load type is memory-intensive, then execute the second target test case on the device under test.

[0109] Optionally, the test case selection device includes an evaluation module, used to perform tests on the second device based on multiple sample test cases and obtain a first average test score corresponding to the multiple sample test cases; perform tests on the second device based on a target test case and obtain a second average test score corresponding to the target test case; if the relative error between the first average test score and the second average test score is less than a preset error threshold, then the target test case is saved as a test case set corresponding to the multiple sample test cases.

[0110] Optionally, the evaluation module is also used to obtain a first test time based on the test time used to perform tests on the second device according to multiple sample test cases; obtain a second test time based on the test time used to perform tests on the second device according to the target test case; if the proportion of the second test time to the first test time is less than a preset proportion threshold, and the relative error between the average score of the first test and the average score of the second test is less than a preset error threshold, then the target test case is saved as a test case set corresponding to multiple sample test cases.

[0111] Optionally, the target test cases include a first target test case and a second target test case. The first target test case is used to test compute-intensive load devices, and the second target test case is used to test memory-intensive load devices. The evaluation module is also used to obtain the load type of the second device under test, which is either compute-intensive or memory-intensive. If the load type is compute-intensive, the first target test case is executed on the device under test to obtain the second average test score; or, if the load type is memory-intensive, the second target test case is executed on the device under test to obtain the second average test score.

[0112] Optionally, the evaluation module is also used to, if the relative error between the first test average score and the second test average score is greater than or equal to a preset error threshold, sequentially add the non-target test case with the largest impact evaluation value among multiple sample test cases as target test cases, and re-obtain the second test average score corresponding to the target test case, until the relative error between the first test average score and the second test average score is less than the preset error threshold.

[0113] For a description of the features in the embodiment corresponding to the test case selection device, please refer to the relevant description in the embodiment corresponding to the test case selection method, which will not be repeated here.

[0114] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described test case selection method embodiments.

[0115] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device provided in this embodiment of the application can execute the processing flow provided in the kernel upgrade method embodiment, such as... Figure 3 As shown, the electronic device 30 includes: a memory 31, a processor 32, a computer program, and a communication interface 33; wherein the computer program is stored in the memory 31 and configured to be executed by the processor 32 using the test case selection method described above. In an exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0116] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described test case selection method embodiments.

[0117] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described test case selection method embodiments.

[0118] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] The foregoing has provided a detailed description of a test case selection method, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A test case selection method, characterized by, The method comprises the following steps: performing multiple rounds of testing on a first device according to multiple sample test cases, obtaining a test score of each sample test case in each round of testing, wherein the operating parameters of the first device are different in each round of testing, and the test score is used to represent the speed of the first device in processing a task when running the sample test case; calculating an influence degree evaluation value corresponding to each sample test case according to the test score and the operating parameters, wherein the influence degree evaluation value corresponding to the sample test case is used to represent the correlation between the sample test case and the operating parameters, and the correlation refers to the influence degree of the change of the operating parameters on the test score of the sample test case; determining at least one sample test case with an influence degree evaluation value greater than an evaluation value threshold as a target test case; calculating an influence degree evaluation value corresponding to each sample test case according to the test score and the operating parameters, comprising: performing linear fitting on the operating parameters as independent variables and the test score as dependent variables for each sample test case to obtain a regression equation corresponding to the sample test case; determining the absolute value of the slope of the regression equation as the influence degree evaluation value corresponding to the test case; after obtaining the regression equation corresponding to the sample test case, the method further comprises determining a theoretical score corresponding to each operating parameter in the multiple rounds of testing according to the regression equation; calculating the square of the difference between the theoretical score corresponding to each operating parameter and the test score to obtain a first reference value corresponding to the operating parameter; calculating the square of the difference between the test score corresponding to each operating parameter and the average value of the test scores in the multiple rounds of testing to obtain a second reference value corresponding to the operating parameter; calculating the ratio of the sum of the first reference values corresponding to the sample test case to the sum of the second reference values to obtain a variation evaluation value of the sample test case; if the variation evaluation value is greater than a preset variation threshold, determining that the sample test case is not the target test case.

2. The test case selection method of claim 1, wherein, The method of performing multiple rounds of testing on a first device according to multiple sample test cases and obtaining a test score of each sample test case in each round of testing comprises: for each sample test case, determining the number of tasks completed per second and the time required to complete a single task when the first device runs the sample test case; calculating the test score of the sample test case in the current test according to the number of tasks completed per second and / or the inverse of the time required to complete a single task of the first device, wherein the test score is positively correlated with the number of tasks completed per second and / or the inverse of the time required to complete a single task of the first device.

3. The test case selection method of claim 1, wherein, The operating parameters include central processor frequency and memory frequency, and the test score includes a first test score and a second test score; the method of performing multiple rounds of testing on a first device according to multiple sample test cases and obtaining a test score of each sample test case in each round of testing comprises: The memory frequency of the first device is fixed as a first preset frequency, and a plurality of rounds of first performance tests are performed on the first device according to the plurality of sample test cases. In each round of the first performance tests, the plurality of sample test cases are executed in sequence after the central processing unit frequency is adjusted according to a first preset step size, and first test scores of the plurality of sample test cases under different central processing unit frequencies are obtained. The central processing unit frequency of the first device is fixed as a second preset parameter, and a plurality of rounds of second performance tests are performed on the first device according to the plurality of sample test cases. In each round of the second performance tests, the plurality of sample test cases are executed in sequence after the memory frequency is adjusted according to a preset second step size, and second test scores of the plurality of sample test cases under different memory frequencies are obtained.

4. The test case selection method of claim 3, wherein, The influence degree evaluation values include first influence degree evaluation values and second influence degree evaluation values. The influence degree evaluation values corresponding to each of the sample test cases are calculated according to the test scores and the running parameters, and include: The first influence degree evaluation values corresponding to each of the sample test cases are calculated according to the first test scores of the plurality of sample test cases under different central processing unit frequencies and the central processing unit frequency of the first device in each round of the first performance tests. The first influence degree evaluation values represent the influence degree of the change of the central processing unit frequency on the first test scores of the sample test cases. The second influence degree evaluation values corresponding to each of the sample test cases are calculated according to the second test scores of the plurality of sample test cases under different memory frequencies and the memory frequency of the first device in each round of the second performance tests. The second influence degree evaluation values represent the influence degree of the change of the memory frequency on the second test scores of the sample test cases.

5. The test case selection method of claim 3, wherein, The influence degree evaluation values include first influence degree evaluation values and second influence degree evaluation values. The evaluation value thresholds include first evaluation value thresholds and second evaluation value thresholds. The at least one sample test case whose influence degree evaluation value is greater than the evaluation value threshold is determined as a target test case, and includes: The at least one sample test case whose first influence degree evaluation value is greater than the first evaluation value threshold is determined as a first target test case. The first influence degree evaluation value represents the influence degree of the change of the central processing unit frequency on the first test scores of the sample test cases. The at least one sample test case whose second influence degree evaluation value is greater than the second evaluation value threshold is determined as a second target test case. The second influence degree evaluation value represents the influence degree of the change of the memory frequency on the second test scores of the sample test cases.

6. The test case selection method of claim 5, wherein, The method further includes: Obtaining the load type of the device to be tested. The load type is computation-intensive or memory-intensive. If the load type is computation-intensive, the first target test case is executed on the device to be tested; or If the load type is memory-intensive, the second target test case is executed on the device to be tested.

7. The test case selection method of claim 1, wherein, The method further includes: According to the plurality of sample test cases, test the second device to obtain a first test average score corresponding to the plurality of sample test cases; According to the target test case, test the second device to obtain a second test average score corresponding to the target test case; If the relative error between the first test average score and the second test average score is less than a preset error threshold, the target test case is saved as a test case subset corresponding to the plurality of sample test cases.

8. The test case selection method of claim 7, wherein, After obtaining the second test average score corresponding to the target test case, the method further comprises: According to the test time used for testing the second device according to the plurality of sample test cases, a first test time is obtained; According to the test time used for testing the second device according to the target test case, a second test time is obtained; If the proportion of the second test time in the first test time is less than a preset proportion threshold, and the relative error between the first test average score and the second test average score is less than a preset error threshold, the target test case is saved as a test case subset corresponding to the plurality of sample test cases.

9. The test case selection method of claim 7, wherein, The target test case includes a first target test case and a second target test case, the first target test case is used for testing a computation-intensive load device, and the second target test case is used for testing a memory-intensive load device; According to the target test case, test the second device to obtain a second test average score corresponding to the target test case, comprising: Obtain the load type of the second device to be tested, the load type being computation-intensive or memory-intensive; If the load type is computation-intensive, execute the first target test case on the device to be tested to obtain the second test average score; or If the load type is memory-intensive, execute the second target test case on the device to be tested to obtain the second test average score.

10. The test case selection method of claim 7, wherein, The method further comprises: If the relative error between the first test average score and the second test average score is greater than or equal to a preset error threshold, the non-target test case with the maximum impact degree evaluation value in the plurality of sample test cases is added as a target test case in turn, and the second test average score corresponding to the target test case is re-obtained until the relative error between the first test average score and the second test average score is less than a preset error threshold.

11. An electronic device, comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the test case selection method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the test case selection method according to any one of claims 1 to 10.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the test case selection method according to any one of claims 1 to 10.

Images