Test case execution method and device, electronic equipment and readable storage medium

Abstract

The invention is suitable for the technical field of software testing, and provides a test case execution method and device, electronic equipment and a computer readable storage medium, and the method comprises the following steps: obtaining a test case of a to-be-tested system; the test case is set according to a test plan of the to-be-tested system, and the test plan is used for indicating a test demand of the to-be-tested system; the test case is executed according to a pre-constructed controller, and the pre-constructed controller at least comprises a parallel controller; under the condition that the test case is successfully executed, adjusting the resource consumption of the test case according to the execution result of the test case, and executing the test case of which the resource consumption is adjusted to obtain a new execution result; according to the new execution result, a performance optimization model of the to-be-tested system is constructed, and the performance optimization model is used for indicating the optimization direction of the to-be-tested system. The test efficiency of the software system can be improved.

Description

Technical Field

[0001] The present application belongs to the field of software testing technology, and in particular, relates to a test case execution method, device, electronic device and computer-readable storage medium. Background Art

[0002] In the field of software system development, the performance and reliability of the software system are crucial. In order to ensure that the software system can run stably under various loads and usage scenarios, effective testing is required. At present, the corresponding test cases are mainly set based on the test requirements of the software system, and then the performance test is completed by executing the test cases.

[0003] However, with the increasing complexity and expansion of software systems, the traditional serial test case execution method has been unable to meet the needs of efficient testing. When faced with large-scale, high-concurrency testing requirements, serial testing often cannot be flexibly adjusted, resulting in the execution of test cases taking a lot of time, seriously affecting the software development and optimization process. Summary of the invention

[0004] The embodiments of the present application provide a test case execution method, device and electronic device, which can improve the testing efficiency of the software system.

[0005] In a first aspect, an embodiment of the present application provides a test case execution method, comprising:

[0006] Acquire test cases for the system to be tested; the test cases are set according to a test plan for the system to be tested, and the test plan is used to indicate test requirements for the system to be tested;

[0007] Executing the test case according to a pre-built controller, wherein the pre-built controller includes at least a parallel controller;

[0008] If the test case is successfully executed, adjusting the resource usage of the test case according to the execution result of the test case, executing the test case after adjusting the resource usage, and obtaining a new execution result;

[0009] A performance optimization model of the system to be tested is constructed according to the new execution result, wherein the performance optimization model is used to indicate an optimization direction of the system to be tested.

[0010] In a second aspect, an embodiment of the present application provides a test case execution device, comprising:

[0011] A test case acquisition module, used to acquire test cases for the system to be tested; the test cases are set according to a test plan for the system to be tested, and the test plan is used to indicate the test requirements of the system to be tested;

[0012] A test case execution module, configured to execute the test case according to a pre-built controller, wherein the pre-built controller includes at least a parallel controller;

[0013] A test case adjustment module, used to adjust the resource usage of the test case according to the execution result of the test case when the test case is successfully executed, and execute the test case after the resource usage is adjusted to obtain a new execution result;

[0014] A performance optimization model building module is used to build a performance optimization model of the system to be tested according to the new execution result, wherein the performance optimization model is used to indicate the optimization direction of the system to be tested.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the test case execution method described in the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the test case execution method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the test case execution method described in any one of the first aspects above.

[0018] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0019] In the embodiment of the present application, the test case is executed by a pre-built controller, and then the resource usage of the test case is adjusted according to the execution result of the test case, and then the performance optimization model of the system to be tested is constructed according to the new execution result, so that the test efficiency of the system to be tested can be improved. Specifically, since the pre-built controller includes at least a parallel controller, the test case can be executed in a parallel execution mode, thereby shortening the test time and improving the test efficiency. Moreover, adjusting the resource usage of the test case according to the execution result of the test case means that the subsequent resource usage of the test case can be adjusted by the current execution result of the test case, so that the test process of the test case can be more matched with the test requirements indicated in the test plan. In addition, since the performance optimization model can indicate the optimization direction of the system to be tested, after the performance optimization model of the system to be tested is constructed according to the new execution result, the performance of the system to be tested can be directly optimized according to the performance optimization model, so that the test process and optimization process of the software system can be combined, further improving the test efficiency of the software system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flowchart of a test case execution method provided by an embodiment of the present application;

[0022] Figure 2 It is a structural diagram of a test case execution device provided in an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0025] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0026] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0028] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0030] Currently, in the development and testing of software systems, performance testing of software systems is usually performed by using performance testing tools to execute test cases. For example, JMeter, as a widely used performance testing tool, can test the performance of software systems by serially executing JMeter test cases.

[0031] However, as software systems become increasingly complex and their scale continues to expand, the load conditions and usage scenarios of software systems are also constantly changing. When facing large-scale, high-concurrency software testing, traditional serial testing methods will have problems such as inability to flexibly adjust testing strategies and unreasonable resource allocation, and cannot meet the load requirements and usage scenario requirements of software systems, resulting in low testing efficiency of software systems. In addition, traditional serial testing methods often cause the testing process to consume a lot of time due to low testing efficiency, which seriously affects the development and optimization process of software systems.

[0032] Therefore, the above test case execution method may have the problem of low testing efficiency of the software system.

[0033] In order to improve the testing efficiency of the software system, the present application provides a test case execution method based on parallel execution, in which the test case is executed by a pre-built controller, wherein the pre-built controller includes at least a parallel controller, and when the above test case is successfully executed, the resource usage of the test case is adjusted according to the execution result of the test case, the test case with adjusted resource usage is executed to obtain a new execution result, and then a performance optimization model of the system to be tested is constructed according to the new execution result, wherein the performance optimization model is used to indicate the optimization direction of the system to be tested.

[0034] Figure 1 A flow chart of a test case execution method provided by an embodiment of the present application is shown, which is described in detail as follows:

[0035] S11, obtaining test cases for the system to be tested; the test cases are set according to a test plan for the system to be tested, and the test plan is used to indicate test requirements for the system to be tested.

[0036] The test plan is a test framework for indicating the entire test process of the system to be tested, and may include basic information such as test objectives, test scope, test environment, etc. The test objectives and test scope may indicate the test requirements of the system to be tested, and the test environment may indicate the test scenario when the test case is executed.

[0037] For example, the above-mentioned test objectives can be at least one of the following: verifying whether the response time of the system to be tested under high concurrency conditions meets the performance requirements, evaluating the stability and reliability of the system to be tested in processing specific business processes, determining the maximum number of concurrent users that the system to be tested can withstand, etc.; the above-mentioned test scope can indicate testing of specific functional modules or specific interfaces of the system to be tested, or it can indicate testing of the main business processes of the entire system to be tested. Of course, it can also indicate that only the key business of the system to be tested is stress tested; the above-mentioned test environment can indicate the current test hardware configuration (such as CPU, memory, storage, etc.), network environment (such as bandwidth, delay, etc.), and software configuration (such as versions of operating system, middleware, database and other software).

[0038] Specifically, assuming that the JMeter testing tool is used for testing, an Excel document containing test cases can be obtained, wherein the test cases in the Excel document correspond to the test plan of the system to be tested, and may include different types of test cases, for example, test cases for testing the query interface, test cases for testing the login function, and the like.

[0039] S12. Execute the test case according to a pre-built controller, wherein the pre-built controller at least includes a parallel controller.

[0040] The parallel controller is used to manage and coordinate the control structure of parallel execution of multiple test cases, and the control structure includes at least one of the following: distributed structure, master-slave mode structure, task queue, token ring structure, hierarchical structure, etc. Corresponding to the control structure, the parallel controller includes at least one of the following: distributed structure controller, master-slave mode structure controller, task queue controller, token ring structure controller, hierarchical structure controller, etc.

[0041] In an optional embodiment, a controller may be pre-constructed according to a test plan. For example, when the number of test cases is greater than 1, a parallel controller may be constructed so as to execute the above-mentioned multiple test cases according to the control structure of the parallel controller.

[0042] In the embodiment of the present application, the above test cases are executed in parallel through the above parallel controller, so that multiple test cases can be executed simultaneously, thereby improving the test efficiency of the system to be tested.

[0043] It should be noted that the above controller may also include a special controller configured according to the test requirements of the system to be tested, such as a loop controller, a condition controller, etc. Through the above controller, a more flexible test logic can be implemented.

[0044] S13. When the test case is successfully executed, adjust the resource usage of the test case according to the execution result of the test case, execute the test case after adjusting the resource usage, and obtain a new execution result.

[0045] The execution result of the test case is used to reflect the execution status of the test case, for example, the resource usage of the current test case, etc. The resource usage refers to the resource usage that can affect the test process of the test case, for example, the resource usage may include one or more of CPU resources, storage resources, network resources, concurrency, etc. when executing the test case.

[0046] Specifically, the new resource usage can be determined based on the execution result of the test case and the test case information, and then the test case is re-executed to obtain a new execution result after the new resource usage is determined. The test case information includes one or more of the type, priority, and quantity of the test case.

[0047] In an embodiment of the present application, the resource usage of the above test cases is adjusted by the execution results of the test cases. The resource usage of the test cases can be dynamically adjusted during the test process so that the test cases are more closely matched with the test requirements indicated in the above test plan to achieve better testing results.

[0048] S14. Constructing a performance optimization model of the system to be tested according to the new execution result, wherein the performance optimization model is used to indicate an optimization direction of the system to be tested.

[0049] The performance optimization model may be a model in the form of a function. For example, the performance optimization model may be expressed in the form of: optimization index = function (parameter), wherein the parameters are parameters reflecting the test process of the system to be tested (e.g., parameters corresponding to the controller, parameters corresponding to resource usage, etc.), the function (parameter) is a preset function relationship, and the optimization index may reflect the performance status of the system to be tested.

[0050] Specifically, different parameters can be selected and combined from the test process of the system to be tested according to the above new execution results, and then the optimization index corresponding to the parameters under the combination can be calculated through a preset functional relationship, and then the optimal parameter combination and the size of each parameter under the parameter combination can be determined according to whether the change of the above optimization index meets the preset index requirements. The optimization direction of the system to be tested can be indicated by the above parameter combination and the change of the parameter size in the parameter combination.

[0051] In the embodiments of the present application, the above-mentioned performance optimization model can be used to better evaluate the performance of the system to be tested under different parameter settings, thereby adapting to different test scenarios and test requirements. At the same time, by constructing the above-mentioned performance optimization model, a unified measurement standard and method for performance optimization is also provided, which can better indicate the optimization direction of the system to be tested and achieve continuous performance improvement.

[0052] For example, assuming that the above functional relationship is a linear function, the above performance optimization model is: optimization index = linear function (parameter). If optimization index 1 is greater than optimization index 2, optimization index 1 = linear function (parameter 1, parameter 2), optimization index 2 = linear function (parameter 1, parameter 3), indicating that the parameter combination of parameter 1 and parameter 3 can better reflect the performance status of the system to be tested, and then continue to adjust the size of parameter 1 and / or parameter 3 until the size of optimization index 1 meets the preset requirements.

[0053] In the embodiment of the present application, the test case is executed by a pre-built controller, and then the resource usage of the test case is adjusted according to the execution result of the test case, and then the performance optimization model of the system to be tested is constructed according to the new execution result, so that the test efficiency of the system to be tested can be improved. Specifically, since the pre-built controller includes at least a parallel controller, the test case can be executed in a parallel execution mode, thereby shortening the test time and improving the test efficiency. Moreover, adjusting the resource usage of the test case according to the execution result of the test case means that the subsequent resource usage of the test case can be adjusted by the current execution result of the test case, so that the test process of the test case can be more matched with the test requirements indicated in the test plan. In addition, since the performance optimization model can indicate the optimization direction of the system to be tested, after the performance optimization model of the system to be tested is constructed according to the new execution result, the performance of the system to be tested can be directly optimized according to the performance optimization model, so that the test process and optimization process of the software system can be combined, further improving the test efficiency of the software system.

[0054] In another optional embodiment of the present application, before executing the above test case according to the pre-built controller, the method further includes:

[0055] According to the above test plan, a corresponding thread group is set for the above system to be tested;

[0056] A specific controller and the parallel controller are set under the thread group; the specific controller is a controller set according to the test plan.

[0057] Wherein, the above-mentioned thread group is a set composed of multiple threads, each thread can be regarded as a virtual user, and in the test process, the thread group can simulate the requests or behaviors of multiple users. The above-mentioned specific controller may include a loop controller (Loop Controller) and / or a condition controller (If Controller), etc. The above-mentioned loop controller may be set according to the test target, test scope, etc. in the test plan. For example, when there is a loop execution operation in a specific business process corresponding to the test target or test scope, a loop controller may be set; the above-mentioned condition controller may be set according to the test environment in the test plan. For example, if the test plan includes multiple test environments, each test environment corresponds to different test cases, then a corresponding number of condition controllers may be set. Assuming that condition controller 1 corresponds to test environment 1, condition controller 1 will execute the corresponding test case when test environment 1 is detected. Of course, the number of the above-mentioned loop controller and / or condition controller may also be adjusted according to the test plan.

[0058] Specifically, before executing the above test case, one or more thread groups can be set for the system to be tested according to the test plan (for example, when the test plan includes multiple specific business processes, a corresponding number of thread groups can be set), and then the parameters corresponding to the above thread groups are set. The parameters corresponding to the above thread groups include: the type of thread group (for example, a fixed number of thread groups or an automatically expanded number of thread groups), the number of thread groups, the number of threads in the thread group, the start time of the thread group, etc. For any thread group, a specific controller can be set based on the parallel controller, and then the controller parameters in the specific controller can be configured. For example, when the parallel controller is a task queue controller, a corresponding specific controller can be added, and controller parameters such as queue length (for example, set to 5 requests), timeout time (for example, set to 30 seconds), and data processing frequency (for example, set to 10 times / second) can be configured. By adjusting the above controller parameters, the specific business logic or business characteristics corresponding to the system to be tested can be precisely controlled and tested, thereby controlling the test process more flexibly.

[0059] In an optional embodiment of the present application, when the type of the thread group is a fixed number of thread groups, the number of threads in the thread group can be calculated according to the following formula: number of threads = expected concurrency / (average processing capacity of a single thread×preset adjustment coefficient).

[0060] In the embodiment of the present application, thread groups and controllers (including specific controllers and parallel controllers) are configured through test plans, which can more comprehensively cover various test scenarios and better adapt to various specific needs and complex situations.

[0061] After the controller is constructed, the test case is executed according to the pre-constructed controller, including:

[0062] The above test cases are executed according to the above specific controller and the above parallel controller.

[0063] Specifically, after the specific controller and the parallel controller are set, the test cases may be directly executed through the test logic included in the specific controller and the parallel controller.

[0064] In the embodiment of the present application, since the above-mentioned thread group, specific controller and parallel controller are set according to the test plan, the above-mentioned test cases are executed through the test logic included in the above-mentioned specific controller and the above-mentioned parallel controller, which can better adapt to the test requirements.

[0065] In some embodiments, the resource usage includes available resources and / or concurrency, wherein the available resources refer to the amount of system available resources allocated to the test case, which may include CPU resources, storage resources, network resources, etc., and the concurrency refers to the number of test cases that can be processed simultaneously by the system at a certain moment. In order to further improve the accuracy of resource usage adjustment, the resource usage of the test case is adjusted according to the execution result of the test case, including:

[0066] Determine performance indicators based on the execution results of the above test cases, and perform test analysis on the above performance indicators;

[0067] Based on the analysis results of the above performance indicators, the available resource amount of the above test case is re-determined, and / or the concurrency of the above system to be tested is re-determined to obtain new available resource amount and / or new concurrency.

[0068] The above performance indicators may include at least one of the following: response time, throughput, current available resources, resource utilization (such as CPU utilization, memory utilization, etc.), network traffic, etc.

[0069] Specifically, one or more performance indicators can be determined through the execution results of the test case, and then the determined performance indicators can be tested and analyzed using a preset analysis method. The new available resources and / or new concurrency can be re-determined based on the analysis results of the performance indicators and a preset resource usage adjustment formula.

[0070] The above-mentioned preset analysis method may include at least one of the following: mean analysis method, variance analysis method, linear regression analysis method, etc. The above-mentioned mean analysis method may determine the mean by the following formula: in, represents the mean of a certain performance indicator, n represents the number of tests, x iIt represents the result of the performance indicator of the i-th test of a certain performance indicator, i≤n. For example, assuming that 5 tests are conducted and the results of the performance indicators are 10, 12, 8, 11, and 9 respectively, then the mean is (10+12+8+11+9) / 5=10. By calculating the mean of the performance indicator, a value representing the overall level of these test results can be obtained, which helps to have a general understanding of the overall performance. The above variance analysis method can determine the variance through the following formula: Among them, S 2 The variance of a certain performance indicator can be used to measure the degree of dispersion or fluctuation of the test, thereby reflecting the density of the test result distribution and the degree of deviation from the mean, which is helpful to understand the stability and consistency of the test; the linear regression analysis method includes: determining the linear regression coefficient by the least square method and other methods, and then constructing a linear regression equation based on the linear regression coefficient, taking the performance indicator as an independent variable or a dependent variable, and performing regression analysis on the performance indicator through the linear regression equation, so as to evaluate the correlation between different performance indicators. For example, suppose that a linear regression equation is constructed: y = a + bx, where a and b are linear regression coefficients, x represents the independent variable, y represents the dependent variable, the independent variable x can be the user load (for example, the number of users online at the same time), the request frequency or the number of users, and the dependent variable y can be the system response time, throughput or resource utilization (such as CPU utilization, etc.). Through the above linear regression analysis method, the degree of influence of one variable on another variable can be determined, so as to more accurately analyze the trend, predict and make decisions on the test results.

[0071] Optionally, the preset resource usage adjustment formula may include an available resource amount adjustment formula and a concurrency amount adjustment formula, wherein the available resource amount adjustment formula may be: the new available resource amount allocated to a test case = (the priority coefficient of the test case × the current available resource amount of the system) / the sum of the priority coefficients of all test cases. The specific resource amount to be allocated to each test case can be calculated by the priority coefficient of each test case and the current actual available resources of the system through the above formula, and then the corresponding resource allocation operation is performed to realize dynamic and accurate resource allocation according to the priority and real-time resource conditions, thereby avoiding performance bottlenecks caused by resource waste or unreasonable allocation. Bottlenecks or test failures enhance the flexibility and adaptability of the test method, wherein the priority coefficient can be determined based on factors such as business importance, user impact, usage frequency, correlation with recent updates, and fault impact scope; the concurrency adjustment formula can be: new concurrency = current concurrency × (value of current performance indicator / preset performance threshold), so that the concurrency is automatically adjusted according to real-time conditions, avoiding possible waste or shortage of resources caused by fixed concurrency, and dynamically finding the concurrency that best suits the current system status, making the test process more efficient and adaptable to different system states and load conditions. In addition, there is no need to frequently adjust the concurrency manually, which also reduces operational complexity and labor costs.

[0072] In another optional embodiment of the present application, after executing the above test case according to the pre-built controller, the method further includes:

[0073] In the event that the test case fails to execute, a recovery strategy is determined according to the execution result of the test case, and the test case is re-executed using the determined recovery strategy.

[0074] Specifically, when a test case fails to execute, the execution result of the test case may include the error type, error frequency, etc., and then a recovery strategy may be determined based on the above error type and / or error frequency, and the test case may be skipped and / or re-executed using the above recovery strategy.

[0075] In an embodiment of the present application, a recovery strategy is determined by the execution results of the test case, and the test case is then re-executed using the recovery strategy. This can reduce the impact of test interruptions or errors caused by execution failures, avoid unnecessary long waits, and promptly resume the execution of the test case to ensure the continuity and stability of the test.

[0076] In some embodiments, determining a recovery strategy according to the execution result of the test case and re-executing the test case using the determined recovery strategy includes:

[0077] Determine the recovery time and number of retries of the above test cases according to the execution results of the above test cases;

[0078] Re-execute the above test case based on the above recovery time and number of retries.

[0079] Specifically, when the above execution results include error type and error frequency, the recovery time in the recovery strategy can be determined using the preset recovery time formula: recovery time = error severity coefficient × average processing time, and the number of retries in the recovery strategy can be determined according to the preset retry number formula: number of retries = error occurrence frequency coefficient / set constant, wherein the error severity coefficient in the above recovery time formula is determined according to the above error type, and the error occurrence frequency coefficient in the above retry number formula is determined according to the above error frequency.

[0080] Optionally, the correspondence between the error severity coefficient and the error type, and the correspondence between the error frequency coefficient and the error frequency can be pre-set. After determining the error type and error frequency of the test case, the error severity coefficient and the error frequency coefficient are determined according to the correspondence. For example, the correspondence between the error type and the error severity coefficient can be: the error type is a network connection timeout error, the severity coefficient is 0.8; the error type is a database query failure error, the severity system is 0.7; the error type is a page loading exception error, and the severity coefficient is 0.6. The correspondence between the error frequency coefficient and the error frequency can be: the error frequency is more than 10 times per hour, the error frequency coefficient is 0.9; the error frequency is 5-10 times per hour, the error frequency coefficient is 0.7; the error frequency is 1-5 times per hour, the error frequency coefficient is 0.5; the error frequency is less than 1 time per hour, and the error frequency coefficient is 0.3.

[0081] In the embodiment of the present application, the recovery time and number of retries of the test case are determined according to the error type and error frequency, and the error situation can be handled automatically. When an error occurs, the test can be continued as much as possible by arranging an appropriate recovery time and number of retries, thereby better simulating the fault tolerance capability in actual applications and better evaluating the response performance and recovery capability of the system to be tested when an error occurs.

[0082] In some embodiments, the step of constructing the performance optimization model of the system to be tested according to the new execution result includes:

[0083] Determine a performance parameter according to the new execution result, wherein the performance parameter includes at least one of the following: a parameter corresponding to the thread group, a parameter corresponding to the resource usage, and a parameter corresponding to the controller;

[0084] The above performance parameters are used to construct a performance optimization model of the above system to be tested.

[0085] Specifically, assuming that the above performance optimization model is a model in the form of a function, at least one parameter can be selected from the parameters corresponding to the thread group, the parameters corresponding to the resource usage, and the parameters corresponding to the controller according to the above new execution results and the test plan, and the performance optimization model of the system to be tested can be constructed according to the above parameter combination. Among them, the performance optimization model can be: optimization index = function (parameters corresponding to the thread group (T), and / or, parameters corresponding to the resource usage (R), and / or, parameters corresponding to the controller (C), etc.).

[0086] Optionally, when selecting the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage, assuming that the optimization index is linearly related to the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage, the performance optimization model can be: optimization index = aT + bR + c, where a, b, and c are preset coefficients; assuming that the optimization index is quadratically related to the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage, the performance optimization model can be: optimization index = aT^2+bT+cR^2+dR+e, where a, b, c, d, and e are preset coefficients; assuming that the optimization index is exponentially related to the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage, the performance optimization model can be: optimization index = exp(aT+bR), where a and b are preset coefficients. When selecting parameters corresponding to the thread group (T), parameters corresponding to resource usage (R) and parameters corresponding to the controller (C), the corresponding performance optimization model can be a performance optimization model in the form of a complex combination: optimization index = aTR + bTC + cRC, where a, b, and c are preset coefficients.

[0087] It should be noted that the above linear relationship is applicable to relatively simple systems or scenarios. If it is determined through new execution results and test plans that the test process only involves some basic performance tests (such as concurrency tests), or the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage have a relatively direct and linear impact on the optimization index, then the optimization performance model of the above linear relationship can be constructed based on the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage. When it is found that with the changes in the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage, the optimization index shows a certain nonlinear growth or decreasing trend, and there are some quadratic term characteristics, then the optimization performance model of the above quadratic relationship can be constructed based on the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage. For example, after the available resources of the system are allocated to a certain extent, the performance index improvement brought about by further increasing the resources is no longer linear, but shows the characteristics of a quadratic curve. When it is found that with the changes in the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage, the optimization index shows exponential growth or decay, then the optimization performance model of the above exponential relationship can be constructed based on the parameters (T) corresponding to the thread group and the parameters (R) corresponding to the resource usage. When there are strong interactions and complex correlations between the parameters corresponding to the thread group (T), the parameters corresponding to the resource usage (R), and the parameters corresponding to the controller (C) during the test, the optimization performance model in the form of the above complex combination can be constructed based on the complex combination of the parameters corresponding to the thread group (T), the parameters corresponding to the resource usage (R), and the parameters corresponding to the controller (C). For example, in some highly complex distributed systems or scenarios with multiple types of controllers and resource interactions, simple linear or quadratic relationships cannot accurately reflect the actual situation, and a complex combination optimization performance model can be constructed.

[0088] Exemplarily, the parameters (T) corresponding to the above-mentioned thread group may include the thread group type, the number of threads in the thread group, etc., the parameters (R) corresponding to the above-mentioned resource usage may include the CPU allocation amount, the memory allocation amount, etc., and the parameters (C) corresponding to the above-mentioned controller may include the queue length, data processing speed, timeout time, data processing frequency, etc. Assume that after determining the performance optimization model, it is found through the performance optimization model that when the number of threads in the parameters corresponding to the thread group increases, the optimization index decreases significantly, which indicates that the current system may not be able to withstand too many concurrent threads, then the number of threads can be reduced in a targeted manner to optimize the performance; or, it is found that when the allocation amount of a specific resource (such as memory) in the parameters corresponding to the resource usage is small, the optimization index will be seriously reduced, then the allocation amount of the resource can be increased; for example, from the performance optimization model, it can be seen that the parameters corresponding to a specific controller lead to a large number of performance bottlenecks, such as the unreasonable settings of the controller of a queuing mechanism, resulting in a large number of request backlogs, then the relevant parameters of the controller, such as queue length, processing speed, etc., can be adjusted in a targeted manner to improve performance; or, through the optimization indicators corresponding to the performance optimization model, it is found that the use of a certain type of thread group (such as a fixed number of thread groups) does not match the current system architecture, resulting in poor performance, and it can be switched to other more suitable thread group types (such as automatically expanding the number of thread groups) for optimization.

[0089] In the embodiment of the present application, the optimization index corresponding to the above performance optimization model can reflect or predict the performance under different parameter settings, providing a basis and direction for the decision optimization of the system to be tested. At the same time, the performance optimization model can be continuously improved and adjusted to adapt to different scenarios and needs, achieve continuous performance improvement, and provide a unified measurement standard and method for the performance optimization of the system to be tested.

[0090] In another optional embodiment of the present application, after determining the performance indicator according to the execution result of the above test case, the method further includes:

[0091] The performance indicators are monitored using preset monitoring elements to obtain visual monitoring results corresponding to the performance indicators, wherein the monitoring elements correspond to the performance indicators one-to-one.

[0092] Specifically, in order to obtain key performance data during the test process in real time, the performance indicators that need to be monitored can be determined according to the test plan, and then the corresponding monitoring elements can be set. The monitoring results corresponding to each performance indicator are generated and visually displayed according to the monitoring elements to obtain the above-mentioned visual monitoring results. For example, assuming that the performance indicator is resource utilization, its corresponding monitoring element may include a CPU utilization monitoring element, a memory utilization monitoring element, etc. The corresponding CPU utilization and memory utilization can be displayed in real time through the above-mentioned CPU utilization monitoring element and memory utilization monitoring element. Of course, the CPU utilization and memory utilization over a period of time can also be counted and displayed through a line graph, a tree diagram, etc., and the form of visual display is not limited here.

[0093] In the embodiment of the present application, by monitoring the performance indicators through the above-mentioned monitoring elements, the operating status and performance of the system to be tested can be grasped in real time, and potential problems can be discovered in time. In addition, the performance indicators can be intuitively displayed through the above-mentioned visual monitoring results, which is convenient for accurately evaluating the correlation between the test results and system performance.

[0094] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] Corresponding to the test case execution method described in the above embodiment, Figure 2 A schematic diagram of the structure of a test case execution device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0096] Reference Figure 2 The device may be a test case execution device 21, and the test case execution device 21 may include a test case acquisition module 211, a test case execution module 212, a test case adjustment module 213 and a performance optimization model construction module 214.

[0097] Reference Figure 2 , the test case execution device 21 comprises:

[0098] The test case acquisition module 211 is used to acquire test cases for the system to be tested; the test cases are set according to the test plan of the system to be tested, and the test plan is used to indicate the test requirements of the system to be tested;

[0099] The test case execution module 212 is used to execute the test case according to the pre-built controller, wherein the pre-built controller at least includes a parallel controller;

[0100] The test case adjustment module 213 is used to adjust the resource usage of the test case according to the execution result of the test case when the test case is successfully executed, and execute the test case with the adjusted resource usage to obtain a new execution result;

[0101] The performance optimization model building module 214 is used to build a performance optimization model of the system to be tested according to the new execution result, wherein the performance optimization model is used to indicate the optimization direction of the system to be tested.

[0102] In another optional embodiment of the present application, the test case execution device 21 further includes a controller construction module, and the controller construction module is used to execute the test case according to the pre-constructed controller before further including:

[0103] According to the above test plan, a corresponding thread group is set for the above system to be tested;

[0104] A specific controller and the parallel controller are set under the thread group; the specific controller is a controller set according to the test plan.

[0105] After the controller is constructed, correspondingly, when executing the test case according to the pre-constructed controller, the test case execution module 212 includes:

[0106] The above test cases are executed according to the above specific controller and the above parallel controller.

[0107] In some embodiments, the resource usage includes available resources and / or concurrency, wherein the available resources refer to the system available resources allocated to the test case, which may include CPU resources, storage resources, network resources, etc., and the concurrency refers to the number of test cases that can be processed simultaneously by the system at a certain moment. In order to further improve the accuracy of resource usage adjustment, the test case adjustment module 213, when adjusting the resource usage of the test case according to the execution result of the test case, includes:

[0108] Determine performance indicators based on the execution results of the above test cases, and perform test analysis on the above performance indicators;

[0109] Based on the analysis results of the above performance indicators, the available resource amount of the above test case is re-determined, and / or the concurrency of the above system to be tested is re-determined to obtain new available resource amount and / or new concurrency.

[0110] In another optional embodiment of the present application, the test case execution device 21 further includes a recovery module, and the recovery module is used to, after the test case is executed according to the pre-built controller, include:

[0111] In the event that the test case fails to execute, a recovery strategy is determined according to the execution result of the test case, and the test case is re-executed using the determined recovery strategy.

[0112] In some embodiments, when the recovery module determines a recovery strategy according to the execution result of the test case and re-executes the test case using the determined recovery strategy, it includes:

[0113] Determine the recovery time and number of retries of the above test cases according to the execution results of the above test cases;

[0114] Re-execute the above test case based on the above recovery time and number of retries.

[0115] In some embodiments, when the performance optimization model building module 214 builds the performance optimization model of the system to be tested according to the new execution result, it includes:

[0116] Determine a performance parameter according to the new execution result, wherein the performance parameter includes at least one of the following: a parameter corresponding to the thread group, a parameter corresponding to the resource usage, and a parameter corresponding to the controller;

[0117] The above performance parameters are used to construct a performance optimization model of the above system to be tested.

[0118] In another optional embodiment of the present application, the test case execution device 21 further includes a monitoring module, and the monitoring module is used to, after determining the performance indicator according to the execution result of the test case, include:

[0119] The performance indicators are monitored using preset monitoring elements to obtain visual monitoring results corresponding to the performance indicators, wherein the monitoring elements correspond to the performance indicators one-to-one.

[0120] It should be noted that the information interaction, execution process, etc. between the devices / units are based on the same concept as the method embodiments of the present application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0121] Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: at least one processor 30 ( Figure 3 The at least one processor 30 includes a memory 31 and a computer program 32 stored in the memory 31 and executable on the at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the method embodiments are implemented.

[0122] The electronic device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 3 and does not constitute a limitation of the electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include an input sending device, a network access device, a bus, etc.

[0123] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0124] In some embodiments, the memory 31 may be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 31 may also be an external storage device of the electronic device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 3. Further, the memory 31 may also include both an internal storage unit of the electronic device 3 and an external storage device. The memory 31 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 31 may also be used to temporarily store data that has been sent or is to be sent.

[0125] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the functional units and modules is used as an example. In practical applications, the function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0126] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the method embodiments when executing the computer program.

[0127] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the various method embodiments can be implemented.

[0128] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the various method embodiments when executing the computer program product.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0130] In the embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A test case execution method, characterized in that: include: Get the test cases for the system to be tested; The test case is set according to a test plan of the system to be tested, wherein the test plan is used to indicate the test requirements of the system to be tested; Executing the test case according to a pre-built controller, wherein the pre-built controller includes at least a parallel controller; If the test case is successfully executed, adjusting the resource usage of the test case according to the execution result of the test case, executing the test case after adjusting the resource usage, and obtaining a new execution result; A performance optimization model of the system to be tested is constructed according to the new execution result, wherein the performance optimization model is used to indicate an optimization direction of the system to be tested.

2. The test case execution method according to claim 1, characterized in that: Before executing the test case according to the pre-built controller, it also includes: Setting a corresponding thread group for the system to be tested according to the test plan; Setting a specific controller and the parallel controller under the thread group; the specific controller is a controller set according to the test plan; Executing the test case according to the pre-built controller includes: The test case is executed according to the specific controller and the parallel controller.

3. The test case execution method according to claim 2, characterized in that: The resource usage includes available resource usage and / or concurrent usage, and adjusting the resource usage of the test case according to the execution result of the test case includes: Determine performance indicators according to the execution results of the test cases, and perform test analysis on the performance indicators; The available resource amount of the test case is re-determined according to the analysis result of the performance indicator, and / or the concurrency of the system to be tested is re-determined to obtain a new available resource amount and / or a new concurrency.

4. The test case execution method according to any one of claims 1 to 3, characterized in that: After executing the test case according to the pre-built controller, the method further includes: In the event that the test case fails to execute, a recovery strategy is determined according to the execution result of the test case, and the test case is re-executed using the determined recovery strategy.

5. The test case execution method according to claim 4, characterized in that: Determining a recovery strategy according to the execution result of the test case, and re-executing the test case using the determined recovery strategy, comprises: Determine the recovery time and number of retries of the test case according to the execution result of the test case; Re-execute the test case according to the recovery time and the number of retries.

6. The test case execution method according to claim 2, characterized in that: The step of constructing the performance optimization model of the system to be tested according to the new execution result includes: Determine a performance parameter according to the new execution result, wherein the performance parameter includes at least one of the following: a parameter corresponding to the thread group, a parameter corresponding to the resource usage, and a parameter corresponding to the controller; A performance optimization model of the system to be tested is constructed using the performance parameters.

7. The test case execution method according to any one of claims 1 to 3, characterized in that: After determining the performance indicator according to the execution result of the test case, the method further includes: The performance indicator is monitored using a preset monitoring element to obtain a visual monitoring result corresponding to the performance indicator, wherein the monitoring element corresponds to the performance indicator one-to-one.

8. A test case execution device, characterized in that: include: A test case acquisition module is used to acquire test cases for the system to be tested; The test case is set according to a test plan of the system to be tested, wherein the test plan is used to indicate the test requirements of the system to be tested; A test case execution module, configured to execute the test case according to a pre-built controller, wherein the pre-built controller includes at least a parallel controller; A test case adjustment module, used to adjust the resource usage of the test case according to the execution result of the test case when the test case is successfully executed, and execute the test case after the resource usage is adjusted to obtain a new execution result; A performance optimization model building module is used to build a performance optimization model of the system to be tested according to the new execution result, wherein the performance optimization model is used to indicate the optimization direction of the system to be tested.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images