Method and device for determining target test allocation scheme, electronic equipment and product

Abstract

The invention is suitable for the technical field of testing, and particularly relates to a method and device for determining a target test allocation scheme, electronic equipment and a product. The invention provides a method for determining a target test allocation scheme, and the method comprises the steps: determining a first test set cost of a test set according to a first test allocation scheme, and determining a target test allocation scheme according to the first test set cost and a second test set cost, the second test set cost is the cost obtained when the test set is tested based on a second test allocation scheme, and according to the target test allocation scheme, determining an operation environment corresponding to each test case in the plurality of test cases. According to the method and the device, cost and income quantification is carried out on different conditions that the test case runs on the entity rack and the virtual rack, and a balance point of using the entity rack and the virtual rack to carry out hybrid scheduling can be dynamically found, so that the overall cost is lowest, and the overall income is maximized.

Description

Technical Field

[0001] The present application belongs to the field of testing technology, and in particular, relates to a method, device, electronic equipment and product for determining a target test allocation scheme. Background Art

[0002] In the process of automobile development and testing, a hybrid deployment of virtual and physical test benches is generally used. However, virtual and physical test benches are like the two ends of a scale, and a balance needs to be found. Using a physical test bench can test real scenarios, but the cost is too high and the physical environment configuration is complex. Using a virtual test bench is very low in cost and simple to configure, but it is not as realistic as a physical test bench and the test effect is slightly inferior. When a hybrid deployment of virtual and physical test benches is used, how to find the balance between virtual and physical test benches is a technical difficulty in the industry. Summary of the invention

[0003] The embodiments of the present application provide a method, device, electronic device and product for determining a target test allocation scheme, which can solve the technical problem of how to find the balance point between a virtual test bench and a physical test bench.

[0004] In a first aspect, an embodiment of the present application provides a method for determining a target test allocation scheme, comprising:

[0005] Determine the cost of a first test set of a test set according to a first test allocation scheme, wherein the test set includes a plurality of test cases, the first test allocation scheme includes an operating environment corresponding to each of the plurality of test cases, the operating environment includes a virtual bench or a physical bench, and the cost of the first test set of the test set is determined based on the case cost of the plurality of test cases;

[0006] Determining a target test allocation scheme according to the first test set cost and the second test set cost, wherein the second test set cost is a cost obtained when the test set is tested based on the second test allocation scheme;

[0007] According to the target test allocation scheme, a running environment corresponding to each of the plurality of test cases is determined.

[0008] In a possible implementation of the first aspect, the use case cost of the test case is determined based on first information corresponding to the test case, and the first information includes one or more of the following: continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems discovered.

[0009] In a possible implementation manner of the first aspect, determining a target test allocation scheme according to the first test set cost and the second test set cost includes:

[0010] The first test set cost is less than the second test set cost, and the difference is less than a cost difference threshold, and the target test allocation scheme is determined to be the first test allocation scheme.

[0011] In a possible implementation manner of the first aspect, the method further includes:

[0012] The first test set cost is less than the second test set cost, the difference is greater than the cost difference threshold, and the number of tests is greater than the number threshold, and the first test allocation scheme is determined to be the target test allocation scheme.

[0013] In a possible implementation manner of the first aspect, the method further includes:

[0014] If the cost of the first test set is less than the cost of the second test set, and the difference is greater than the cost difference threshold, and the number of tests is less than the number threshold, the first test allocation plan is adjusted, and the step of determining the cost of the first test set of the test set according to the first test allocation plan is returned to until the target test allocation plan is determined.

[0015] In a possible implementation manner of the first aspect, adjusting the test allocation scheme includes:

[0016] Adjust the running environment of the test case with the largest number of consecutive successes in the physical bench successful test set to the virtual bench;

[0017] Adjust the running environment of the test case with the largest number of errors in the virtual bench error test set to the physical bench.

[0018] In a possible implementation manner of the first aspect, the method further includes:

[0019] Performing tests according to the operating environment of multiple test cases in the test set included in the target test allocation scheme to obtain test results, wherein the test results include the continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems found for the test cases in the current test;

[0020] According to the test result, the continuous use time of the test bench, the hardware cost, the number of manual interventions, and the number of problems found, which are used to determine the use case cost of the test case, are updated.

[0021] In a second aspect, an embodiment of the present application provides a device for determining a target test allocation scheme, including:

[0022] A first determination module is configured to determine a first test set cost of a test set according to a first test allocation scheme, wherein the test set includes a plurality of test cases, the first test allocation scheme includes an operating environment corresponding to each of the plurality of test cases, the operating environment includes a virtual bench or a physical bench, and the first test set cost of the test set is determined based on the case cost of the plurality of test cases;

[0023] A second determination module is used to determine a target test allocation scheme according to the first test set cost and the second test set cost, wherein the second test set cost is a cost obtained when the test set is tested based on the second test allocation scheme;

[0024] The third determination module is used to determine the operating environment corresponding to each of the multiple test cases according to the target test allocation plan.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method for determining a target test allocation scheme according to the first aspect above.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for determining a target test allocation scheme as described in the first aspect above is implemented.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed, enables the method for determining a target test allocation scheme as described in the first aspect to be executed.

[0028] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0029] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0030] The present application determines the cost of the first test set of the test set according to the first test set cost and the second test set cost, determines the target test allocation scheme, the second test set cost is the cost obtained when the test set is tested based on the second test allocation scheme, and determines the operating environment corresponding to each of the multiple test cases according to the target test allocation scheme. Wherein, the test set includes multiple test cases, the first test allocation scheme includes the operating environment corresponding to each of the multiple test cases, the operating environment includes a virtual bench or a physical bench, and the cost of the first test set of the test set is determined based on the test case cost of multiple test cases. The present application adjusts the test allocation scheme and determines the target test allocation scheme based on the test set cost and the number of tests, quantifies the cost and benefit of different situations of test cases running on physical benches and virtual benches, and can dynamically find the balance point for mixed scheduling using physical benches and virtual benches to minimize the overall cost and maximize the overall benefit. The target test allocation scheme is determined based on the test set cost and the number of tests, and the cost-benefit is used as the final anchor point to achieve the optimal hybrid scheduling strategy of physical and virtual test benches, so that the virtual and physical test benches are in a long-term balance and game state. By adjusting the test allocation scheme based on the heuristic algorithm, the cost-optimal solution can be found through continuous attempts. The independent variable used to determine the use case cost of the test case is updated based on the test results, and dynamic self-learning adjustment is achieved, so that the use of physical and virtual test benches is always in a dynamic balance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 It is a flowchart of a method for determining a target test allocation scheme provided by an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of a test allocation solution provided in an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of adjusting a test allocation scheme provided in an embodiment of the present application;

[0035] Figure 4 is a schematic diagram of a test process provided by an embodiment of the present application;

[0036] Figure 5 This is a flow chart of a method for determining a target test allocation scheme provided by an application embodiment of the present application;

[0037] Figure 6 It is a schematic diagram of a device for determining a target test allocation scheme provided by an embodiment of the present application;

[0038] Figure 7 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0039] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0040] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0041] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0042] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0043] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0044] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0045] The method for determining the target test allocation scheme provided in the embodiment of the present application can be applied to electronic devices such as cloud servers, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0046] For example, the electronic device may be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set top box (STB), a customer premises equipment (CPE) and / or other devices for communicating on a wireless system and a next generation communication system, for example, a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network, etc.

[0047] In the prior art, the following two testing schemes are usually used in the process of automobile development and testing.

[0048] The first is to use a virtual bench for testing by default. However, this testing solution often ignores the disadvantages of using a virtual bench and ignores or misses the problems encountered by the target test software. In addition, there is no cost quantification method for this scheduling.

[0049] The second method is to perform static division according to specific test sets, dividing which test cases can be run on virtual benches and which test cases can be run on physical benches (also called physical benches). However, this testing solution is not flexible enough and will eventually cause a large number of test cases to run on physical benches, resulting in high costs.

[0050] The physical test bench is characterized by high coverage and the ability to run all test cases. However, it is complex to maintain, prone to errors, unstable, and costly. The virtual test bench is characterized by the ability to run 90% of test cases, stable operation, low error rate, and low cost. During testing, test parameters, the program under test, and the operating environment (including the virtual test bench or the physical test bench) are usually used as inputs for testing of test cases. For newly generated test cases, they must usually be successfully verified on the physical test bench before entering the virtual test bench. Each test case has the characteristic of idempotence, that is, no matter when it is run or how many times it is run, as long as the input remains unchanged, the results will be the same.

[0051] When virtual rigs and physical rigs are deployed and used in a mixed manner, how to find the balance between the virtual rigs and the physical rigs is a technical difficulty in the industry.

[0052] Figure 1 It is a flowchart of a method for determining a target test allocation scheme provided in an embodiment of the present application.

[0053] S11, determining a first test set cost of a test set according to a first test allocation scheme.

[0054] The test set includes multiple test cases, the first test allocation scheme includes an operating environment corresponding to each of the multiple test cases, the operating environment includes a virtual test bench or a physical test bench, and the cost of the first test set of the test set is determined based on the case cost of the multiple test cases.

[0055] A test allocation scheme refers to an allocation scheme for the operating environment assigned to each of the multiple test cases in a test set. The test allocation scheme includes the operating environment corresponding to each of the multiple test cases, and the operating environment includes a virtual bench or a physical bench. Different test allocation schemes (for example, different operating environments for a certain test case, and different orders in which multiple test cases are tested in sequence) will result in different test set costs. Therefore, the first test set cost of the test set can be determined based on the current first test allocation scheme.

[0056] For example, the test set cost of the test set can be determined based on the use case costs of multiple test cases in the test set. For example, the sum of the use case costs of multiple test cases can be used as the test set cost of the test set. The test set cost of the test set can be calculated based on the following formula based on the use case costs of multiple test cases:

[0057] cost(test set) = sum{cost(test case)}.

[0058] Among them, cost(test set) represents the test set cost of the test set, cost ( Test Cases ) Represents the use case cost of a test case, and sum{cost(test case)} represents the sum of the use case costs of multiple test cases.

[0059] In one embodiment, the use case cost of the test case is determined based on first information corresponding to the test case, and the first information includes one or more of the following: continuous use time of the test stand, hardware cost, number of manual interventions, and number of problems found.

[0060] Here, the use case cost of the test case can be determined based on the first information corresponding to the test case. The first information includes one or more of the following: continuous use time of the bench, hardware cost, number of manual interventions, and number of problems found. Those skilled in the art should understand that the use case cost of the test case can be determined based on any one, any two, any three, or all four of the above. In this application, the use case cost of the test case is often determined based on the four first information items of continuous use time of the bench, hardware cost, number of manual interventions, and number of problems found. Other ways of determining the use case cost of the test case also fall within the scope of protection of this application. The use case cost of the test case is determined based on the four items of continuous use time of the bench, hardware cost, number of manual interventions, and number of problems found, and can be expressed based on the following formula.

[0061] The hardware cost includes, for example, the platform cost, CPU cost, storage cost, electricity cost, etc.

[0062]

[0063] Among them, g0 represents the calculation function of the continuous use time of the test bench (proportional to the final test set cost P), g1 represents the calculation function of the hardware cost (proportional to the final test set cost P), g2 represents the calculation function of the number of manual interventions (proportional to the final test set cost P), g3 represents the calculation function of the number of problems found (the more problems found, the better, which is understood as benefit, and is inversely proportional to the final test set cost P), f{} represents the calculation function of the use case cost of the test case, and cost(test case) represents the use case cost of the test case. Among them, the calculation function can also be called a decision function. g0, g1, and g2 are monotonically increasing functions (indicating that the higher the continuous use time of the test bench, the higher the hardware cost, and the number of manual interventions, the higher the use case cost of the test case), and g3 is a monotonically decreasing function (indicating that the higher the number of problems found, the higher the benefit, and the lower the use case cost of the corresponding test case, that is, an increase in benefit means a decrease in use case cost).

[0064] For the calculation function g0 of the continuous use time of the test bench, taking the virtual test bench as an example, the longer the continuous use time of the virtual test bench, the greater the potential risk of the test case, and it is possible to deviate from the real environment. The longer the continuous use time, the greater the cost of using the virtual test bench, and the lower the cost of using the physical test bench.

[0065] g2 (number of manual interventions) = g21 (number of hardware exceptions)

[0066] +g22 (number of virtual environment exceptions)

[0067] g21 (number of hardware exceptions) represents the calculation function of the number of hardware exceptions (proportional to the final test set cost P), and g22 (number of virtual environment exceptions) represents the calculation function of the number of virtual environment exceptions (proportional to the final test set cost P).

[0068] In one embodiment, the use case cost of the test case is determined based on the following formula according to the continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems found.

[0069]

[0070] Among them, g0 represents the calculation function of the continuous use time of the test bench (proportional to the final test set cost P), g1 represents the calculation function of the hardware cost (proportional to the final test set cost P), g2 represents the calculation function of the number of manual interventions (proportional to the final test set cost P), g3 represents the calculation function of the number of problems found (the more problems found, the better, which is understood as benefit, and is inversely proportional to the final test set cost P), f{} represents the calculation function of the use case cost of the test case, and cost(test case) represents the use case cost of the test case. x0 represents the continuous use time of the test bench, x1 represents the hardware cost (running time, which can be expressed as run_cost), x21 represents the number of hardware anomalies, x22 represents the number of virtual environment anomalies, the sum of x21 and x22 is the number of manual interventions, x3 represents the number of problems found, kk and k0 represent the parameters of the continuous use time of the test bench, k1 represents the parameter of the hardware cost (the cost of the running unit, which can be expressed as money), k21 represents the parameter of the number of hardware anomalies, k22 represents the parameter of the number of virtual environment anomalies, k2 is the parameter of the number of manual interventions, k3 represents the parameter of the number of problems found, f{} represents the calculation function of the use case cost of the test case, and cost(test case) represents the use case cost of the test case.

[0071] Among them, the continuous use time of the test bench x0, the hardware cost x1, the number of hardware anomalies x21, the number of virtual environment anomalies x22, the sum of the number of manual interventions x21 and x22, and the number of problems found x3 are the core variables of the calculation function (also called core elements or independent variables). The core variables will change in real time with time or other actual conditions.

[0072] The parameters kk and k0 of the continuous use time of the test bench, the parameter k1 of the hardware cost, the parameter k21 of the number of hardware anomalies, the parameter k22 of the number of virtual environment anomalies, the parameter k2 of the number of manual interventions, and the parameter k3 of the number of problems found are the parameters of the calculation function. The core variables are used as independent variables to calculate the parameters of the calculation function. For cases with a small amount of data, the parameter values ​​of the calculation function can be obtained by solving equations. For cases with a large number of data, the parameter values ​​of the calculation function can be fitted through training. The closer the selected calculation function (decision function) is to the actual situation, the closer it will be to the optimal solution.

[0073] Figure 2 It is a schematic diagram of a test allocation scheme provided in an embodiment of the present application.

[0074] like Figure 2As shown, multiple test cases in the test set are allocated to the test set of the physical bench and the test set of the virtual bench. The test set of the physical bench includes an error test set and a success test set, wherein the error test set is arranged from large to small according to the number of errors, and the success test set is arranged from small to large according to the number of consecutive successes. The test set of the virtual bench includes an error test set and a success test set, wherein the error test set is arranged from large to small according to the number of errors, and the success test set is arranged from small to large according to the number of consecutive successes.

[0075] According to the above test allocation scheme, for a physical test bench, the case cost of the test case with the most errors in the error test set can be determined first, and then the case cost of the test cases with relatively smaller errors in the error test set can be calculated successively according to the number of errors from large to small. After the case cost of the test cases in the error test set is calculated, the case cost of the test case with the smallest number of consecutive successes in the successful test set is calculated, and then the case cost of the test cases with relatively larger number of consecutive successes in the successful test set is calculated successively according to the number of consecutive successes from small to large. For a virtual test bench, the case cost of the test case with the most errors in the error test set can be determined first, and then the case cost of the test cases with relatively smaller errors in the error test set can be calculated successively according to the number of errors from large to small. After the case cost of the test cases in the error test set is calculated, the case cost of the test case with the smallest number of consecutive successes in the successful test set is calculated, and then the case cost of the test cases with relatively larger number of consecutive successes in the successful test set is calculated successively according to the number of consecutive successes from small to large.

[0076] According to the first test allocation scheme, the case costs of the multiple test cases in the test set are determined, and then the first test set cost of the test set can be determined based on the case costs of the multiple test cases.

[0077] S12: determining a target test allocation scheme according to the first test set cost and the second test set cost, wherein the second test set cost is a cost obtained when the test set is tested based on the second test allocation scheme.

[0078] It is understandable that, before step S11, the method provided in the embodiment of the present application may further include: testing the test set according to at least one different test allocation scheme to obtain at least one test set cost. Wherein, any test set cost corresponds to a test allocation scheme. Wherein, at least one test allocation scheme includes a second test allocation scheme.

[0079] Here, the first test set cost and the second test set cost may be comprehensively considered to determine whether the target test allocation solution has been found.

[0080] As an example, the second test set cost may be the test set cost calculated last time, that is, the test cost of the test set determined using the second test allocation scheme before the first test set cost. Alternatively, the second test set cost may also be the lowest test set cost among multiple test set costs calculated before the current calculation of the first test set cost.

[0081] S13: Determine, according to the target test allocation scheme, a running environment corresponding to each of the plurality of test cases.

[0082] Here, when the target test allocation scheme has been found, the operating environment corresponding to each of the multiple test cases can be determined according to the target test allocation scheme.

[0083] In one embodiment, determining a target test allocation scheme according to the first test set cost and the second test set cost includes:

[0084] The first test set cost is less than the second test set cost, and the difference is less than a cost difference threshold, and the target test allocation scheme is determined to be the first test allocation scheme.

[0085] As an example, the difference between the first test set cost and the second test set cost may be calculated based on the following formula.

[0086] diff = cost (this test set) - cost (previous test set)

[0087] Among them, cost(this test set) represents the test set cost calculated this time (that is, the first test set cost), cost(previous test set) represents the test set cost calculated previously (that is, the second test set cost), and diff represents the difference between the first test set cost and the second test set cost. The cost difference threshold can be set based on experience.

[0088] If the cost of the first test set is less than the cost of the second test set, and the difference is less than the cost difference threshold, it means that the change in the cost of the first test set of the test set when calculated using the first test allocation plan is small enough, so the first test allocation plan can be determined as the target test allocation plan.

[0089] For example, as an example, when the difference between the first test set cost and the second test set cost is less than or equal to the cost difference threshold, the test allocation scheme corresponding to the lowest test cost between the first test set cost and the second test set cost can be determined as the target test allocation scheme. For example, if the first test set cost is lower than the second test set cost, and the difference between the first test set cost and the second test set cost is less than or equal to the cost difference threshold, the first test set cost is determined to be the target test allocation scheme.

[0090] In another possible embodiment of the present application, if the first test set cost is greater than the second test set cost, it can be determined that the target test allocation scheme has been found, and the test allocation scheme corresponding to the second test set cost is used as the target test allocation scheme.

[0091] In one embodiment, the method for determining a target test allocation scheme further includes:

[0092] The first test set cost is less than the second test set cost, the difference is greater than the cost difference threshold, and the number of tests is greater than the number threshold, and the first test allocation scheme is determined to be the target test allocation scheme.

[0093] Here, the number of tests can be continuously counted. When the cost of the first test set (that is, the test set cost calculated this time) is less than the cost of the second test set (the test set cost calculated previously), and the difference is greater than the cost difference threshold, it means that there is still room for operation to reduce the test set cost. Therefore, it can be further determined whether the number of tests is greater than the number threshold. If the number of tests is greater than the number threshold, it is determined that the target test allocation scheme has been found, and the first test allocation scheme during this calculation is determined as the target test allocation scheme. The number threshold can be set based on experience.

[0094] In one embodiment, the method for determining a target test allocation scheme further includes:

[0095] If the cost of the first test set is less than the cost of the second test set, and the difference is greater than the cost difference threshold, and the number of tests is less than the number threshold, the first test allocation plan is adjusted, and the step of determining the cost of the first test set of the test set according to the first test allocation plan is returned to until the target test allocation plan is determined.

[0096] Here, when the cost of the first test set (that is, the cost of the test set calculated this time) is less than the cost of the second test set (the cost of the test set calculated previously), and the difference is greater than the cost difference threshold, it means that there is still room for operation to reduce the cost of the test set, so it can be further determined whether the number of tests is greater than the number threshold. If the number of tests is less than the number threshold, it means that there is still room for operation to reduce the cost of the test set, and the operating environment of the test case in the test allocation scheme can be adjusted, for example, adjusting the test case running on the virtual bench to running on the physical bench, and adjusting the test case running on the physical bench to running on the virtual bench. After adjusting the test allocation scheme, return to S11, and determine the cost of the first test set of the test set according to the first test allocation scheme (adjusted test allocation scheme). S12, according to the first test set cost and the second test set cost (here, the second test set cost can be the original first test set cost, that is, the second test set cost is the lowest value of the previous multiple test set costs), determine the target test allocation scheme until the target test allocation scheme is found, and according to the target test allocation scheme, determine the operating environment corresponding to each of the multiple test cases in the test set.

[0097] In one embodiment, adjusting the test allocation scheme includes:

[0098] Adjust the running environment of the test case with the largest number of consecutive successes in the physical bench successful test set to the virtual bench;

[0099] Adjust the running environment of the test case with the largest number of errors in the virtual bench error test set to the physical bench.

[0100] Figure 3 It is a schematic diagram of adjusting the test allocation scheme provided in an embodiment of the present application.

[0101] like Figure 3 As shown, the operating environment of the test case with the largest number of consecutive successes in the physical bench successful test set can be adjusted to the virtual bench, for example, it can be placed in the first position in the virtual bench error test set. The operating environment of the test case with the largest number of errors in the virtual bench error test set can be adjusted to the physical bench, for example, it can be placed in the last position in the physical bench successful test set.

[0102] When the number of tests is less than the number threshold, each adjustment of the test allocation scheme can be performed in the above manner until there is no test case in a test set. A test set can also be called a test bucket.

[0103] In one embodiment, the method for determining a target test allocation scheme further includes:

[0104] Performing tests according to the operating environment of multiple test cases in the test set included in the target test allocation scheme to obtain test results, wherein the test results include the continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems found for the test cases in the current test;

[0105] According to the test result, the continuous use time of the test bench, the hardware cost, the number of manual interventions, and the number of problems found, which are used to determine the use case cost of the test case, are updated.

[0106] Here, if the target test allocation scheme has been found, the test can be performed according to the operating environment of multiple test cases in the test set included in the target test allocation scheme, that is, the test set can be run. The operating environment includes a virtual bench or a physical bench. The test results include information such as the continuous use time of the bench, hardware cost, number of manual interventions, number of problems found, etc. for each test case in the current test. The relevant information such as the continuous use time of the bench, hardware cost, number of manual interventions, number of problems found, etc. used to determine the use case cost of the test case can be updated based on the above information to recalculate the use case cost of the test case. For example, if the number of problems found for the first test case in the current test is 2, the total number of problems found used to determine the use case cost of the first test case is increased by 2 times.

[0107] Figure 4 It is a schematic diagram of a test process provided in an embodiment of the present application.

[0108] like Figure 4 As shown, a set of test cases constitutes a test set. The test set is allocated according to the target test allocation scheme determined by the scheduling strategy, and the running environment of each test case in the multiple test cases of the test set is allocated. Several test cases are tested in the physical bench, and several other test cases are tested in the virtual bench. For the test cases that run successfully in the physical bench, you can enter the virtual bench for testing. For the test cases that fail to run successfully in the physical bench, through manual troubleshooting, if it is a problem of hardware environment abnormality, rerun it, if it is a software abnormality, it is recorded as a problem found. For the test cases that run successfully in the virtual bench, if the time period has not been exceeded, you can continue to test in the virtual bench; if the time period has exceeded, you can enter the physical bench for testing. For the test cases that fail to run successfully in the virtual bench, through manual troubleshooting, if it is a virtual environment abnormality, rerun it and determine whether the time period has exceeded. If the time period has not exceeded, you can continue to test in the virtual bench; if the time period has exceeded, you can enter the physical bench for testing. For the test cases that fail to run successfully in the virtual bench, through manual troubleshooting, if it is a problem of software abnormality, it is recorded as a problem found.

[0109] Here, after the above test process is completed, the test results will be obtained. The test results include information such as the continuous use time of the test bench, hardware cost, number of manual interventions, number of problems found, etc. for each test case in the current test. The continuous use time of the test bench, hardware cost, number of manual interventions, number of problems found, etc. for determining the use case cost of the test case can be updated based on the above information to recalculate the use case cost of the test case. For example, if the number of problems found for the first test case in the current test is 1, the total number of problems found used to determine the use case cost of the first test case increases by 1.

[0110] The present application determines the first test set cost of the test set according to the first test set cost and the second test set cost, determines the target test allocation scheme, and determines the operating environment corresponding to each of the multiple test cases according to the target test allocation scheme. Wherein, the test set includes multiple test cases, the test allocation scheme includes the operating environment corresponding to each of the multiple test cases, the operating environment includes a virtual bench or a physical bench, the first test set cost of the test set is determined based on the case cost of multiple test cases, and the second test set cost is the cost obtained when the test set is tested based on the second test allocation scheme. The present application adjusts the test allocation scheme and determines the target test allocation scheme based on the test set cost and the number of tests, and quantifies the costs and benefits of different situations of test cases running on physical benches and virtual benches, and can dynamically find the balance point for mixed scheduling using physical benches and virtual benches to minimize the overall cost and maximize the overall benefit. Based on the test set cost and the number of tests, it is determined whether the target test allocation scheme has been found, and the cost-benefit is used as the final anchor point, realizing the optimal hybrid scheduling strategy of physical test benches and virtual test benches, so that the virtual test benches and physical test benches are in a long-term balance and game state. By adjusting the test allocation scheme based on the heuristic algorithm, the cost-optimal solution can be found through continuous attempts. Based on the test results, the independent variables used to determine the use case cost of the test case are updated, realizing dynamic self-learning adjustment, so that the use of physical test benches and virtual test benches is always in a dynamic balance.

[0111] The method for determining the target test allocation scheme proposed in this application can be applied to regression testing scenarios. The triggering time of regression testing is: every time the program is modified (such as development code submission, program upgrade, recompilation), or the system integration changes, regression testing will be triggered. The purpose of regression testing is to ensure that each iteration can ensure the stable operation of existing functions. In addition, regression testing requires re-verification: that is, every once in a while, it is necessary to return to the physical test bench to implement operation and verification.

[0112] Figure 5 It is a flowchart of a method for determining a target test allocation scheme provided by an application embodiment of the present application.

[0113] like Figure 5 As shown, after starting, for multiple test cases in the test set, according to the current first test allocation scheme (that is, a combination of the test set), the use case cost of each test case is calculated to determine the first test set cost of the test set. The test allocation scheme includes a running environment corresponding to each of the multiple test cases, and the running environment includes a virtual bench or a physical bench. The first test set cost of the test set is determined based on the use case costs of the multiple test cases.

[0114] According to the test set cost calculated this time (ie, the first test set cost) and the test set cost calculated previously (ie, the second test set cost), it is determined whether the target test allocation scheme has been found.

[0115] If the test set cost calculated this time is less than the test set cost calculated previously, and the difference is less than the cost difference threshold, it is determined that the target test allocation scheme has been found, and the target test allocation scheme is determined to be the first test allocation scheme.

[0116] If the test set cost calculated this time is less than the test set cost calculated previously, and the difference is greater than the cost difference threshold, it is determined whether the number of tests is greater than the number threshold.

[0117] If the number of tests is greater than the number threshold, it is determined that the target test allocation scheme has been found. The first test allocation scheme is determined to be the target test allocation scheme.

[0118] If the number of tests is less than the number threshold, the test allocation plan is adjusted, and the process returns to the step of determining the first test set cost of the test set according to the current first test allocation plan until the target test allocation plan is found.

[0119] If the target test allocation scheme has been found, the operating environment corresponding to each of the multiple test cases is determined according to the target test allocation scheme. That is, the test set is split according to the target test allocation scheme, and the multiple test cases in the test set are split into a test set of a physical bench and a test set of a virtual bench.

[0120] The run is finished.

[0121] Figure 6 It is a schematic diagram of a device for determining a target test allocation scheme provided in an embodiment of the present application.

[0122] like Figure 6 As shown, the device 6 for determining the target test allocation scheme includes:

[0123] A first determination module 61 is used to determine a first test set cost of a test set according to a first test allocation scheme, wherein the test set includes a plurality of test cases, the first test allocation scheme includes an operating environment corresponding to each of the plurality of test cases, the operating environment includes a virtual bench or a physical bench, and the first test set cost of the test set is determined based on the case cost of the plurality of test cases;

[0124] A second determination module 62, configured to determine a target test allocation scheme according to the first test set cost and the second test set cost, wherein the second test set cost is a cost obtained when the test set is tested based on the second test allocation scheme;

[0125] The third determination module 63 is used to determine the operating environment corresponding to each of the multiple test cases according to the target test allocation scheme.

[0126] Another embodiment of the present invention discloses a device 6 for determining a target test allocation scheme. Figure 6 On the basis of the corresponding embodiment, the use case cost of the test case is determined based on the first information, and the first information includes one or more of the following: continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems found.

[0127] Another embodiment of the present invention discloses a device 6 for determining a target test allocation scheme. Figure 6 Based on the corresponding embodiment, the second determining module 62 is used for:

[0128] The first test set cost is less than the second test set cost, and the difference is less than a cost difference threshold, and the target test allocation scheme is determined to be the first test allocation scheme.

[0129] Another embodiment of the present invention discloses a device 6 for determining a target test allocation scheme. Figure 6 Based on the corresponding embodiment, the second determining module 62 is further used for:

[0130] The first test set cost is less than the second test set cost, the difference is greater than the cost difference threshold, and the number of tests is greater than the number threshold, and the first test allocation scheme is determined to be the target test allocation scheme.

[0131] Another embodiment of the present invention discloses a device 6 for determining a target test allocation scheme. Figure 6 Based on the corresponding embodiment, the second determining module 62 is further used for:

[0132] If the cost of the first test set is less than the cost of the second test set, and the difference is greater than the cost difference threshold, and the number of tests is less than the number threshold, the first test allocation plan is adjusted, and the step of determining the cost of the first test set of the test set according to the first test allocation plan is returned to until the target test allocation plan is determined.

[0133] Another embodiment of the present invention discloses a device 6 for determining a target test allocation scheme. Figure 6 Based on the corresponding embodiment, the second determining module 62 is further used for:

[0134] Adjust the running environment of the test case with the largest number of consecutive successes in the physical bench successful test set to the virtual bench;

[0135] Adjust the running environment of the test case with the largest number of errors in the virtual bench error test set to the physical bench.

[0136] Another embodiment of the present invention discloses a device 6 for determining a target test allocation scheme. Figure 6 On the basis of the corresponding embodiment, the device 6 for determining the target test allocation scheme further includes:

[0137] A first acquisition module is used to perform tests according to the operating environment of multiple test cases in the test set included in the target test allocation scheme to obtain test results, wherein the test results include the continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems found in the test case in the current test;

[0138] The first updating module is used to update the continuous use time of the test bench, the hardware cost, the number of manual interventions, and the number of problems found, which are used to determine the use case cost of the test case, according to the test result.

[0139] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0140] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0141] The present application also provides an electronic device, such as Figure 7 As shown, the electronic device 7 includes: at least one processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, and when the processor 70 executes the computer program 72, the steps in any of the above-mentioned method embodiments are implemented.

[0142] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0143] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0145] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0146] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for determining a target test allocation scheme, characterized in that: include: Determine the cost of a first test set of a test set according to a first test allocation scheme, wherein the test set includes a plurality of test cases, the first test allocation scheme includes an operating environment corresponding to each of the plurality of test cases, the operating environment includes a virtual bench or a physical bench, and the cost of the first test set of the test set is determined based on the case cost of the plurality of test cases; Determining a target test allocation scheme according to the first test set cost and the second test set cost, wherein the second test set cost is a cost obtained when the test set is tested based on the second test allocation scheme; According to the target test allocation scheme, a running environment corresponding to each of the plurality of test cases is determined.

2. The method for determining a target test allocation scheme according to claim 1, characterized in that: The use case cost of the test case is determined based on first information corresponding to the test case, where the first information includes one or more of the following: continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems discovered.

3. The method for determining a target test allocation scheme according to claim 1, characterized in that: The determining of a target test allocation scheme according to the first test set cost and the second test set cost includes: The first test set cost is less than the second test set cost, and the difference is less than a cost difference threshold, and the target test allocation scheme is determined to be the first test allocation scheme.

4. The method for determining a target test allocation scheme according to claim 3, characterized in that: The method further comprises: The first test set cost is less than the second test set cost, the difference is greater than the cost difference threshold, and the number of tests is greater than the number threshold, and the first test allocation scheme is determined to be the target test allocation scheme.

5. The method for determining a target test allocation scheme according to claim 3, characterized in that: The method further comprises: If the cost of the first test set is less than the cost of the second test set, and the difference is greater than the cost difference threshold, and the number of tests is less than the number threshold, the first test allocation plan is adjusted, and the step of determining the cost of the first test set of the test set according to the first test allocation plan is returned to until the target test allocation plan is determined.

6. The method for determining a target test allocation scheme according to claim 5, characterized in that: The adjusting the first test allocation scheme includes: Adjust the running environment of the test case with the largest number of consecutive successes in the physical bench successful test set to the virtual bench; Adjust the running environment of the test case with the largest number of errors in the virtual bench error test set to the physical bench.

7. The method for determining a target test allocation scheme according to any one of claims 2 to 6, characterized in that: The method further comprises: Performing tests according to the operating environment of multiple test cases in the test set included in the target test allocation scheme to obtain test results, wherein the test results include the continuous use time of the test bench, hardware cost, number of manual interventions, and number of problems found for the test cases in the current test; According to the test result, the continuous use time of the test bench, the hardware cost, the number of manual interventions, and the number of problems found, which are used to determine the use case cost of the test case, are updated.

8. A device for determining a target test allocation scheme, characterized in that: include: A first determination module is configured to determine a first test set cost of a test set according to a first test allocation scheme, wherein the test set includes a plurality of test cases, the first test allocation scheme includes an operating environment corresponding to each of the plurality of test cases, the operating environment includes a virtual bench or a physical bench, and the first test set cost of the test set is determined based on the case cost of the plurality of test cases; A second determination module is used to determine a target test allocation scheme according to the first test set cost and the second test set cost, wherein the second test set cost is a cost obtained when the test set is tested based on the second test allocation scheme; The third determination module is used to determine the operating environment corresponding to each of the multiple test cases according to the target test allocation plan.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method according to any one of claims 1 to 7 to be performed.

Images