Test method and device, equipment and storage medium

Abstract

The invention provides a test method and device, equipment and a storage medium, and relates to the technical field of computers, in particular to the technical fields of artificial intelligence, automatic driving, big data processing, test case priority ranking and the like. According to the specific implementation scheme, the method comprises the steps of obtaining historical test data, wherein the historical test data comprises at least one test index of each test case in a plurality of test cases of a historical test; based on the weight value corresponding to the at least one test index, determining an initial priority coefficient of each test case in a plurality of test cases of a historical test; based on the test mode and the initial priority coefficient of the current test, determining a plurality of test cases of the current test and an execution sequence of the plurality of test cases of the current test; and testing the to-be-tested object according to the plurality of test cases and the execution sequence of the test. The test efficiency and accuracy of the to-be-tested object can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, autonomous driving, big data processing, and test case prioritization. Background Technology

[0002] In recent years, as the scale and complexity of systems under test, such as autonomous driving systems, have continued to increase, the workload of testing has grown significantly. Executing a large number of test cases not only consumes considerable time and resources but also makes it difficult to ensure sufficient testing of critical functions. Therefore, how to accurately and efficiently prioritize test cases to quickly identify key cases has become crucial for improving testing efficiency and quality, and has gradually become an important problem that urgently needs to be solved in the current testing field. Summary of the Invention

[0003] This disclosure provides a test method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, a testing method is provided, comprising: Obtain historical test data, which includes at least one test metric for each test case in multiple test cases from the historical tests; Based on the weight value corresponding to at least one test metric, determine the initial priority coefficient of each test case in multiple test cases of historical tests; Based on the test mode and initial priority coefficient of this round of testing, determine the multiple test cases of this round of testing and the execution order of the multiple test cases of this round of testing; Based on the multiple test cases and execution order of this round of testing, the object to be tested is tested.

[0005] According to another aspect of this disclosure, a testing apparatus is provided, comprising: The data acquisition module is used to acquire historical test data, which includes at least one test metric for each test case in multiple test cases of historical tests. The priority determination module is used to determine the initial priority coefficient of each test case in a series of test cases in historical testing based on the weight value corresponding to at least one test metric. The execution order determination module is used to determine the execution order of multiple test cases in this round of testing based on the test mode and initial priority coefficient of this round of testing. The testing module is used to test the object to be tested based on multiple test cases and their execution order in this round of testing.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] This disclosure introduces a data-driven dynamic prioritization mechanism, which objectively calculates the initial priority coefficient of each test case based on the weight values ​​corresponding to the test metrics of multiple test cases in historical testing. This reduces the problem of bias caused by subjective sorting. Furthermore, it adjusts the test cases and their execution order for this round of testing in conjunction with the testing mode, improving the targeting of the testing work. Finally, based on the multiple test cases and their execution order in this round of testing, the object to be tested is tested, improving the efficiency and accuracy of testing.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure; Figure 2 This is a flowchart illustrating the implementation of a testing method according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of a test process for an object under test according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating the determination of a test time period according to an embodiment of the present disclosure; Figure 5 This is a functional relationship between the exploration rate and the number of test rounds according to an embodiment of the present disclosure. Figure 1 ; Figure 6 This is a functional relationship between the exploration rate and the number of test rounds according to an embodiment of the present disclosure. Figure 2; Figure 7 This is a schematic diagram of the structure of a test apparatus 700 according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram of the structure of a test apparatus 800 according to an embodiment of the present disclosure; Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] The term "and / or" in this disclosure indicates that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document means any combination of at least two of a plurality of options, such as including at least one of A, B, and C, which can mean including any one or more elements selected from the set of A, B, and C. The terms "first" and "second" in this document refer to and distinguish multiple similar technical terms, and do not imply a specific order or a limitation to only two. For example, "first feature" and "second feature" refer to two types / two features; the first feature can be one or more, and the second feature can also be one or more.

[0014] In recent years, as the scale and complexity of systems under test, such as autonomous driving systems, have continued to increase, the workload of testing has also grown significantly. Executing a large number of test cases not only consumes considerable time and resources but also makes it difficult to ensure sufficient testing of critical functions. Therefore, to improve the efficiency and quality of testing, it is necessary to prioritize the execution of test cases before commencing testing.

[0015] Traditional test case prioritization methods mainly include the following: first, prioritization based on historical defect rate; second, prioritization based on code coverage; and third, prioritization based on requirement priority. These methods are characterized by simple rules and ease of implementation. For example, test cases can be marked using tables or priority labels, and no complex calculations are required. Therefore, they are suitable for small-scale testing or projects in a stable operational phase.

[0016] In addition, there are several algorithm-based test case priority ranking methods. Greedy algorithm-based methods, which aim for local optima, have high computational efficiency, with a time complexity of O(n) (meaning that when the input data size (e.g., the number of test cases) is n, the execution time of the greedy algorithm increases linearly with n). They are suitable for quickly selecting key test cases, but their drawback is a lack of awareness of global risks. Genetic algorithm-based priority ranking methods, on the other hand, have strong global search capabilities and are suitable for complex multi-objective optimization scenarios. However, their computational cost is relatively high, depending on the population size and the number of iterations.

[0017] Traditional test case prioritization methods, such as those based on static metrics like historical defect rate and code coverage, have the following drawbacks: (1) These priority ranking methods are static and it is difficult to dynamically and flexibly adjust the ranking strategy based on the real-time feedback information generated during the test. (2) The testing efficiency is low. When facing high-risk scenarios, the relevant test cases may be ranked late in the execution order, which may result in defects not being discovered and exposed in a timely manner. (3) It is easy to waste resources. Low-priority test cases will take up too much test time, while key high-priority test cases cannot be fully covered.

[0018] Although greedy algorithms and genetic algorithms optimize traditional test case priority ranking methods, these algorithms also lack the ability to respond in real time to changes in risk during the testing process.

[0019] To address the aforementioned issues, this disclosure proposes a testing method applicable to scenarios such as software testing, hardware verification, and system integration testing. Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure, such as... Figure 1As shown in the illustration, the application scenario diagram of this disclosure may include, but is not limited to, a priority sorting device 110 and a test case execution device 120. The priority sorting device 110 and the test case execution device 120 can communicate via any type of wired or wireless network. Specifically, the priority sorting device 110 can be used to receive test cases input by testers. The test cases may include test objectives, test procedures, etc., for the object under test. Further, the priority sorting device 110 can sort the test cases to obtain a sequence of test cases with an execution order. The test case execution device 120 can be used to receive the test case sequence and execute the test cases in the sequence according to the execution order to test the object under test. The test case execution device 120 may include an electronic device or server for providing background management for the priority sorting device 110. Furthermore, this disclosure does not impose a specific limitation on the number of priority sorting devices 110 or test case execution devices 120. For example, the application scenario diagram of this disclosure may include one or more priority sorting devices 110 or one or more test case execution devices 120.

[0020] Figure 2 This is a flowchart illustrating the implementation of a testing method according to an embodiment of the present disclosure, including: S210. Obtain historical test data, which includes at least one test metric for each test case in multiple test cases of historical tests. S220. Based on the weight value corresponding to at least one test metric, determine the initial priority coefficient of each test case in multiple test cases of historical testing. S230. Based on the test mode and initial priority coefficient of this round of testing, determine the multiple test cases of this round of testing and the execution order of the multiple test cases of this round of testing; S240. Based on the multiple test cases and execution order of this round of testing, test the object to be tested.

[0021] In this embodiment, a test case is a set of specific test inputs, execution conditions, and expected results designed to verify whether the object under test (such as software, hardware, or a system) meets characteristic requirements or achieves expected functionalities, performance, security, or other standards. It clarifies under what circumstances what kind of data will be used for operation and pre-sets the output results. By executing test cases, testers can systematically examine the performance of the object under test in different scenarios, thereby discovering potential defects and errors.

[0022] In this embodiment of the disclosure, historical test data may include a collection of information accumulated from a series of past testing activities conducted on the object under test. It focuses on multiple test cases used in historical testing, and for each test case, it covers at least one test metric that reflects the execution performance of that test case. The details of the test metrics will be described in detail later.

[0023] In this embodiment of the disclosure, the weight value can be used to measure the relative proportion of each test indicator in evaluating the priority of test cases, which reflects the degree of contribution or influence of different test indicators to the test objective.

[0024] The initial priority coefficient is a quantitative value used to represent the relative importance of each test case in historical testing after comprehensively considering at least one test metric involved and its corresponding weight value. It can be calculated using the weight value corresponding to the test metric.

[0025] In this embodiment of the disclosure, the test mode for this round of testing clarifies the overall strategy and objectives of this round of testing, such as smoke testing focusing on rapid verification of core functions and full-function testing comprehensively covering all functions. This disclosure can comprehensively consider the objectives and requirements set by the test mode for this round of testing, as well as the importance of test cases reflected by the initial priority coefficient, determine multiple test cases that meet the requirements of this round of testing from multiple test cases, and reasonably arrange the execution order of these test cases according to the initial priority coefficient and the characteristics of the test mode.

[0026] Furthermore, based on the multiple test cases in this round of testing, and the pre-planned execution order of these test cases, the actual testing work is carried out on the object under test. In one example, testers can extract the test cases one by one according to the execution order, and perform operations and verifications on the object under test according to the detailed test steps, input data, and expected results specified in each test case. During the testing process, the actual output results of the object under test for each test case can be recorded and compared in detail with the expected results to determine whether the object under test meets the requirements in the aspects examined by the test case, thereby discovering potential defects and errors in the object under test.

[0027] By adopting the above approach and introducing a data-driven dynamic prioritization mechanism, the initial priority coefficient of each test case can be objectively calculated based on the weight values ​​corresponding to the test metrics of multiple test cases in historical testing. This reduces the problem of bias caused by subjective sorting. Furthermore, the test cases and their execution order used in this round of testing can be adjusted in conjunction with the testing mode, improving the targeting of the testing work. Finally, based on the multiple test cases and their execution order in this round of testing, the object to be tested is tested, improving the efficiency and accuracy of testing.

[0028] The following content details the testing process for the object to be tested.

[0029] Figure 3 This is a schematic diagram of a test process for an object under test according to an embodiment of the present disclosure, such as... Figure 3 As shown, the testing process includes the following steps.

[0030] S301. Determine the initial priority coefficient of each test case among multiple test cases in the history of testing.

[0031] In this embodiment of the disclosure, the initial priority coefficient of each test case can be determined based on the weight value corresponding to at least one test metric of each test case in a plurality of test cases in historical testing.

[0032] In some implementations, at least one of the following test metrics includes test case attribute level, resource utilization performance coefficient, positive reward coefficient, and negative penalty coefficient.

[0033] In this embodiment, the test case attribute level is a hierarchical representation of the relative importance and execution order of test cases within the testing system. By setting several different levels and assigning corresponding weight values ​​to each level, the value differences of each test case in the overall testing work are reflected.

[0034] In some implementations, the test case attribute level includes at least one of the following: test cases for new or changed functions, test cases that have triggered historical defects, test cases for functions related to new or changed functions, test cases that failed the test, and test cases of other attribute levels. Each test case attribute level corresponds to a preset weight value.

[0035] The following content details test cases for new or changed features.

[0036] In this embodiment of the disclosure, when a completely new function is developed for the object under test, or when existing functions are modified or optimized, the test cases corresponding to these new or modified functions can be considered as test cases for the new or modified functions. In one example, the new or modified functions of the object under test need to have problems discovered early on; therefore, the test cases for the new or modified functions have the highest weight.

[0037] The following describes test cases that have triggered historical defects.

[0038] In this embodiment, test cases that have triggered historical defects can include those that discovered defects (such as functional abnormalities, crashes, or performance degradation) in the object under test during previous testing processes or historical runtimes. In one example, the historical runtime can include the release time of the baseline object under test (a snapshot of a stable version that has been formally reviewed and approved during the development of the object under test) to the release time of the current version of the object under test. In one example, as long as a historical defect has been triggered within a specified time, the attribute level of the corresponding test case is increased accordingly; in other words, in this disclosure, the weight value of test cases that have triggered historical defects is at the second highest level.

[0039] The following describes test cases for features related to new or changed functionalities.

[0040] In this embodiment of the disclosure, new or modified functions may affect other related functions of the object under test. Test cases designed for these affected related functions are test cases for functions related to the new or modified functions. Generally, test cases for these functions related to the new or modified functions require the successful execution of other test cases as a prerequisite. In one example, when performing the first test on the object under test, the relationships between test cases need to be clearly defined before the test starts. For a set of test cases with relationships, they can be treated as a whole set of test cases during the test process.

[0041] In one example, for features related to new or changed features, defects need to be discovered early, and the corresponding test cases (i.e., test cases for features related to new or changed features) need to have the third highest weight (i.e., the second highest weight after test cases that have triggered historical defects).

[0042] The following describes test cases that failed the test.

[0043] In this embodiment of the disclosure, if the result of executing a certain test case in a previous test round does not meet expectations, that is, the test case is considered a failed test case. In one example, a failed test case (false) may fall into two categories: either the object under test has a vulnerability, or the test case design has a problem. For the case where the object under test has a vulnerability, the attribute level of the corresponding test case can be determined as a test case that has triggered a historical defect; while for a test failure due to a test case design problem, the weight value of that test case should be at the second lowest level, requiring testers to modify the test case and retest.

[0044] The following section describes test cases for other attribute levels.

[0045] Besides the aforementioned types of test cases with distinct characteristics, there may also be some test cases that cannot be categorized into the previous types. These test cases can be classified into test cases of other attribute levels. In one example, some routine functional verification test cases do not involve new or changed functions, have not triggered historical defects, are not directly related to relevant functional changes, and the object under test passes the test cases. These test cases belong to test cases of other attribute levels.

[0046] Based on the above, test case attribute levels can be divided into five levels, from highest to lowest: test cases for entirely new or modified functions, test cases that triggered historical defects, test cases for functions related to entirely new or modified functions, test cases that failed tests, and test cases of other attribute levels. Furthermore, each test case attribute level corresponds to a preset weight value. In this embodiment, the weight value for test cases for entirely new or modified functions can be 1.4, the weight value for test cases that triggered historical defects can be 1.3, the weight value for test cases for functions related to entirely new or modified functions can be 1.2, the weight value for test cases that failed tests can be 1.1, and the weight value for test cases of other attribute levels can be 1. In one example, test cases with higher weight values ​​may be executed first.

[0047] By adopting the above method, and by clearly defining the attribute level of test cases and corresponding it with the preset weight value, the importance of different types of test cases can be accurately identified. This reduces the excessive consumption of test resources on low-level test cases during the testing of the object under test, thereby improving testing efficiency, accuracy and reliability.

[0048] In some implementations, the resource utilization performance factor includes at least one of microcontroller utilization, memory utilization, and bus load.

[0049] In this embodiment of the disclosure, the microcontroller unit (MCU) utilization rate and memory utilization rate are core indicators used to measure the resource utilization of the test object during operation in historical tests. They can directly reflect the performance status of the test object, potential bottlenecks, and stability risks.

[0050] In one example, the MCU utilization rate is calculated as follows: MCU utilization rate = MCU busy time / total observation time × 100% (1) In this example, the total observation time is the entire time span for monitoring and statistically analyzing the MCU's operating status during the execution of a specific test case; it can be a pre-defined fixed time period. MCU busy time refers to the cumulative time consumed by the MCU in its actual working state and performing various tasks during the execution of a specific test case, directly reflecting the MCU's workload in the test scenario.

[0051] In one example, memory usage is calculated as follows: Memory utilization rate = used memory / total physical memory × 100% (2) In this example, used memory refers to the amount of memory currently occupied when executing a specific test case, reflecting the actual amount of memory resources consumed during the execution of that test case. Total physical memory refers to the total capacity of physical memory actually installed on the hardware device being tested, representing the upper limit of memory resources at the hardware level.

[0052] In one example, when testing the object under test, if the MCU utilization rate exceeds 80%, the priority of the current test case needs to be increased and the number of times the test case is repeated should be increased; if the memory utilization rate exceeds 95%, the priority of the current test case should also be increased and the number of times the test case is repeated should be increased.

[0053] In this embodiment of the disclosure, the Controller Area Network (CAN) bus load rate (hereinafter referred to as bus load rate) measures the bus communication busyness of the test object in historical tests, reflecting the proportion of the actual data transmitted by the bus to its maximum transmission capacity per unit time. Calculating the bus load rate helps evaluate network performance, reduce bus overload, and provide guidance for test optimization. In one example, the formula for calculating the bus load rate can be: Bus load rate = actual number of bits transmitted / maximum number of available bits on the bus × 100% (3) In this example, the actual number of bits transmitted refers to the amount of data successfully transmitted by the bus during the execution of a specific test case, measured in bits. This reflects the actual flow of data that the bus is used to transmit valid information during the test. The maximum available number of bits on the bus refers to the maximum amount of data that the bus can theoretically transmit during the execution of a specific test case, also measured in bits. This depends on factors such as the physical characteristics of the bus and the communication rate.

[0054] In this embodiment of the disclosure, the weight values ​​of MCU utilization, memory utilization, and bus load can be dynamically adjusted between 0 and 1 according to their own values.

[0055] By adopting the above method and introducing a resource utilization performance coefficient that includes at least one of microcontroller utilization, memory utilization, and bus load rate, we can understand the resource utilization of the test object during testing, thereby providing a data basis for prioritizing test cases.

[0056] In this embodiment, the execution time of test cases, the number of historical defect triggers, MCU utilization, memory utilization, and bus load rate can be real-time metrics for test cases. New or changed features and test pass rates can be risk metrics for test cases. The dependencies between the functions related to the new or changed features and the execution results of test cases can be contextual information for test cases.

[0057] In some implementations, the positive reward coefficient includes at least one of the defect detection coefficient, the critical path test duration reduction coefficient, and the test case optimization coefficient.

[0058] The following section introduces the defect detection coefficient.

[0059] In this embodiment, the defect detection coefficient focuses on the discovery of defects during the testing of the test object. It not only considers whether a defect has occurred but also emphasizes the severity of the discovered defect. High-severity defects typically refer to defects that have a significant impact on the core functionality, stability, and security of the test object, such as causing system crashes or data loss. This disclosure uses a defect detection coefficient to represent the detection of high-severity defects by a test case. For example, if a test case detects a high-severity defect, its defect detection coefficient is 1; if it does not detect a high-severity defect, its defect detection coefficient is 0. Furthermore, test cases that detect high-severity defects are rewarded to increase their priority.

[0060] In one example, defects in the object under test can be graded according to their severity, such as into severe, moderate, and minor levels, with high-severity defects being classified as severe. The specific grading can be determined based on the impact of the defect on the object under test.

[0061] In this embodiment of the disclosure, when a test case can detect a high-severity defect in historical tests, the defect detection coefficient of the test case can be 1, and the corresponding weight value can be 1.1; when a test case does not detect a high-severity defect, the defect detection coefficient of the test case can be 0, and the corresponding weight value can be 1.

[0062] The following content introduces the critical path test duration reduction factor.

[0063] In this embodiment of the disclosure, the critical path can refer to the path formed by testing the object under test using test cases, which defines the shortest time to complete the test of the object under test. The critical path test duration reduction factor can refer to an indicator of the effectiveness in reducing the test duration of the critical path, reflecting the degree to which the time spent on the critical path test process is reduced compared to the original time or baseline time of the test case after optimization of the test cases.

[0064] In one example, during historical testing, if the time spent executing a test case is reduced compared to the original time or baseline time, the critical path test duration reduction factor for that test case can be set to 1, with a corresponding weight value of 1.05; if the time spent executing a test case is not reduced compared to the original time or baseline time, the critical path test duration reduction factor for that test case can be set to 0, with a corresponding weight value of 1.

[0065] The following section describes the test case optimization coefficients.

[0066] In the embodiments of this disclosure, as the test object is continuously iterated and the testing work continues, the test cases may face some problems, such as excessive redundancy of test cases, weak test targeting, and inability to discover new defects. The phenomenon that causes these problems can be called the "pesticide effect", that is, the ability to discover defects will gradually decrease when the same test cases are used repeatedly.

[0067] The test case optimization coefficient is an indicator set to measure the effectiveness of test case optimization and improvement. It focuses on whether the overall quality and effectiveness of test cases are improved through reasonable methods, such as deleting redundant content, optimizing test steps, and adjusting expected results.

[0068] In one example, this disclosure adjusts the weight value corresponding to the test case optimization coefficient to the baseline value for test cases whose testing functions have not changed within one year, no vulnerabilities of the test object have occurred, and whose test pass rate has remained stable; for test cases that have been optimized and modified, the weight value of their test case optimization coefficient is increased accordingly.

[0069] For example, if a test case is optimized and adjusted, such as by adding test steps for new features, the test case optimization coefficient can be set to 1, and the corresponding weight value can be set to 1.03. If the test case is not optimized and adjusted, or if the optimization measures do not achieve the expected results, the test case optimization coefficient can be set to 0, and the corresponding weight value can be set to 1.

[0070] By adopting the above method, this disclosure introduces a positive reward coefficient, which can quantify the actual contribution of test cases in defect detection, critical path verification and self-optimization, and transform it into an additive priority gain, thereby improving the accuracy and objectivity of test case priority calculation.

[0071] In some implementations, the negative penalty coefficient includes at least one of the risk coverage coefficient, the vulnerability detection coefficient, and the resource consumption ratio of a specific test case; wherein the specific test case includes test cases with a priority lower than or equal to a preset priority requirement.

[0072] The following content introduces the risk coverage factor.

[0073] In this embodiment, the risk coverage coefficient focuses on the coverage of test cases in historical testing of high-risk scenarios (such as core business logic errors, data calculation errors, performance bottlenecks or degradation) of the object under test. When a test case fails to cover predefined high-risk scenarios in the object under test, the risk coverage coefficient is triggered and has a negative impact on the test case.

[0074] This disclosure can determine the coverage of high-risk scenarios of the test case for the object under test during the testing process. If it is found during the testing process that a predefined high-risk scenario has not been executed by the test case, it is determined that the test case does not cover the high-risk scenario.

[0075] In one example, if a test case fails to cover predefined high-risk scenarios in the object under test in historical tests, the risk coverage coefficient of the test case can be 0, and the corresponding weight value can be 0.97; if a test case can cover predefined high-risk scenarios in the object under test, the risk coverage coefficient of the test case can be 1, and the corresponding weight value can be 1.

[0076] The following content describes the vulnerability detection coefficient.

[0077] In this embodiment of the disclosure, the vulnerability detection coefficient is used to quantitatively evaluate the vulnerability detection performance of a test case on the object under test. Here, vulnerabilities may include various types, such as security vulnerabilities (e.g., buffer overflow vulnerabilities), functional vulnerabilities (e.g., calculation errors affecting the normal operation of business logic), performance vulnerabilities (e.g., excessively long response times in high-concurrency scenarios), etc. This disclosure can determine the vulnerability detection coefficient of a test case based on the detection performance of the test case on these vulnerabilities in historical tests.

[0078] For example, in historical testing, if a test case does not detect a vulnerability in the object under test, the vulnerability detection coefficient of that test case can be set to 0, and the corresponding weight value can be 0.99; if a test case detects a vulnerability in the object under test, the vulnerability detection coefficient of that test case can be set to 1, and the corresponding weight value can be 1.

[0079] The following content describes the resource usage ratio for a specific test case.

[0080] In this embodiment of the disclosure, specific test cases include test cases with a priority level lower than or equal to a preset priority requirement. The preset priority requirement may include the weight value corresponding to the attribute level that meets the preset requirement among the test case attribute levels. In one example, this disclosure may determine test cases that fail the test as attribute levels that meet the preset requirement, and their corresponding weight values ​​may be considered as the preset priority requirement.

[0081] Based on the foregoing, the weight value of a test case that fails the test is 1.1. Accordingly, specific test cases (i.e. test cases that are lower than or equal to the preset priority requirement) mainly include test cases that fail the test and test cases of other attribute levels, that is, test cases whose weight values ​​are 1.1 and 1 corresponding to the test case attribute levels.

[0082] In this embodiment of the disclosure, the resource usage ratio of a specific test case may include the proportion of host computer resources used during the execution of the test case. Here, host computer resources may include the utilization rate of the central processing unit, the utilization rate of storage resources, and the utilization rate of network bandwidth, etc. By setting corresponding weights for various resources of the host computer, and then using the weights and various resources for weighted summation, the resource usage ratio of the test case can be obtained.

[0083] This disclosure allows setting a resource usage threshold. If the resource usage ratio of a specific test case in historical testing is greater than or equal to the resource usage threshold, the weight value corresponding to the negative penalty coefficient of that test case can be 0.95. If the resource usage ratio of a specific test case in historical testing is less than the resource usage threshold, the weight value corresponding to the negative penalty coefficient of that test case can be 1.

[0084] By adopting the above method, this disclosure can accurately quantify the negative performance of test cases in risk control, vulnerability detection and resource utilization by introducing a negative penalty coefficient, and convert it into an additive priority decay value, thereby improving the accuracy and objectivity of test case priority evaluation, and thus improving the reliability of testing the test object.

[0085] Furthermore, this disclosure can utilize the weight value corresponding to at least one test metric of each test case to determine the initial priority coefficient of each test case. In one example, the formula for calculating the initial priority coefficient is: Initial priority coefficient = 1 × (1 + MCU utilization rate) × (1 + memory utilization rate) × (1 + bus load rate) × weight value of test case attribute level × weight value of defect detection coefficient × weight value of critical path test duration reduction coefficient × weight value of test case optimization coefficient × weight value of risk coverage coefficient × weight value of vulnerability detection coefficient × weight value of resource utilization ratio of specific test case.

[0086] This disclosure determines the initial priority coefficient of each test case based on the weight value corresponding to at least one test metric of multiple test cases in historical testing. This method comprehensively considers multiple different types of test metrics, which can fully and accurately reflect the actual situation of test cases, and thus obtain objective and accurate initial priority coefficients, providing a reliable basis for subsequent test resource allocation and test process planning.

[0087] like Figure 3 As shown, the testing process also includes the following steps.

[0088] S302. Determine the multiple test cases and their execution order for this round of testing.

[0089] In this embodiment of the disclosure, multiple test cases and their execution order in this round of testing can be determined based on a mathematical model between the test mode and the initial priority coefficient of this round of testing.

[0090] In some implementations, based on the test mode and initial priority coefficient of this round of testing, the execution order of multiple test cases in this round of testing is determined, including: In this round of testing, the test mode is smoke testing. Based on the initial priority coefficient, multiple initial smoke test cases that meet the preset requirements are determined from multiple test cases in the historical test. Based on the initial smoke test case and the exploration rate, the first smoke test case is determined; and, from multiple test cases in the historical test, the second smoke test case is determined from the other test cases besides the initial smoke test case. Based on the first smoke test case and the second smoke test case, determine the multiple test cases for this round of testing and the execution order of the multiple test cases for this round of testing.

[0091] In this embodiment of the disclosure, considering the tight project schedule, there is often not enough time to conduct comprehensive testing. In this case, the testing mode for this round of testing can be smoke testing to quickly verify whether the basic functions of the object under test are normal. This disclosure addresses this practical application scenario by sorting multiple test cases from historical tests in descending order of initial priority coefficients, and determining multiple initial smoke test cases that meet preset requirements from these test cases.

[0092] Here, the preset requirements can include preset quantity requirements and preset test duration requirements. The preset quantity requirement can include the number of initial smoke test cases to be selected. For example, multiple initial smoke test cases that meet the preset quantity requirement are selected from a list of sorted test cases. The preset test duration requirement refers to the total time limit for the entire smoke test. In other words, when determining the initial smoke test cases, it must be ensured that the sum of the estimated test durations of the selected test cases does not exceed the preset total test duration. For example, if the preset test duration requirement (i.e., the preset total test duration) is two hours, then when selecting test cases from the list of sorted test cases, the estimated duration of each test case needs to be summed to ensure that the total execution time of the multiple initial smoke test cases does not exceed two hours.

[0093] In this embodiment, smoke testing may fall into a local optimum, meaning that test cases not identified as initial smoke test cases may harbor serious defects. Since the determination of initial smoke test cases is based on initial priority coefficients and resource constraints, it is difficult to comprehensively cover all functions. This may result in some functions being relegated to a relatively minor position within the test object. If problems arise, they can significantly impact the basic operation of the test object, and these problems cannot be detected by the initial smoke test cases, thus affecting subsequent testing processes and quality control of the test object.

[0094] To address this issue, this disclosure determines the first smoke test case from the initial smoke test cases based on the initial smoke test cases and the exploration rate. Here, the exploration rate is a parameter used to balance the determinism and exploratory nature of testing. It determines the proportion or extent to which new, potentially undertested situations are explored based on the initial smoke test cases. For example, if the exploration rate is set to 20%, then 20% of the initial smoke test cases may be randomly selected or selected according to certain rules to form the first smoke test case.

[0095] Furthermore, to broaden the coverage of smoke testing, in addition to the first smoke test case determined based on the initial priority coefficient and exploration rate, a number of second smoke test cases need to be randomly selected from multiple test cases in historical testing, excluding the initial smoke test cases. In one example, the number of second smoke test cases is equal to the number of first smoke test cases.

[0096] By merging the first smoke test case and the second smoke test case, multiple test cases for this round of testing are determined. Furthermore, the execution order of these multiple test cases can be determined based on the initial priority coefficient. In one example, the formula for determining the multiple test cases for this round of testing is: Multiple test cases for this round of testing = Multiple initial smoke test cases × Exploration rate (…). (1) + multiple second smoke test cases. For example, assuming the initial number of smoke test cases is 500 and the exploration rate is 20%, then the number of first smoke test cases is 100. Accordingly, 100 second smoke test cases are randomly selected from the test cases other than the initial smoke test cases in the multiple test cases of the historical test. The 100 first smoke test cases and the 100 second smoke test cases are merged to obtain 200 test cases used for smoke testing.

[0097] In some implementations, the exploration rate decreases linearly with the number of test rounds of the smoke test.

[0098] In this embodiment of the disclosure, smoke testing can typically be performed multiple times at different stages of the research and development of the test object, and each smoke test can be referred to as a test round. As the number of smoke test rounds increases, the exploration rate can gradually decrease at a fixed rate. In one example, if the initial value of the exploration rate is 20%, and the exploration rate decreases by 0.5% after each smoke test round, then after several smoke test rounds, the exploration rate will gradually approach its minimum value (such as a preset decay threshold).

[0099] Using the above method, the first smoke test case is determined based on the initial smoke test case and the exploration rate. At the same time, the second smoke test case is determined from the other test cases in the historical test cases except for the initial smoke test case. Combining the two to determine multiple test cases for this round of testing achieves a balance between exploration and utilization, expands the test coverage, and improves the quality and reliability of smoke testing.

[0100] In some implementations, it also includes: If the exploration rate is less than or equal to the preset decay threshold, the exploration rate will be adjusted to the initial value.

[0101] In this embodiment of the disclosure, the preset decay threshold can be a pre-set value used to determine whether the exploration rate has decayed to the point where it needs to be readjusted. It can be determined based on factors such as the specific needs of the test object, the test objectives, and historical experience.

[0102] For example, the initial exploration rate is 20%, which decreases by 0.5% after one round of smoke testing. The preset decay threshold can be set to 5%. When the exploration rate decreases to 5%, it is readjusted back to 20%.

[0103] By using the above method, exploratory testing can be reactivated by resetting the exploration rate to its initial value when it reaches a preset decay threshold. This allows multiple test cases to cover all functions and scenarios of the object under test, improving the completeness and reliability of the test.

[0104] In some implementations, determining the execution order of multiple test cases in this round of testing, based on the test mode and initial priority coefficient of this round of testing, further includes: If the test mode for this round of testing is full-function testing, and the time interval between this round of testing and multiple historical tests meets the preset conditions, then a historical test is randomly selected from multiple historical tests, and the execution order of multiple test cases for this round of testing is determined based on the randomly selected historical test.

[0105] In this embodiment of the disclosure, full-function testing includes comprehensive testing of all functions of the object under test, which means that it is necessary to cover all functional modules of the object under test and test cases under different usage scenarios.

[0106] This disclosure can preset a time interval threshold. When the interval between the start time of the current test and the completion time of multiple historical tests is less than the time interval threshold, the preset condition can be considered met. For example, if the time interval threshold is three months, and the current test is conducted within two months after the completion of a certain historical test, then the current test and the historical test meet the preset condition.

[0107] Furthermore, from a plurality of historical tests that meet the aforementioned preset conditions, one historical test is selected randomly. Here, random selection does not follow any specific pattern or preference, and each historical test has an equal probability of being selected.

[0108] Based on randomly selected historical tests, multiple test cases used in those historical tests and their execution order are extracted to determine the multiple test cases for this round of testing and their execution order.

[0109] For example, the test mode for this round of testing is full-function testing, and there are three historical tests to choose from if the time interval meets preset conditions: historical test 1, historical test 2, and historical test 3. Here, historical test 3 can be considered as the previous round of testing relative to this round. In this embodiment of the disclosure, the above three historical tests can be randomly selected, and the multiple test cases corresponding to the selected historical test and the execution order of the multiple test cases are used as the multiple test cases and execution order of this round of testing. In this way, this disclosure utilizes the experience playback mechanism to conduct multiple rounds of iterative full-function testing on the object under test at a certain stage. A historical test can be randomly selected from this stage, and multiple test cases and execution order of this round of testing can be determined for that historical test, thereby breaking the high correlation between continuous experience.

[0110] By adopting the above method, multiple historical tests that meet preset time intervals are randomly selected to determine multiple test cases and execution order for this round of full-function testing. This reduces the need to design test cases and plan execution order from scratch, thus improving testing efficiency. At the same time, random selection of historical tests breaks the limitations of conventional testing and improves test coverage and reliability.

[0111] like Figure 3 As shown, the testing process also includes the following steps.

[0112] S303, Execute multiple test cases in this round of testing.

[0113] In this embodiment of the disclosure, multiple test cases can be executed sequentially, starting with the test case of highest priority. In one example, testers can operate in the test environment according to the input data and execution steps specified in the test cases and record the actual execution results.

[0114] S304. Save at least one test metric from this round of testing.

[0115] S305. Conduct a risk assessment for this round of testing.

[0116] S306. Update the weight values ​​corresponding to the test metrics.

[0117] S307. Adjust the execution order of test cases in this round of testing.

[0118] In some implementations, it also includes: Save at least one test metric for each test case in the multiple test cases of this round of testing; Update the weight value corresponding to at least one of the test metrics based on the test results of the test object.

[0119] In this embodiment of the disclosure, after the multiple test cases in this round of testing are executed in the order of execution, at least one test metric in each test case can be stored in memory according to coefficient labels (such as resource utilization performance coefficient, test case attribute level, etc.).

[0120] Specifically, this disclosure can store the execution time, defect information, and resource consumption information of the test cases in this round of testing. Here, execution time can refer to the time spent completing a test case from start to finish; defect information can include the number, type, and severity of defects in the object under test discovered during the execution of the test cases; resource consumption can refer to the resources consumed during the execution of the test cases, or the resources consumed when running the object under test based on the test cases.

[0121] Furthermore, based on at least one test metric generated from multiple test cases executed in this round of testing, a risk assessment is performed on this round of testing. This involves calculating the risk value of the test scenario for this round of testing. The sources of risk are broad, potentially involving the test environment, test data, and the object under test itself. In one example, calculating the risk value of the test scenario for this round of testing requires comprehensive consideration of the probability and severity of the risk's occurrence. For instance, if the probability of a test scenario in this round of testing being high, and its occurrence would severely impact the object under test, then the risk value of that test scenario would be relatively high.

[0122] In the actual testing process of the test object disclosed herein, the test results may differ from the expected results, and the importance of the issues reflected by different test indicators may also change. Therefore, it is necessary to dynamically adjust the weight values ​​corresponding to at least one test indicator obtained in this round of testing based on the actual test results, in order to update the mathematical model used to record the test indicators.

[0123] In one example, the test status of the object under test can include multiple aspects of information, such as the object's functional performance, performance data, and defect information. Furthermore, based on the test objectives and business requirements, the test status is analyzed to determine the changes in the importance of each test metric. If the test objective is to ensure the stability of the object under test's functionality, and a certain functional module is found to frequently exhibit serious defects, then the corresponding test metrics for test cases related to the functional integrity and stability of that module should be given higher weight.

[0124] Using the above method, test metrics for multiple test cases in this round of testing can be saved, providing a data foundation for subsequent test analysis of this round of testing; and dynamically updating the weight values ​​of test metrics can make the test evaluation system closely align with the actual characteristics and test objectives of the object under test, thereby improving the reliability and accuracy of subsequent tests on the object under test.

[0125] S308. Determine whether to proceed with the next round of testing. If yes, execute S301, where the current round of testing can be used as a historical test for the next round. If no, end the process.

[0126] In this embodiment of the disclosure, for scenarios involving multiple rounds of testing within a short period, this judgment mechanism can be used to determine whether to proceed with the next round of testing. If the time span between two adjacent rounds of testing is large, the current round of testing can be terminated directly, and the relevant test data can be obtained at the start of the next round of testing, thereby enabling the next round of testing to proceed.

[0127] It is understandable that, in this disclosure, after each round of testing on the test object is completed, the weight value corresponding to at least one test metric saved in the current round can be updated.

[0128] In some implementations, it also includes: Determine the test duration for each test case; The test duration of the test cases is compared with the preset duration threshold to determine the execution time period of the test cases in this round of testing.

[0129] In this embodiment of the disclosure, the test duration of a test case can be determined from the test data of the corresponding test case in historical tests. For example, if the same test case has been executed in historical tests, the test duration of that test case can be determined from the test data of the test case executed in historical tests.

[0130] Furthermore, the test duration of the test case is compared with a preset duration threshold. If the test duration is greater than or equal to the preset duration threshold, the execution time period corresponding to this situation (or the first execution time period) is determined; if the test duration is less than the preset duration threshold, the execution time period corresponding to this situation (or the second execution time period) is determined.

[0131] Figure 4 This is a schematic diagram illustrating the determination of a test duration segment according to an embodiment of the present disclosure.

[0132] like Figure 4 As shown, this disclosure allows setting two execution time periods: from 9:00 AM to 9:00 PM on the test start date and from 9:00 PM to 9:00 AM on the following day. Test cases with a test duration less than a preset duration threshold can be executed from 9:00 AM to 9:00 PM on the test start date (i.e., the second execution time period), while test cases with a test duration greater than or equal to the preset duration threshold can be executed from 9:00 PM to 9:00 AM on the following day (i.e., the first execution time period).

[0133] For example, the preset duration threshold can be 0.5 hours. If the test case's test duration exceeds 0.5 hours, it will be scheduled to be executed from 21:00 on the test start date to 9:00 on the next test day (i.e., the first execution time period); if the test case's test duration does not exceed 0.5 hours, it will be scheduled to be executed from 9:00 on the test start date to 21:00 on the test start date (i.e., the second execution time period).

[0134] If all test cases with a test duration of no more than 0.5 hours are executed before 21:00 on the test start date, test cases with a test duration of more than 0.5 hours can be executed ahead of schedule; if all test cases with a test duration of more than 0.5 hours are executed before 9:00 on the next day, test cases with a test duration of no more than 0.5 hours can be executed ahead of schedule.

[0135] By using the above method, the test duration of test cases is determined and compared with a preset duration threshold, which can accurately plan the execution time of test cases in this round of testing, reduce the conflict and chaos of test case execution time, improve the orderliness and efficiency of the testing process, and thus improve the overall quality and reliability of testing.

[0136] Figure 5 This is a functional relationship between the exploration rate and the number of test rounds according to an embodiment of the present disclosure. Figure 1 .

[0137] like Figure 5As shown, the initial exploration rate is 20%, and it decreases by 0.5% with each round of smoke testing. Without external interference, the exploration rate will decrease to its lowest value after the 40th round of smoke testing.

[0138] This disclosure can determine the lower limit of the exploration rate based on a preset decay threshold, and adjust the exploration rate to the initial value when the exploration rate is less than or equal to the preset decay threshold.

[0139] Figure 6 This is a functional relationship between the exploration rate and the number of test rounds according to an embodiment of the present disclosure. Figure 2 .

[0140] like Figure 6 As shown, the preset decay threshold can be 5%. Based on an initial exploration rate of 20% in the first round of smoke testing, when the smoke test reaches the 30th round, the exploration rate decays to the preset decay threshold (i.e., 5%). At this point, this disclosure can adjust the exploration rate to 20% to determine multiple test cases in the 30th round of smoke testing. Figure 6 In the game, the exploration rate needs to be adjusted back to its initial value every 30 rounds of smoke testing.

[0141] In summary, the dynamic priority optimization framework for test cases proposed in this disclosure can achieve dynamic optimization of test case priorities in the following ways: (1) Real-time feedback collection: During the execution of test cases, various real-time data are collected, including new or changed functions, functions related to new or changed functions, the pass rate of test cases, the frequency of defect triggering, and resource consumption, etc. (2) Risk quantification: Based on historical test data and real-time feedback data of this round of testing, the risk value of each test scenario is dynamically calculated, such as the probability of defect occurrence and the possible impact range of the defect; (3) Application of test index mathematical model engine: By constructing a test index mathematical model, the priority of test cases is dynamically adjusted with the goal of maximizing long-term defect detection efficiency; (4) Establishment of closed-loop optimization mechanism: Update the mathematical model based on the test results of this round of testing, thereby forming a continuous optimization cycle.

[0142] Based on the dynamic priority optimization framework for test cases proposed in the embodiments of this disclosure, this disclosure can achieve at least the following effects: (1) Dynamic weight allocation: It can improve the priority of test cases in high-risk scenarios in real time, rather than relying on pre-set rules; (2) Reinforcement learning driven: The test sequence is optimized through trial and error learning, which can adapt to complex and non-linear risk changes; (3) Multi-dimensional feedback fusion: Combine execution time, defect severity, coverage and other multi-source data to construct a comprehensive state space.

[0143] This disclosure can be applied to hardware-in-the-loop testing of the vehicle control unit (VCU) in autonomous driving systems, and can achieve efficient defect detection in software testing, hardware verification, system integration testing and other scenarios.

[0144] This disclosure combines real-time feedback mechanisms and mathematical model enhancement techniques to achieve dynamic adaptive priority optimization of test cases, which has significant technological progress and practical value compared with existing technologies.

[0145] This disclosure also proposes a testing apparatus. Figure 7 This is a schematic diagram of the structure of a test apparatus 700 according to an embodiment of the present disclosure, comprising: The data acquisition module 710 is used to acquire historical test data, which includes at least one test metric for each test case in multiple test cases of historical tests. The priority determination module 720 is used to determine the initial priority coefficient of each test case in multiple test cases of historical testing based on the weight value corresponding to at least one test metric. The execution order determination module 730 is used to determine the execution order of multiple test cases in this round of testing based on the test mode and initial priority coefficient of this round of testing. Test module 740 is used to test the object to be tested based on multiple test cases and execution order in this round of testing.

[0146] In some implementations, the execution order determination module 730 is used for: In this round of testing, the test mode is smoke testing. Based on the initial priority coefficient, multiple initial smoke test cases that meet the preset requirements are determined from multiple test cases in the historical test. Based on the initial smoke test case and the exploration rate, the first smoke test case is determined; and, from multiple test cases in the historical test, the second smoke test case is determined from the other test cases besides the initial smoke test case. Based on the first smoke test case and the second smoke test case, determine the multiple test cases for this round of testing and the execution order of the multiple test cases for this round of testing.

[0147] In some implementations, the exploration rate decreases linearly with the number of test rounds of the smoke test.

[0148] In some implementations... Figure 8 This is a schematic diagram of the structure of a test apparatus 800 according to an embodiment of the present disclosure, including: The exploration rate adjustment module 850 is used to adjust the exploration rate to the initial value when the exploration rate is less than or equal to a preset decay threshold.

[0149] In some implementations, the execution order determination module 730 is further configured to: If the test mode for this round of testing is full-function testing, and the time interval between this round of testing and multiple historical tests meets the preset conditions, then a historical test is randomly selected from multiple historical tests, and the execution order of multiple test cases for this round of testing is determined based on the randomly selected historical test.

[0150] In some implementations, at least one of the following test metrics includes test case attribute level, resource utilization performance coefficient, positive reward coefficient, and negative penalty coefficient.

[0151] In some implementations, the test case attribute level includes at least one of the following: test cases for new or changed functions, test cases that have triggered historical defects, test cases for functions related to new or changed functions, test cases that failed the test, and test cases of other attribute levels. Each test case attribute level corresponds to a preset weight value.

[0152] In some implementations, the resource utilization performance factor includes at least one of microcontroller utilization, memory utilization, and bus load.

[0153] In some implementations, the positive reward coefficient includes at least one of the defect detection coefficient, the critical path test duration reduction coefficient, and the test case optimization coefficient.

[0154] In some implementations, the negative penalty coefficient includes at least one of the risk coverage coefficient, the vulnerability detection coefficient, and the resource consumption ratio of a specific test case; wherein the specific test case includes test cases that are lower than or equal to the preset priority requirement.

[0155] In some implementations, a weight update module 860 is also included, for: Save at least one test metric for each test case in the multiple test cases of this round of testing; Update the weight value corresponding to at least one of the test metrics based on the test results of the test object.

[0156] In some implementations, an execution time period determination module 870 is also included, for: Determine the test duration for each test case; The test duration of a test case is compared with a preset duration threshold to determine the execution time period of the test case in this round of testing.

[0157] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0158] The acquisition, storage, and application of personal information by users involved in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0159] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0160] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0161] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0162] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0163] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as test methods. For example, in some embodiments, the test method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the test method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform test methods by any other suitable means (e.g., by means of firmware).

[0164] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0168] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0169] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0170] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0171] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A testing method, comprising: Obtain historical test data, which includes at least one test metric for each test case in multiple test cases from historical tests; Based on the weight value corresponding to the at least one test metric, determine the initial priority coefficient of each test case in the multiple test cases of the historical test; Based on the test mode of this round of testing and the initial priority coefficient, determine the multiple test cases of this round of testing and the execution order of the multiple test cases of this round of testing; Based on the multiple test cases and the execution order of this round of testing, the object to be tested is tested.

2. The method according to claim 1, wherein, The determination of multiple test cases and their execution order in this round of testing, based on the test mode and the initial priority coefficient, includes: When the test mode of this round of testing is smoke testing, multiple initial smoke test cases that meet the preset requirements are determined from multiple test cases of the historical tests according to the initial priority coefficient. Based on the initial smoke test case and the exploration rate, a first smoke test case is determined; and a second smoke test case is determined from the test cases other than the initial smoke test case from the multiple test cases in the historical tests. Based on the first smoke test case and the second smoke test case, determine the execution order of multiple test cases in this round of testing.

3. The method according to claim 2, wherein, The exploration rate decreases linearly with the number of test rounds of the smoke test.

4. The method according to claim 3, further comprising: If the exploration rate is less than or equal to a preset decay threshold, the exploration rate is adjusted to its initial value.

5. The method according to any one of claims 1-4, wherein determining the multiple test cases of the current test and the execution order of the multiple test cases of the current test based on the test mode of the current test and the initial priority coefficient further includes: If the test mode of this round of testing is full-function testing, and the time interval between this round of testing and multiple historical tests meets preset conditions, a historical test is randomly selected from the multiple historical tests, and the execution order of multiple test cases in this round of testing is determined based on the randomly selected historical test.

6. The method according to claim 1, wherein, The at least one test metric includes at least one of the following: test case attribute level, resource utilization performance coefficient, positive reward coefficient, and negative penalty coefficient.

7. The method according to claim 6, wherein, The test case attribute level includes at least one of the following: test cases for new or changed functions, test cases that have triggered historical defects, test cases for functions related to new or changed functions, test cases that failed the test, and test cases with other attribute levels. Each of the test case attribute levels corresponds to a preset weight value.

8. The method according to claim 6, wherein, The resource utilization performance coefficient includes at least one of microcontroller utilization, memory utilization, and bus load.

9. The method according to claim 6, wherein, The positive reward coefficient includes at least one of the following: defect detection coefficient, critical path test duration reduction coefficient, and test case optimization coefficient.

10. The method according to claim 6, wherein, The negative penalty coefficient includes at least one of the risk coverage coefficient, vulnerability detection coefficient, and resource consumption ratio of a specific test case; wherein the specific test case includes test cases with a priority lower than or equal to the preset priority requirement.

11. The method according to any one of claims 1-10, further comprising: Save at least one test metric for each test case in the multiple test cases of this round of testing; Based on the test results of the object to be tested, update the weight value corresponding to at least one of the saved test indicators.

12. The method according to claim 1, further comprising: Determine the test duration for the test cases; The test duration of the test case is compared with a preset duration threshold to determine the execution time period of the test case in this round of testing.

13. A testing apparatus, comprising: The data acquisition module is used to acquire historical test data, which includes at least one test metric for each test case in multiple test cases of historical tests. The priority determination module is used to determine the initial priority coefficient of each test case in the multiple test cases of the historical test based on the weight value corresponding to the at least one test indicator. The execution order determination module is used to determine the execution order of multiple test cases in this round of testing based on the test mode of this round of testing and the initial priority coefficient. The testing module is used to test the object to be tested based on the multiple test cases in this round of testing and the execution order.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images