Coverage rate collection method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of simulation testing, and discloses a coverage rate collection method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining the coverage rates of a plurality of test cases for testing a hardware circuit and N expected coverage rates; the coverage rate is divided into N types; determining the maximum coverage rate in each type to obtain N maximum coverage rates; comparing the N maximum coverage rates with the corresponding N expected coverage rates, and judging whether the N maximum coverage rates meet preset conditions or not; if not, reordering the plurality of test cases to obtain a plurality of test cases of a second order; and testing the hardware circuit based on the plurality of test cases of the second sequence to obtain a plurality of coverage rates corresponding to the plurality of test cases of the second sequence, and iterating the coverage rate test of the hardware circuit until the plurality of generated coverage rates reach a preset condition. Therefore, the technical effect of dynamically adjusting and optimizing the test case in the coverage rate test process is achieved.

Description

Technical Field

[0001] The present invention relates to the field of simulation test technology, and in particular to a coverage collection method, device, computer equipment and storage medium. Background Art

[0002] Coverage collection faces multiple challenges. Data collection is complex and requires verification personnel to have in-depth knowledge of circuits and rich verification experience in order to design test cases that fully cover edge cases. In addition, adjusting test cases after analyzing the collected coverage will extend the overall test cycle.

[0003] Therefore, how to adjust and optimize test cases is a technical problem that needs to be solved urgently in the process of collecting coverage. Summary of the invention

[0004] In view of this, the present invention provides a coverage collection method, apparatus, computer equipment and storage medium to solve the problem of how to adjust and optimize test cases.

[0005] In a first aspect, the present invention provides a method for collecting coverage, the method comprising:

[0006] Obtaining M coverages obtained after testing a hardware circuit with multiple test cases, and N expected coverages of different types, where M≥2, N≥2 and M and N are positive integers; the multiple test cases are sorted in the first order;

[0007] Divide the M coverages into N types, each type including at least one coverage;

[0008] Determine the maximum coverage of each of the above types and obtain N maximum coverages;

[0009] Compare the N maximum coverage rates with the corresponding N different types of expected coverage rates to determine whether the N maximum coverage rates meet a preset condition, where the preset condition is that each of the maximum coverage rates is greater than or equal to the expected coverage rate of the corresponding type;

[0010] If not, reorder the multiple test cases to obtain multiple test cases in a second order;

[0011] The hardware circuit is tested based on the multiple test cases arranged in the second order to obtain multiple coverage rates corresponding to the multiple test cases arranged in the second order, and the coverage rate test of the hardware circuit is iterated until the generated multiple coverage rates reach the preset conditions.

[0012] The present invention provides a method for processing server data, which has the following advantages:

[0013] The M coverage rates obtained after testing the hardware circuit with multiple test cases are divided into N types, and the N maximum coverage rates are compared with the expected coverage rates of N different types to determine whether the N coverage rates are not less than the expected coverage rate of the corresponding type. If not, the order of multiple test cases is adjusted, the hardware circuit is retested, the corresponding coverage rate is reacquired, and the coverage rate test of the hardware circuit is iterated until the multiple coverage rates generated meet the preset conditions. Thus, the technical effect of dynamically adjusting and optimizing the test cases during the coverage rate test process is achieved, so that the iterative process of the coverage rate test of the hardware circuit can converge faster. At the same time, it avoids problems such as slow project progress and high human errors caused by insufficient experience of designers, and improves the efficiency and accuracy of the entire coverage rate test process.

[0014] In an optional implementation, the first order sorting is to sort the multiple test cases from high to low priority;

[0015] The plurality of test cases are reordered to obtain a plurality of test cases in a second order, including:

[0016] Generate M error rates according to the M coverage rates and the N different types of expected coverage rates;

[0017] Compare the magnitude relationships among the M error rates to obtain the priorities of the multiple test cases.

[0018] According to the priorities of the multiple test cases, the multiple test cases are reordered to obtain multiple test cases in a second order.

[0019] Specifically, in the steps of the method, the priority of the corresponding test case is determined by determining the error rate between the coverage rate and the expected coverage rate, and then the multiple test cases are reordered according to the priorities of the multiple test cases. This achieves dynamic adjustment of the weight coefficients of different test cases, further speeds up the convergence speed of the iterative process, and improves the efficiency of coverage testing.

[0020] In an optional implementation, M error rates are generated according to the M coverage rates and the N different types of expected coverage rates, including:

[0021] Based on the N different types of expected coverage, M coverages are respectively generated, and M·N characteristic error rates corresponding to the N different types of expected coverages are generated;

[0022] Based on a preset rule, the M error rates of the M coverage rates are determined from the M·N feature error rates.

[0023] Specifically, in the steps of the method, by generating a characteristic error rate corresponding to each type of each coverage, and then determining the error rate of each coverage based on a preset rule, it is possible to determine an error rate of the coverage from multiple error rates between the coverage and different types of expected coverage, thereby achieving the technical effect of determining the error rate of each coverage.

[0024] In an optional implementation, the plurality of test cases are reordered to obtain a plurality of test cases in a second order, including:

[0025] Compare the magnitude relationships between the above error rates of each test case to obtain the minimum error rate of each test case;

[0026] According to the type of expected coverage corresponding to each of the above minimum error rates, the above multiple test cases are divided into N types, each of the above types includes at least one test case;

[0027] Determine a first test case of a first type by judging whether each of the minimum error rates is less than a first threshold, wherein the minimum error rate of the first test case is less than the first threshold, and the first type includes at least one test case;

[0028] Lower the priority of the first type of test cases mentioned above;

[0029] Or, by judging whether each of the above minimum error rates is greater than the second threshold and less than the third threshold, determining a second test case of the second type, the minimum error rate of the above second test case is greater than the above second threshold, and the above second type includes at least one test case;

[0030] Increase the priority of the second type of test cases mentioned above;

[0031] Or, by judging whether each of the above minimum error rates is greater than or equal to the above third threshold, a third test case is determined, and the minimum error rate of the above third test case is greater than or equal to the above third threshold;

[0032] The third test case is deleted from the multiple test cases.

[0033] Specifically, in the steps of the method, the minimum error rate of each test case is determined by comparing the error rate of each coverage rate, and then the test cases are classified based on the type of expected coverage rate corresponding to each minimum error rate. Then, the first, second, and third test cases are determined by judging the relationship between each minimum error rate and the first, second, and third thresholds. According to the relationship between each minimum error rate and multiple thresholds, the priority of test cases of the same type as the first and second test cases is adjusted or the third test case is deleted from the multiple test cases, thereby achieving the technical effect of dynamically adjusting the test cases based on the error rate, thereby improving the efficiency and accuracy of the entire testing process.

[0034] In an optional implementation, after obtaining the plurality of test cases in the second sorting, the method further includes:

[0035] Obtain a first minimum error rate and a second minimum error rate for each of the test cases, where one test case corresponds to one first minimum error rate and one second minimum error rate, the first minimum error rate being generated after each test case tests the hardware circuit according to the first sorting, and the second minimum error rate being generated after each test case tests the hardware circuit according to the second sorting;

[0036] Determine the fourth test case or the fifth test case by judging whether the second minimum error rate of each of the test cases is less than the first minimum error rate, the second minimum error rate of the fourth test case is less than the first minimum error rate, and the second minimum error rate of the fifth test case is greater than or equal to the first minimum error rate;

[0037] Raise the priority of the fourth test case, or delete the fifth test case from the multiple test cases.

[0038] Specifically, in the steps of the method, by comparing the two minimum error rates corresponding to the same test case in the two coverage tests, namely the first minimum error rate and the second minimum error rate, it is determined whether the error rate of the latter coverage test is lower than the error rate of the previous coverage test. If so, the priority of the corresponding test case is increased; if not, the corresponding test case is deleted from multiple test cases, thereby determining whether the test case contributes to improving the coverage, increasing the priority of the test cases that contribute, and deleting the test stimuli that do not contribute, thereby accelerating the convergence speed of the iterative process of the coverage test of the hardware circuit.

[0039] In an optional implementation, after obtaining multiple coverages corresponding to multiple test cases in the second sorting, the method further includes:

[0040] Sending the multiple coverage rates corresponding to the multiple test cases in the second sorting to a monitoring port; the monitoring port is used to receive and process the multiple coverage rates;

[0041] receiving a plurality of updated test cases outputted after the monitoring port analyzes and processes the plurality of coverages, wherein the plurality of updated test cases are in a third ranking;

[0042] The multiple test cases sorted above are replaced with the multiple test cases updated above.

[0043] Specifically, in the method steps, the collected coverage is sent to the monitoring port, and after the receiving monitoring port analyzes and processes the collected coverage, multiple test cases are updated, and the updated multiple test cases are used to continue the coverage test, thereby achieving the technical effect of updating the test cases through the monitoring port, and further improving the convergence speed of the iterative process of the coverage test of the hardware circuit.

[0044] In an optional implementation, after obtaining multiple coverages corresponding to multiple test cases in the second sorting, the method further includes:

[0045] Sending the multiple coverage rates corresponding to the multiple test cases in the second sorting to a monitoring port; the monitoring port is used to receive and process the multiple coverage rates;

[0046] After receiving the above-mentioned monitoring port and analyzing and processing the above-mentioned multiple coverages, the updated K different types of expected coverages are output, where K ≥ 2 and K is a positive integer;

[0047] According to the updated expected coverage rates of the K different types, it is determined whether the K maximum coverage rates corresponding to the plurality of test cases in the second sorting satisfy the preset conditions.

[0048] Specifically, in the method steps, the collected coverage is sent to the monitoring port. After the receiving monitoring port analyzes and processes the collected coverage, multiple different types of expected coverage are updated. The updated expected coverage is used to judge the preset conditions, thereby achieving the technical effect of updating the expected coverage through the monitoring port, further improving the efficiency and accuracy of the entire coverage testing process.

[0049] In a second aspect, the present invention provides a coverage collection device, the device comprising:

[0050] An acquisition module is used to acquire M coverages obtained after multiple test cases test the hardware circuit, and N types of expected coverages, where M≥2, N≥2 and M and N are positive integers; the multiple test cases are sorted in the first order;

[0051] A classification module, used for classifying the M coverage rates into N types, each type including at least one coverage rate;

[0052] A determination module, used to determine the maximum coverage rate of each of the above types, and obtain N maximum coverage rates;

[0053] A judgment module, used for comparing the N maximum coverage rates with the corresponding expected coverage rates of the N types, and judging whether the N maximum coverage rates meet a preset condition, wherein the preset condition is that each of the above maximum coverage rates is greater than or equal to the expected coverage rate of the corresponding type;

[0054] A sorting module, for re-sorting the plurality of test cases to obtain a plurality of test cases in a second sorting order if no;

[0055] An iteration module is used to test the above-mentioned hardware circuit based on the above-mentioned multiple test cases of the second sorting to obtain multiple coverage rates corresponding to the multiple test cases of the second sorting, and iterate the coverage rate test of the above-mentioned hardware circuit until the generated multiple coverage rates reach the above-mentioned preset conditions.

[0056] The server data processing device provided by the present invention has the following advantages:

[0057] The M coverage rates obtained after testing the hardware circuit with multiple test cases are divided into N types, and the N maximum coverage rates are compared with the expected coverage rates of N different types to determine whether the N coverage rates are not less than the expected coverage rate of the corresponding type. If not, the order of multiple test cases is adjusted, the hardware circuit is retested, the corresponding coverage rate is reacquired, and the coverage rate test of the hardware circuit is iterated until the multiple coverage rates generated meet the preset conditions. Thus, the technical effect of dynamically adjusting and optimizing the test cases during the coverage rate test is achieved, so that the iterative process of the coverage rate test of the hardware circuit can converge faster. At the same time, it avoids problems such as slow project progress and high human errors caused by insufficient experience of designers, and improves the efficiency and accuracy of the entire test process.

[0058] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the coverage collection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0059] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the coverage collection method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 is a flow chart of a method for collecting coverage according to an embodiment of the present invention;

[0062] Figure 2 is a flow chart of another coverage collection method according to an embodiment of the present invention;

[0063] Figure 3 is a schematic diagram of a single collection process of a method for collecting coverage according to an embodiment of the present invention;

[0064] Figure 4 is a schematic diagram of an iterative process of a method for collecting coverage according to an embodiment of the present invention;

[0065] Figure 5 is a schematic diagram of the system architecture of a method for collecting coverage according to an embodiment of the present invention;

[0066] Figure 6 is a schematic structural diagram of a coverage collection device according to an embodiment of the present invention;

[0067] Figure 7 It is a structural schematic diagram of a computer device provided by an optional embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0069] According to an embodiment of the present invention, a coverage collection method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0070] Before introducing the method embodiment, the coverage types that may be introduced later in the embodiments of the present application are first explained. The coverage types may include, for example: code coverage, line coverage, flip coverage, conditional coverage, etc.

[0071] In this embodiment, a coverage collection method is provided, which can be used for the above-mentioned computer. Figure 1 is a flow chart of a method for collecting coverage according to an embodiment of the present invention. Figure 1 As shown, the above method includes:

[0072] Step S101, obtaining M coverages obtained after testing a hardware circuit with multiple test cases, and N different types of expected coverages.

[0073] M≥2, N≥2 and M and N are positive integers. The above multiple test cases are sorted according to the first order.

[0074] The test case may be a plurality of stimulus signals arranged in sequence. The hardware circuit may be a chip to be tested. The coverage and expected coverage may be any of the above types. After a test case tests the hardware circuit, multiple different types of coverage may be obtained.

[0075] Specifically, the coverage acquisition method can be: sending multiple test cases to multiple input ports of the hardware circuit, collecting information such as the code path, branches, and conditions output by the hardware circuit, and generating corresponding code coverage, line coverage, flip coverage, condition coverage, etc. based on the collected information.

[0076] Specifically, the method for obtaining the expected coverage may be: obtaining a target percentage of the code executed by a predefined test case. A target percentage of the control flow branches executed by the test case. A target percentage of the possible values ​​of the Boolean condition verified by the test case. A target percentage of the functions implemented in the hardware circuit executed by the test case.

[0077] Step S102, dividing the M coverage rates into N types.

[0078] Each of the above types includes at least one coverage ratio.

[0079] The N types into which the M coverages are divided may be the same as the expected coverages of the N different types.

[0080] Specifically, the division method may be: when generating corresponding code coverage, line coverage, flip coverage, condition coverage and other coverage based on the collected information, a type identifier for each coverage is generated to indicate the type of each coverage.

[0081] Step S103, determining the maximum coverage rate in each type, and obtaining N maximum coverage rates.

[0082] Specifically, the maximum coverage of each type may be obtained by comparing coverages of the same type.

[0083] Step S104: compare the N maximum coverages with the corresponding N different types of expected coverages to determine whether the N maximum coverages meet a preset condition.

[0084] The above preset condition is that each of the above maximum coverage rates is greater than or equal to the expected coverage rate of the corresponding type.

[0085] Specifically, the judgment method may be: storing the values ​​of the maximum coverage and the expected coverage, checking whether each maximum coverage satisfies its corresponding expected coverage condition, and outputting the result, indicating which maximum coverages meet the condition and which do not.

[0086] Step S105: if not, reorder the multiple test cases to obtain multiple test cases in a second order.

[0087] Specifically, the reordering method may be: store the test cases together with their maximum coverage and expected coverage. Determine whether the maximum coverage of each test case meets the expected coverage. Put aside the test cases that do not meet the expectations, and then sort the test cases that meet the conditions and the test cases that do not meet the conditions. Return the sorted test cases.

[0088] Step S106, testing the hardware circuit based on the multiple test cases sorted in the second order to obtain multiple coverage rates corresponding to the multiple test cases sorted in the second order, and iterating the coverage rate test of the hardware circuit until the generated multiple coverage rates meet a preset condition.

[0089] Specifically, the iteration method may be: iteratively testing the hardware circuit according to the reordered test cases, obtaining the actual coverage of each test case, and checking whether the current actual coverage meets the preset conditions. If the conditions are not met, repeating the test and updating the coverage.

[0090] The server data processing method provided by the present invention has the following advantages:

[0091] The M coverage rates obtained after testing the hardware circuit with multiple test cases are divided into N types, and the N maximum coverage rates are compared with the expected coverage rates of N different types to determine whether the N coverage rates are not less than the expected coverage rate of the corresponding type. If not, the order of multiple test cases is adjusted, the hardware circuit is retested, the corresponding coverage rate is reacquired, and the coverage rate test of the hardware circuit is iterated until the multiple coverage rates generated meet the preset conditions. Thus, the technical effect of dynamically adjusting and optimizing the test cases during the coverage rate test process is achieved, so that the iterative process of the coverage rate test of the hardware circuit can converge faster. At the same time, it avoids problems such as slow project progress and high human errors caused by insufficient experience of designers, and improves the efficiency and accuracy of the entire coverage rate test process.

[0092] In an optional implementation, in order to further refine how to reorder the multiple test cases, the above-mentioned first order sorting is to sort the multiple test cases from high to low according to priority. Figure 2 is a flow chart of another coverage collection method according to an embodiment of the present invention. Figure 2 As shown, the above step S105 includes:

[0093] Step S1051, generating M error rates according to the M coverage rates and the above-mentioned N different types of expected coverage rates.

[0094] Specifically, the error rate may be generated by calculating the percentage of the difference between an expected coverage rate and the coverage rate in the expected coverage rate.

[0095] Step S1052, comparing the magnitude relationships among the M error rates to obtain the priorities of the multiple test cases.

[0096] The priority may be a weight coefficient of each test case among multiple test cases. The larger the weight coefficient is, the higher the position of the corresponding test case in the sorting.

[0097] Specifically, the priority can be determined by comparing the error rates of multiple test cases, with the goal of assigning a weight coefficient to each test case based on the size of the error rate. The larger the weight coefficient, the higher the priority of the test case. First, the error rate of each test case needs to be sorted. Generally speaking, the smaller the error rate (that is, the closer the coverage is to the expected coverage), the higher the priority. In order to avoid the influence of the magnitude difference of the error rate on the weight coefficient, the error rate can be normalized. The goal of normalization is to adjust the error rate to a uniform range, such as [0,1], to facilitate subsequent comparison and weight allocation. Furthermore, the weight coefficient can be assigned by the normalized error rate.

[0098] Step S1053: reorder the plurality of test cases according to their priorities to obtain a plurality of test cases in a second order.

[0099] Specifically, the reordering may be to first calculate a weight coefficient for each test case, calculate a total weight or an average weight for each group of test cases, and order the test cases according to the combined weights.

[0100] The method provided in this embodiment determines the priority of the corresponding test case by determining the error rate between the coverage rate and the expected coverage rate, and then reorders the multiple test cases according to the priorities of the multiple test cases. This achieves dynamic adjustment of the weight coefficients of different test cases, further speeds up the convergence speed of the iterative process, and improves the efficiency of coverage testing.

[0101] In an optional implementation, in order to further refine the method for generating the error rate, the above step S1051 includes:

[0102] Based on the N different types of expected coverage, M·N feature error rates whose M coverages correspond to the N different types of expected coverage are generated.

[0103] M·N feature error rates are calculated by comparing each coverage with the expected coverage of each different type, and a total of M·N feature error rates are obtained. For example, if there are 100 coverages and 5 different types of expected coverages, a total of 500 feature error rates can be obtained.

[0104] Based on a preset rule, the M error rates of the M coverage rates are determined from the M·N feature error rates.

[0105] The preset rule may be: taking the smallest characteristic error rate among the N characteristic error rates of each coverage rate as the error rate of the coverage rate.

[0106] Specifically, the error rate can be determined by initializing an error rate list for each actual coverage rate to store the corresponding N feature error rates. According to the above steps, M·N feature error rates are calculated. For each coverage rate, the minimum value is selected from the corresponding N feature error rates as the error rate of the coverage rate.

[0107] The method provided in this embodiment generates a characteristic error rate for each type corresponding to each coverage rate, and then determines the error rate of each coverage rate based on a preset rule, thereby determining an error rate of the coverage rate from multiple error rates between the coverage rate and the expected coverage of different types, thereby achieving the technical effect of determining the error rate of each coverage rate.

[0108] In an optional implementation, in order to further refine the manner of reordering multiple test cases when the coverage does not meet the preset condition, the above step S105 includes:

[0109] The magnitude relationships between the above error rates of each test case are compared to obtain the minimum error rate of each of the above test cases.

[0110] The data output after a test case is input into the hardware circuit can be used for multiple types of coverage, that is, a test case may correspond to multiple of the M coverages. Therefore, it is necessary to compare the error rates of multiple coverages corresponding to each test case to obtain the minimum error rate corresponding to each test case.

[0111] According to the type of expected coverage corresponding to each of the above minimum error rates, the above multiple test cases are divided into N types, and each of the above types includes at least one test case.

[0112] The specific classification method may be to first define a suitable data structure to store test cases and their corresponding multiple coverages and error rates. For each test case, find out the error rates of each corresponding coverage and calculate the minimum error rate. The test cases are divided into N types according to the expected coverage type corresponding to the minimum error rate.

[0113] By judging whether each of the above minimum error rates is less than a first threshold, a first test case of the first type is determined, the minimum error rate of the above first test case is less than the above first threshold, and the above first type includes at least one test case.

[0114] Lower the priority of the test cases of the first type mentioned above.

[0115] The first threshold may be a preset value used to determine whether the error between the coverage and the expected coverage is small enough to reduce the priority of the corresponding test case to reduce the iteration cost required to improve the corresponding coverage. The first test case is a test case whose priority can be reduced. The first type is the type of the first test case.

[0116] Specifically, the determination method may be to set a preset first threshold value to determine whether the error is small enough. Traverse the minimum error rates of all test cases, and check whether the minimum error rate of each test case is less than the first threshold value. Record the test cases that meet the conditions, and these test cases will be marked as "test cases whose priority can be lowered". Determine the first type according to the expected coverage type of the test cases that meet the conditions.

[0117] Or, by judging whether each of the above-mentioned minimum error rates is greater than the second threshold and less than the third threshold, a second test case of the second type is determined, the minimum error rate of the above-mentioned second test case is greater than the above-mentioned second threshold, and the above-mentioned second type includes at least one test case.

[0118] Increase the priority of the second type of test cases mentioned above.

[0119] The second threshold value may be a preset value used to determine whether the error between the coverage and the expected coverage is large enough to increase the priority of the corresponding test case to increase the iteration cost required to increase the corresponding coverage. The second test case is the test case whose priority needs to be increased. The second type is the type of the second test case.

[0120] Specifically, the determination method may be to set a preset second threshold value to determine whether the error is large enough. Traverse the minimum error rates of all test cases and check whether the minimum error rate of each test case is greater than the second threshold value. Record the test cases that meet the conditions, and these test cases will be marked as "test cases that need to increase priority". Determine the second type according to the expected coverage type of the test cases that meet the conditions.

[0121] Or, by judging whether each of the above minimum error rates is greater than or equal to a third threshold, a third test case is determined, and the minimum error rate of the above third test case is greater than or equal to the above third threshold.

[0122] The third test case is deleted from the multiple test cases.

[0123] The third threshold may be 100%, which is used to determine whether the error between the coverage and the expected coverage is so large that the corresponding test case does not need to be used any more, that is, the test case is invalid, so as to reduce the iteration cost of the corresponding test case. The third test case is the test case that needs to be discontinued. The third type is the type of the third test case. The third test case may be recorded separately for inspection and analysis after the test is completed.

[0124] Specifically, the determination method may be to set a preset third threshold value to determine whether the error is large enough. Traverse the minimum error rates of all test cases and check whether the minimum error rate of each test case is greater than the third threshold value. Record the test cases that meet the conditions, and these test cases will be marked as "test cases that need to be discontinued". Determine the third type according to the expected coverage type of the test cases that meet the conditions.

[0125] The method provided in this embodiment determines the minimum error rate of each test case by comparing the error rate of each coverage rate, and then classifies the test cases based on the type of expected coverage rate corresponding to each minimum error rate. Then, by judging the relationship between each minimum error rate and the first, second, and third thresholds, the first, second, and third test cases are determined. According to the relationship between each minimum error rate and multiple thresholds, the priority of test cases of the same type as the first and second test cases is adjusted or the third test case is deleted from multiple test cases, thereby achieving the technical effect of dynamically adjusting the test cases based on the error rate, and improving the efficiency and accuracy of the entire testing process.

[0126] In an optional implementation, in order to further determine whether the coverage of the test case is in an optimization trend during multiple iterations of testing, after the above step S105, the following is further included:

[0127] Obtain the first minimum error rate and the second minimum error rate of each of the above test cases, where one test case corresponds to a first minimum error rate and a second minimum error rate, the above first minimum error rate is generated after each test case tests the above hardware circuit according to the first sorting, and the above second minimum error rate is generated after each test case tests the above hardware circuit according to the second sorting.

[0128] Specifically, the acquisition method may be to first define a first minimum error rate and a second minimum error rate, then determine a list of test cases to be analyzed, and collect error rate data for each test case after running the test.

[0129] The fourth test case or the fifth test case is determined by judging whether the second minimum error rate of each of the above test cases is less than the first minimum error rate, the second minimum error rate of the above fourth test case is less than the first minimum error rate, and the second minimum error rate of the above fifth test case is greater than or equal to the first minimum error rate.

[0130] Raise the priority of the fourth test case, or delete the fifth test case from the multiple test cases.

[0131] Specifically, when the multiple test cases are reordered next time, the order of the fourth test case may be advanced, or the fifth test case may be deleted from the multiple test cases before reordering.

[0132] The method provided in this embodiment determines whether the error rate of the latter coverage test is lower than the error rate of the previous coverage test by comparing the two minimum error rates corresponding to the same test case in two coverage tests, namely the first minimum error rate and the second minimum error rate. If so, the priority of the corresponding test case is increased; if not, the corresponding test case is deleted from multiple test cases, thereby determining whether the test case contributes to improving the coverage, increasing the priority of the test case that contributes, and deleting the test stimulus that does not contribute, thereby accelerating the convergence speed of the iterative process of the coverage test of the hardware circuit.

[0133] In an optional implementation, in order to further update multiple test cases through the monitoring port, after the above step S106, it also includes:

[0134] The multiple coverage rates corresponding to the multiple test cases in the second sorting are sent to a monitoring port. The monitoring port is used to receive and process the multiple coverage rates.

[0135] The monitoring port can be an output device or an input device in a computer that executes the method. The specific sending method can be sending through a bus system of the computer.

[0136] After receiving the monitoring port and analyzing the multiple coverage rates, the updated multiple test cases are output, and the updated multiple test cases are in the third sorting order.

[0137] Specifically, it can receive coverage data and parse it into a structured format. Analyze the coverage of each test case to determine which test cases have low coverage and which need to be improved or updated. Based on the analysis results, update the test cases, such as adjusting their priority, adding new test steps, or deleting test cases that are no longer necessary. Output the updated test case list, marking the reason for the update and the priority of the adjustment.

[0138] The multiple test cases sorted above are replaced with the multiple test cases updated above.

[0139] That is, continue the subsequent iteration process using the updated multiple test cases.

[0140] The method provided in this embodiment sends the collected coverage to the monitoring port, and after the receiving monitoring port analyzes and processes the collected coverage, multiple test cases are updated, and the updated multiple test cases are used to continue the coverage test, thereby achieving the technical effect of updating the test cases through the monitoring port, and further improving the convergence speed of the iterative process of the coverage test of the hardware circuit.

[0141] In an optional implementation, in order to further update multiple expected coverages through monitoring ports, after the above step S106, the following is further included:

[0142] The multiple coverage rates corresponding to the multiple test cases in the second sorting are sent to a monitoring port. The monitoring port is used to receive and process the multiple coverage rates.

[0143] The monitoring port can be an output device or an input device in a computer that executes the method. The specific sending method can be sending through a bus system of the computer.

[0144] After receiving the above-mentioned monitoring port and analyzing and processing the above-mentioned multiple coverage rates, K updated expected coverage rates of different types are output, where K≥2 and K is a positive integer.

[0145] Specifically, the data of the monitoring port may be received first, and then parsed and analyzed to determine the coverage of each test case. Determine which test cases have coverage converged too quickly and which expected coverage needs to be improved or updated. Update the expected coverage, such as adjusting the size of the expected coverage, adding new types of expected coverage, or deleting expected coverage of types that are no longer necessary. Based on the analysis results, output K different types of expected coverage.

[0146] According to the updated expected coverage rates of the K different types, it is determined whether the K maximum coverage rates corresponding to the plurality of test cases in the second sorting satisfy the preset conditions.

[0147] That is, the updated expected coverage is used to continue the subsequent iteration process.

[0148] The method provided in this embodiment sends the collected coverage to the monitoring port. After the receiving monitoring port analyzes and processes the collected coverage, it updates multiple different types of expected coverage, and uses the updated expected coverage to judge the preset conditions, thereby achieving the technical effect of updating the expected coverage through the monitoring port, further improving the efficiency and accuracy of the entire coverage testing process.

[0149] Each time the coverage is collected, multiple coverages and their corresponding test cases are stored in a historical database, and the historical database is used to record the execution status and coverage performance of each test case.

[0150] Analyze test cases based on the historical database: before generating multiple test cases arranged in the second order, analyze each test case based on the coverage performance and execution time in the historical database to generate performance indicators for each test case. The above performance indicators include but are not limited to execution time, coverage improvement rate and failure rate.

[0151] Adjust test case priority: When determining the priority of each test case, combine the performance indicators with the current coverage requirements and recalculate the priority in a weighted manner so that test cases with higher execution efficiency and coverage improvement potential are given higher priority.

[0152] Execute test cases according to priority: When entering test cases, the test cases with the highest priority are executed first until the preset coverage conditions are met.

[0153] Output coverage report: When the generated multiple coverages meet the preset conditions, a coverage report is automatically generated. The report includes the execution status of each test case, coverage changes and contribution to the overall coverage, so as to facilitate subsequent analysis and improvement.

[0154] Figure 3 FIG. 1 is a schematic diagram of a single collection process of a method for collecting coverage according to an embodiment of the present invention. Figure 3 As shown, including:

[0155] Step S301, coverage collection. This is the above step S101. In this stage, the system will collect all relevant coverage data. This data may come from code execution tracking, unit testing, integration testing or other forms of testing. The goal of this step is to obtain the execution coverage of each test case during the test process.

[0156] Step S302, feature selection. Feature selection is to select important features that affect coverage based on the currently collected coverage data. Features can be: coverage of each test case, test case running time, test case complexity, execution frequency of each module, etc. The purpose of this stage is to extract the key data that best reflects the coverage and test effect, such as the above-mentioned minimum error rate, the above-mentioned maximum coverage, etc., to provide a basis for subsequent analysis.

[0157] Step S303, determine whether it meets expectations. If yes, end the coverage test. If not, execute step S304. In this step, the current coverage data needs to be compared with the expected coverage. The expected coverage may be based on the requirements of the project (for example, the code coverage needs to reach more than 90%). If the current coverage has met the expected requirements, the test process can end.

[0158] Step S304, dynamic adjustment. When the coverage does not meet expectations, the system will dynamically adjust the test strategy. These adjustments may include: selecting additional test cases for execution; optimizing existing test cases to increase uncovered areas; re-evaluating feature selection and placing more emphasis on uncovered areas; dynamically changing the priority of test cases to ensure that high-risk areas are adequately covered.

[0159] Figure 4 is a schematic diagram of an iterative process of a method for collecting coverage according to an embodiment of the present invention, such as Figure 4 As shown, including:

[0160] Step S401, test case generation. In this stage, test cases corresponding to the hardware circuit are generated. These test cases may be based on design documents, functional requirements or past test data. The generated test cases are intended to cover a variety of possible input scenarios to fully test the functionality of the circuit.

[0161] Step S402: Input the hardware circuit. Input the generated test case into the hardware circuit to be tested. This step ensures that the hardware circuit can accept external input and start to perform related operations.

[0162] Step S403, signal monitoring. During the test case execution, the output signals of the hardware circuit are monitored. This may involve using an oscilloscope, logic analyzer or other monitoring equipment to capture the behavior of the circuit to ensure that all outputs can be accurately recorded.

[0163] Step S404: Coverage generation: Based on the monitored signals, a coverage report is generated, which includes which test cases successfully cover which circuit parts, and detailed information on uncovered parts.

[0164] Step S405, feature selection. Features that affect circuit performance may be selected from the generated coverage report. These features may be signal frequency, delay, power consumption, etc., which help determine important parameters of circuit performance.

[0165] Step S406, determine whether it meets expectations. If yes, end the coverage test. If no, execute step S407. It is necessary to compare the current coverage with the expected coverage to determine whether it meets the preset conditions.

[0166] Step S407, calibration: According to the result of the previous step, the weight of the test case is calibrated.

[0167] Step S408: Priority adjustment: Based on the calibration of the weights of the test cases, the priority of each test case during reordering is modified, and then the above step S401 is repeatedly performed to generate a plurality of newly ordered test cases.

[0168] Figure 5 is a schematic diagram of the system architecture of the method for collecting coverage according to an embodiment of the present invention. Figure 5 As shown, the system includes: a training set, a calibration module, a stimulus module, a training module, a storage module, and peripherals. The above-mentioned coverage collection method can be executed on this system architecture.

[0169] The training set includes multiple types of expected coverage, which are input into the calibration module. The calibration module receives multiple coverages generated by each iteration stored in the storage module, compares the expected coverage with the multiple coverages, and then calibrates the priorities of multiple test cases, and sends the calibration results to the stimulus module. The stimulus module is used to generate multiple sequentially arranged test cases required for each iteration coverage test, and specifically, it can reorder the multiple test cases according to the priorities of the multiple test cases sent by the calibration module, and send the multiple sequentially arranged test cases to the training module. The training module uses multiple test cases to test the hardware circuit, and generates corresponding multiple coverages, and sends the generated multiple coverages to the storage module for storage. The storage module can also connect peripherals, that is, external devices, and designers can filter and classify the relevant data recorded in the storage module according to specific needs. In this way, the parameters that designers are concerned about can be recorded in a targeted manner, thereby significantly reducing the time for designers to screen and identify the recorded results during the design scheme adjustment stage, which helps to shorten the overall design cycle and process.

[0170] In another embodiment, the present invention further provides a coverage collection device. Figure 6 is a schematic diagram of the structure of a coverage collection device according to an embodiment of the present invention, the device comprising:

[0171] The acquisition module 601 is used to acquire M coverages obtained after multiple test cases test the hardware circuit, and N types of expected coverages, where M≥2, N≥2 and M and N are positive integers. The multiple test cases are sorted in the first order.

[0172] The classification module 602 is used to classify the M coverage rates into N types, each of which includes at least one coverage rate.

[0173] The determination module 603 is used to determine the maximum coverage rate in each of the above types to obtain N maximum coverage rates.

[0174] The judgment module 604 is used to compare the above-mentioned N maximum coverage rates with the corresponding expected coverage rates of the above-mentioned N types to determine whether the above-mentioned N maximum coverage rates meet a preset condition, and the above-mentioned preset condition is that each of the above-mentioned maximum coverage rates is greater than or equal to the expected coverage rate of the corresponding type.

[0175] The sorting module 605 is used to re-sort the multiple test cases if no, to obtain multiple test cases sorted in a second order.

[0176] The iteration module 606 is used to test the above-mentioned hardware circuit based on the multiple test cases sorted in the second order to obtain multiple coverage rates corresponding to the multiple test cases sorted in the second order, and iterate the coverage rate test of the above-mentioned hardware circuit until the generated multiple coverage rates reach the above-mentioned preset conditions.

[0177] The server data processing device provided by the present invention has the following advantages:

[0178] The M coverage rates obtained after testing the hardware circuit with multiple test cases are divided into N types, and the N maximum coverage rates are compared with the expected coverage rates of N different types to determine whether the N coverage rates are not less than the expected coverage rate of the corresponding type. If not, the order of multiple test cases is adjusted, the hardware circuit is retested, the corresponding coverage rate is reacquired, and the coverage rate test of the hardware circuit is iterated until the multiple coverage rates generated meet the preset conditions. Thus, the technical effect of dynamically adjusting and optimizing the test cases during the coverage rate test is achieved, so that the iterative process of the coverage rate test of the hardware circuit can converge faster. At the same time, it avoids problems such as slow project progress and high human errors caused by insufficient experience of designers, and improves the efficiency and accuracy of the entire test process.

[0179] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0180] The coverage collection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0181] The embodiment of the present invention also provides a computer device having the above Figure 6 The coverage of the collection device is shown.

[0182] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0183] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include an integrated circuit. The integrated circuit may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0184] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0185] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0187] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0188] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0189] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0190] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for collecting coverage, characterized in that: The method comprises: Obtaining M coverages obtained after testing a hardware circuit with multiple test cases, and N expected coverages of different types, where M≥2, N≥2 and M and N are positive integers; the multiple test cases are sorted in the first order; Dividing the M coverages into N types, each type including at least one coverage; Determine the maximum coverage rate in each of the types to obtain N maximum coverage rates; Compare the N maximum coverage rates with the corresponding N different types of expected coverage rates to determine whether the N maximum coverage rates meet a preset condition, where the preset condition is that each of the maximum coverage rates is greater than or equal to the expected coverage rate of the corresponding type; If not, reorder the multiple test cases to obtain multiple test cases in a second order; The hardware circuit is tested based on the multiple test cases sorted in the second order to obtain multiple coverage rates corresponding to the multiple test cases sorted in the second order, and the coverage rate test of the hardware circuit is iterated until the generated multiple coverage rates meet the preset conditions.

2. The method according to claim 1, characterized in that The first order sorting is to sort the multiple test cases from high to low priority; The step of reordering the plurality of test cases to obtain a plurality of test cases ordered in a second order includes: Generate M error rates according to the M coverage rates and the N different types of expected coverage rates; Comparing the magnitude relationships among the M error rates to obtain priorities of the multiple test cases; The plurality of test cases are reordered according to their priorities to obtain a plurality of test cases in the second order.

3. The method according to claim 2, characterized in that Generate M error rates according to the M coverage rates and the expected coverage rates of the N types, including: Based on the N different types of expected coverages, respectively, M coverages are generated, and M·N feature error rates correspond to the N different types of expected coverages; Based on a preset rule, M error rates of the M coverage rates are determined from the M·N feature error rates.

4. The method according to claim 2 or 3, characterized in that: Reordering the multiple test cases to obtain multiple test cases in a second order, including: Comparing the magnitude relationship between the error rates of each test case to obtain the minimum error rate of each test case; According to the type of expected coverage corresponding to each of the minimum error rates, the multiple test cases are divided into N types, each of the types including at least one test case; Determine a first test case of a first type by judging whether each of the minimum error rates is less than a first threshold, wherein the minimum error rate of the first test case is less than the first threshold, and the first type includes at least one test case; Lowering the priority of the test cases of the first type; or, determining a second test case of a second type by judging whether each of the minimum error rates is greater than a second threshold and less than a third threshold, wherein the minimum error rate of the second test case is greater than the second threshold, and the second type includes at least one test case; Raising the priority of the test cases of the second type; Or, determining a third test case by judging whether each of the minimum error rates is greater than or equal to the third threshold, wherein the minimum error rate of the third test case is greater than or equal to the third threshold; The third test case is deleted from the plurality of test cases.

5. The method according to claim 4, characterized in that After obtaining the second sorted multiple test cases, it also includes: Obtain a first minimum error rate and a second minimum error rate for each of the test cases, where one test case corresponds to one first minimum error rate and one second minimum error rate, the first minimum error rate being generated after each test case tests the hardware circuit according to the first sorting, and the second minimum error rate being generated after each test case tests the hardware circuit according to the second sorting; Determine a fourth test case or a fifth test case by judging whether the second minimum error rate of each of the test cases is less than the first minimum error rate, the second minimum error rate of the fourth test case is less than the first minimum error rate, and the second minimum error rate of the fifth test case is greater than or equal to the first minimum error rate; Raise the priority of the fourth test case, or delete the fifth test case from the multiple test cases.

6. The method according to any one of claims 1 to 3, characterized in that: After obtaining the multiple coverages corresponding to the multiple test cases in the second sorting, the method further includes: Sending the multiple coverage rates corresponding to the multiple test cases in the second sorting to a monitoring port; the monitoring port is used to receive and process the multiple coverage rates; receiving a plurality of updated test cases outputted after the monitoring port analyzes and processes the plurality of coverage rates, wherein the plurality of updated test cases are in a third order; The second sorted plurality of test cases are replaced with the updated plurality of test cases.

7. The method according to any one of claims 1 to 3, characterized in that: After obtaining the multiple coverages corresponding to the multiple test cases in the second sorting, the method further includes: Sending the multiple coverage rates corresponding to the multiple test cases in the second sorting to a monitoring port; the monitoring port is used to receive and process the multiple coverage rates; receiving K different types of expected coverages updated after the monitoring port analyzes and processes the multiple coverages, where K≥2 and K is a positive integer; According to the updated expected coverage rates of the K different types, it is determined whether the K maximum coverage rates corresponding to the plurality of test cases in the second sorting meet the preset condition.

8. A coverage collection device, characterized in that: The device comprises: An acquisition module, used to acquire M coverages obtained after a plurality of test cases test a hardware circuit, and N types of expected coverages, where M≥2, N≥2 and M and N are positive integers; the plurality of test cases are in the first order; A classification module, used for classifying the M coverage rates into N types, each type including at least one coverage rate; A determination module, used to determine the maximum coverage rate in each of the types to obtain N maximum coverage rates; A judgment module, used for comparing the N maximum coverage rates with the corresponding expected coverage rates of the N types, and judging whether the N maximum coverage rates meet a preset condition, wherein the preset condition is that each of the maximum coverage rates is greater than or equal to the expected coverage rate of the corresponding type; a sorting module, configured to, if no, re-sort the plurality of test cases to obtain a plurality of test cases in a second sorting order; An iteration module is used to test the hardware circuit based on the multiple test cases in the second sorting to obtain multiple coverage rates corresponding to the multiple test cases in the second sorting, and iterate the coverage rate test of the hardware circuit until the generated multiple coverage rates meet the preset conditions.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the coverage collection method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the coverage collection method according to any one of claims 1 to 7.

Images