Circuit testing method, apparatus and electronic device

Abstract

The present disclosure provides a circuit test method, device and electronic equipment, the method comprising: using M candidate test cases to perform tests on multiple functions to be tested of a circuit to be tested, obtaining M first test results; clustering the M first test results to obtain N sets of candidate test case sets; determining the ranking priority of each candidate test case in the N sets of candidate test case sets; determining N first target test cases from the N sets of candidate test case sets according to the ranking priority, performing tests on the multiple functions to be tested, and obtaining second test results; in the case where the second test results do not reach a preset functional coverage value, iteratively performing the operation of determining N second target test cases from the N sets of candidate test case sets according to the ranking priority until the second test results reach the preset functional coverage value.

Description

Technical Field

[0001] This disclosure relates to the field of circuit testing technology, and more specifically to a circuit testing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Functional verification is a crucial step in the digital circuit design process. Through functional verification, we can test whether the circuit meets the functional requirements of the design specification, thereby ensuring that the circuit can perform its intended function. For example, we can set up a test environment, write test cases, simulate the working state of the circuit in various scenarios, and determine whether the circuit output is consistent with the functional requirements in the design specification.

[0003] However, in order to ensure that the testing process can cover all functional points, a large number of test cases often need to be written in a single regression test, and each test case needs to be executed multiple times with the initial value changed, resulting in long testing time and low testing efficiency. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a circuit testing method, apparatus and electronic device.

[0005] According to a first aspect of this disclosure, a circuit testing method is provided, comprising: executing tests on multiple functions of the circuit under test using M candidate test cases to obtain M first test results, where M is an integer greater than 1; clustering the M first test results to obtain N sets of candidate test cases, where N is an integer greater than or equal to 1; determining the ranking priority of each candidate test case in the N sets of candidate test cases according to a preset evaluation index; determining N first target test cases from the N sets of candidate test cases according to the ranking priority, and executing tests on multiple functions under test using the N first target test cases to obtain second test results; if it is determined that the second test results do not reach a preset value of functional coverage, iteratively executing the operation of determining N second target test cases from the N sets of candidate test cases according to the ranking priority until the second test results reach the preset value of functional coverage, wherein the ranking priority of the second target test cases is lower than the ranking priority of the first target test cases.

[0006] The second aspect of this disclosure provides a circuit testing apparatus, comprising: a first testing module, configured to execute tests on multiple functions of the circuit under test using M candidate test cases, respectively, to obtain M first test results, where M is an integer greater than 1; a clustering module, configured to cluster the M first test results to obtain N sets of candidate test cases; a first determining module, configured to determine the ranking priority of each candidate test case in the N sets of candidate test cases according to a preset evaluation index; a second determining module, configured to determine N first target test cases from the N sets of candidate test cases according to the ranking priority, and execute tests on multiple functions under test using the N first target test cases to obtain second test results; and an iteration module, configured to iteratively execute the operation of determining N second target test cases from the N sets of candidate test cases according to the ranking priority when the second test results are determined not to reach a preset value of functional coverage, until the second test results reach the preset value of functional coverage, wherein the ranking priority of the second target test cases is lower than that of the first target test cases.

[0007] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0008] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0009] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0010] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0011] Figure 1 The diagram illustrates application scenarios of the circuit testing method, apparatus, and electronic device according to embodiments of the present disclosure.

[0012] Figure 2 A flowchart illustrating a circuit testing method according to an embodiment of the present disclosure is shown schematically.

[0013] Figure 3 A schematic diagram of a circuit testing method according to an embodiment of the present disclosure is shown.

[0014] Figure 4A flowchart illustrating a circuit testing method according to another embodiment of the present disclosure is shown schematically;

[0015] Figure 5 A flowchart illustrating the iterative process of a clustering algorithm according to embodiments of the present disclosure is shown schematically.

[0016] Figure 6 A schematic block diagram of a circuit testing apparatus according to an embodiment of the present disclosure is shown.

[0017] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a circuit testing method according to an embodiment of the present disclosure. Detailed Implementation

[0018] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0020] All terms used herein (including technical and scientific terms) shall have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein shall be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid manner. In the use of expressions such as “at least one of A, B, and C,” the expression should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, and C” should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0021] In functional verification, a set of related functional points is typically used as a coverage group. During functional verification, the coverage of functional points in the coverage group can be checked to determine whether the corresponding functions have been tested, thereby effectively optimizing the design of test cases. For example, functional coverage can be used to represent the coverage of functional points in the coverage group, and can be defined by the ratio of executed functional points to all functional points. The test case set used for circuit testing is essentially a constrained set of random test cases. Each test case provides only one set of constraints, and the test vectors are generated by random numbers. It is unknown which code / functions each test case corresponds to, and it is impossible to directly filter test cases corresponding to specific functional points based on the code text. Coverage can only be improved through a large number of random tests.

[0022] In realizing the concept of this invention, the inventors discovered at least the following problems in the related technology: In actual functional verification, due to the large state space (e.g., the verification process needs to cover normal functional scenarios and various abnormal scenarios such as input signal errors and timing deviations, or the functional implementation of the circuit under test usually depends on multiple independent configuration parameters), in order to improve efficiency and save time, verification engineers usually do not write targeted test cases for each functional point in the coverage group, but instead conduct random testing with a large number of test cases. For example, to ensure that the testing process can cover all functional points, a regression test will contain a large number of test cases, and each test case will be executed multiple times with the initial value changed. The above process of repeatedly executing and changing the initial value of a large number of test cases consumes a lot of time and reduces testing efficiency.

[0023] In addition, during the process of realizing the concept of this invention, the inventors found that there are at least the following problems in the related technology: a small number of typical test cases can cover most of the functional points, while the remaining test cases are highly repetitive. Executing these test cases makes it difficult to cover more functional points, but it takes a lot of extra time to execute them, resulting in a long verification cycle and slow project progress.

[0024] In view of the above, embodiments of this disclosure provide a circuit testing method, comprising: using M candidate test cases to perform tests on multiple functions of the circuit under test, obtaining M first test results, where M is an integer greater than 1; clustering the M first test results to obtain N sets of candidate test cases, where N is an integer greater than or equal to 1; determining the ranking priority of each candidate test case in the N sets of candidate test cases according to a preset evaluation index; determining N first target test cases from the N sets of candidate test cases according to the ranking priority, and using the N first target test cases to perform tests on multiple functions under test, obtaining second test results; if it is determined that the second test results do not reach a preset value of functional coverage, iteratively executing the operation of determining N second target test cases from the N sets of candidate test cases according to the ranking priority, until the second test results reach the preset value of functional coverage, wherein the ranking priority of the second target test cases is lower than the ranking priority of the first target test cases.

[0025] Figure 1 The diagram illustrates application scenarios of the circuit testing method, apparatus, and electronic device according to embodiments of the present disclosure.

[0026] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0027] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0028] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0029] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0030] For example, a user can initiate a circuit test command through a first terminal device 101, a second terminal device 102, and a third terminal device 103. In response to the circuit test command, the server 105 can execute the circuit test method of this embodiment. The circuit test method includes: using M candidate test cases to perform tests on multiple functions of the circuit under test, obtaining M first test results, where M is an integer greater than 1; clustering the M first test results to obtain N sets of candidate test cases, where N is an integer greater than or equal to 1; determining the ranking priority of each candidate test case in the N sets of candidate test cases according to a preset evaluation index; determining N first target test cases from the N sets of candidate test cases according to the ranking priority, and using the N first target test cases to perform tests on multiple functions under test, obtaining second test results; if the second test results do not reach the preset value of functional coverage, iteratively executing the operation of determining N second target test cases from the N sets of candidate test cases according to the ranking priority until the second test results reach the preset value of functional coverage, wherein the ranking priority of the second target test cases is less than the ranking priority of the first target test cases.

[0031] It should be noted that the circuit testing method provided in this embodiment can generally be executed by server 105. Correspondingly, the circuit testing device provided in this embodiment can generally be located in server 105. The circuit testing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the circuit testing device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0033] The following will be based on Figure 1The described scene, through Figures 2-5 The circuit testing method of the present disclosure will be described in detail.

[0034] Figure 2 A flowchart illustrating a circuit testing method according to an embodiment of the present disclosure is shown schematically.

[0035] like Figure 2 As shown, the circuit testing method of this embodiment includes operations S210 to S250.

[0036] In operation S210, multiple test functions of the circuit under test are executed using M candidate test cases, resulting in M ​​first test results, where M is an integer greater than 1.

[0037] For example, a verification engineer might pre-write M candidate test cases. The number of candidate test cases is typically large; if all M candidate test cases are used to perform circuit testing each time, it will result in long testing times and low testing efficiency.

[0038] Furthermore, verification engineers typically do not write candidate test cases specifically for each function to be tested. A small number of typical candidate test cases out of M candidate test cases can cover most of the functions to be tested, while the remaining candidate test cases are highly repetitive. Executing the remaining candidate test cases can hardly cover more functional points, but it takes a lot of extra time to execute them, resulting in a long verification cycle and slow progress of the test project.

[0039] For example, M candidate test cases can be used to execute tests on multiple functions of the circuit under test, resulting in M ​​first test results. These first test results can then be used to select a small number of target test cases that can test most of the functions under test. This allows for coverage of more functions with fewer test cases, reducing the number of executions of redundant test cases. The first test results can include test execution data collected after one execution of a single candidate test case, and can be used to generate a coverage report.

[0040] In operation S220, clustering is performed using M first test results to obtain N sets of candidate test cases, where N is an integer greater than or equal to 1.

[0041] For example, a clustering algorithm can be used to cluster the feature vectors corresponding to the M candidate test cases based on the M first test results. Each cluster corresponds to a set of feature vectors. The feature vectors in each cluster cover similar functions to be tested, and they collectively contribute significantly to one or more specific coverage groups, i.e., they have similar coverage features. The overlap of functions to be tested covered by feature vectors in different clusters is low.

[0042] Figure 3 A schematic diagram of a circuit testing method according to an embodiment of the present disclosure is shown.

[0043] like Figure 3 As shown, unclassified test cases can include M candidate test cases, which is a large number. M first test results can be obtained to collect and analyze coverage. Figure 3 It can be seen that different candidate test cases may cover different functions to be tested. For example, test case 1, test case 2, test case 3 and test case x cover different functions to be tested. Figure 3 Hollow squares represent uncovered functions to be tested, while solid squares represent functions that can be covered. M candidate test cases can be classified using a clustering algorithm, and the clustering effect can be iteratively optimized to obtain N sets of candidate test cases. These N sets can include cluster 1, cluster 2, and cluster 3.

[0044] For example, the clustering algorithm can be set according to actual needs, such as including but not limited to at least one of the following: K-means clustering algorithm (K-means), density-based spatial clustering of applications with noise (DBSCAN), Gaussian mixture model (GMM), etc.

[0045] In operation S230, the ranking priority of each candidate test case in the N sets of candidate test cases is determined according to the preset evaluation indicators.

[0046] For example, a pre-defined evaluation metric can characterize the contribution of functional coverage. The higher the contribution of functional coverage, the higher the ranking priority of the candidate test cases.

[0047] For example, a set of candidate test cases may include multiple candidate test cases. For any set of candidate test cases, the indicator value of each candidate test case for a preset evaluation metric can be determined to determine the functional coverage contribution of each candidate test case. The candidate test case with the highest functional coverage contribution can be assigned the highest ranking priority, and the candidate test case with the lowest functional coverage contribution can be assigned the lowest ranking priority, and so on.

[0048] In operation S240, N first target test cases are determined from the N sets of candidate test cases according to the sorting priority, and the N first target test cases are used to perform tests on multiple functions to be tested to obtain the second test results.

[0049] For example, for any set of candidate test cases, candidate test cases that meet the preset sorting priority can be selected as typical test cases (i.e., the first target test cases) that can represent the set of candidate test cases. In subsequent testing, the typical test cases can be used to replace the set of candidate test cases to perform circuit testing. Figure 3 As shown, typical test case 1 can be selected from cluster 1, typical test case 2 can be selected from cluster 2, and typical test case 3 can be selected from cluster 3.

[0050] For example, selecting only one candidate test case from each set of candidate test cases to obtain N first target test cases may be insufficient to cover most of the functions to be tested. Figure 3 As shown, on the verification platform, N first target test cases can be used to perform tests on multiple functions to be tested, and a second test result can be obtained. The second test result can characterize the overall functional coverage of each first target test case for multiple functions to be tested. For example, the second test result can include the overall functional coverage for the currently executed test case set, which can be used to compare with the functional coverage preset value, so as to determine whether to continue to supplement and execute subsequent test cases based on the comparison result.

[0051] Based on the results of the second test, the overall functional coverage of the N first target test cases for multiple functions under test can be calculated. For example... Figure 3 As shown, by using coverage statistics, we can determine the overall functional coverage of multiple functions to be tested by typical test case 1, typical test case 2, and typical test case x.

[0052] In operation S250, if it is determined that the second test result does not reach the preset value of functional coverage, the operation of determining N second target test cases from the N sets of candidate test cases according to the sorting priority is executed iteratively until the second test result reaches the preset value of functional coverage. The sorting priority of the second target test cases is lower than that of the first target test cases.

[0053] For example, the preset functional coverage value is 90%, but the overall functional coverage of the N first target test cases for multiple functions under test is only 70%, indicating that the second test results did not reach the preset functional coverage value. The preset functional coverage value can be set according to actual needs and is not limited here.

[0054] For example, if the second test result fails to reach the preset functional coverage value, it is necessary to continue selecting typical test cases from each set of candidate test cases to obtain second target test cases. This allows more typical test cases to cover more functions under test, thus achieving the preset functional coverage value. For instance, for each set of candidate test cases, the candidate test cases with the highest ranking priority (i.e., first ranking priority) can be determined, resulting in N first target test cases. If the second test result fails to reach the preset functional coverage value, for each set of candidate test cases, the candidate test cases with the second ranking priority can be determined, resulting in N second target test cases. Tests on multiple functions under test are then performed on the N second target test cases and the N first target test cases to determine whether the test results corresponding to the N second target test cases and the N first target test cases reach the preset functional coverage value. If the preset functional coverage value is reached, when testing multiple functions under test of the circuit under test in subsequent tests, only the N first target test cases and the N second target test cases corresponding to the first iteration operation need to be used, without needing to use the M candidate test cases.

[0055] According to embodiments of this disclosure, clustering M candidate test cases based on M first test results allows for the accurate identification and classification of the coverage of the function to be tested for each candidate test case using an unsupervised clustering algorithm. This comprehensive analysis of covered and uncovered functionalities enables more targeted development of test cases for uncovered functionalities, ensuring the sufficiency of verification. Unsupervised learning, as a machine learning method that requires no external labels, analyzes and clusters unlabeled datasets using algorithms to identify hidden patterns and rules within the data without human intervention. In functional verification, clustering algorithms can effectively classify test cases, thereby identifying the coverage pattern and contribution of each test case to functional coverage, providing a basis for resource allocation of test cases.

[0056] According to embodiments of this disclosure, N first target test cases are determined from N sets of candidate test cases based on sorting priority, and the N first target test cases are used to perform tests on multiple functions to be tested to obtain a second test result. If the second test result does not reach the preset value of functional coverage, the operation of determining N second target test cases from N sets of candidate test cases based on sorting priority is iteratively executed until the second test result reaches the preset value of functional coverage. The most typical test cases can be selected from each set of candidate test cases and assigned higher execution weights, thereby reducing the number of executions of redundant test cases. During testing, the expected functions to be tested can be quickly covered in a short time, allowing more focus on developing test cases for uncovered functional points, making the entire verification process more targeted and efficient.

[0057] According to embodiments of this disclosure, clustering using M first test results may include: generating M coverage reports for the M first test results, wherein each coverage report includes P coverage groups and P functional coverage rates corresponding to the P coverage groups, wherein each coverage group includes one type of function to be tested from a plurality of functions to be tested, and P is a positive integer; for each coverage report, using the P functional coverage rates as P features, constructing a feature vector for the P features to obtain M feature vectors; preprocessing the M feature vectors; and clustering the preprocessed M feature vectors using the target number of clusters.

[0058] For example, M coverage reports can be generated for M first test results using automated scripts. A category of functions to be tested may include a set of similar functions to be tested. The functions to be tested contained in each coverage group are similar. For example, a function to be tested can be broken down into multiple function points, and a coverage group can be a set of related function points among multiple functions to be tested, wherein the set of related function points may include a set of function points that revolve around the same function to be tested and belong to the same functional dimension.

[0059] For example, when verifying the receiving circuit of an image sensor, a certain coverage group represents whether the circuit can correctly resolve different pixel formats. Each function point to be tested within the coverage group represents a pixel format to be resolved, such as RAW8 or RAW10.

[0060] Functional coverage can be defined as the ratio of the number of functions executed by candidate test cases to the total number of functions in that class to be tested, for a given category of functions to be tested. Coverage can be expressed as a percentage, with values ​​ranging from 0 to 100, or in other forms, such as a ratio, with values ​​ranging from 0 to 1. The specific form of coverage is not limited here.

[0061] For example, coverage group 1 represents the number of circuit inputs, with coverage points being input 1 and input 2; coverage group 2 represents the number of circuit outputs, with coverage point being output 1. If input 1 and output 1 are used during execution (i.e., the function executed by the candidate test case includes input 1 and output 1), then the coverage rate of coverage group 1 is 50%, and the coverage rate of coverage group 2 is 100%. Therefore, the feature vector corresponding to this test case can be represented as (50, 100).

[0062] According to embodiments of this disclosure, by merging the functions to be tested into coverage groups, using P function coverage rates as P features, and constructing a feature vector for the P features, the feature dimensionality can be reduced, thus avoiding slow clustering speed.

[0063] According to embodiments of this disclosure, the circuit testing method further includes: performing a first clustering on the preprocessed M feature vectors using a preset initial clustering number to generate a first clustering result, wherein the first clustering result includes at least one cluster, and the number of clusters is equal to the initial clustering number; generating a next clustering number for the next clustering based on the index value of the clustering result relative to a preset clustering quality evaluation index, and performing a next clustering on the preprocessed M feature vectors using the next clustering number to generate a next clustering result; iteratively executing the next clustering operation based on the index value of the clustering result relative to the preset clustering quality evaluation index, generating a next clustering number for the next clustering, and performing the next clustering operation until a preset iteration condition is met, and outputting the target clustering number.

[0064] For example, the initial number of clusters can be preset, perhaps based on experience. For instance, if the initial number of clusters is N0, the first clustering can be performed based on N0, yielding the first clustering result. The index value V1 of the first clustering result is automatically calculated, representing the clustering quality. A new number of clusters N1 can be dynamically generated based on V1, and a second clustering can be performed using this new number (the next number of clusters), yielding the second clustering result. The index value V2 of the second clustering result is then automatically calculated, and a new number of clusters N2 is dynamically generated based on V2, and a third clustering can be performed using this new number… This process is iterated until a preset iteration condition is met, such as reaching a preset maximum number of iterations.

[0065] For example, clustering quality assessment metrics can be such as silhouette coefficient, elbow rule, etc. The silhouette coefficient will be used as an example to illustrate this. The formula for calculating the silhouette coefficient is shown in equation (1) below.

[0066] (1)

[0067] Where S(i) represents the silhouette coefficient of the i-th sample, max{a(i),b(i)} represents taking the maximum value between a(i) and b(i), and a(i) is the intra-class cohesion, which represents the degree of cohesion for the i-th sample. Data points Calculate the average distance to all other data points within the same class. b(i) represents the inter-class separation, which indicates the selection... An external class, for calculation The average distance to all data points within the same class is calculated by iterating through all data points in other classes and taking the smallest average distance among all the average distances.

[0068] Assuming there are n data points in a cluster, the formula for calculating a(i) is shown in equation (2) below.

[0069] (2)

[0070] Assuming there are m data points outside the cluster, the formula for calculating b(i) is shown in equation (3).

[0071] (3)

[0072] In equations (2) and (3), D(x) i ,x j ) represents data point x i and data point x j Distance metric between them.

[0073] For example, the silhouette coefficients of all data points can be calculated using the above formula, and the average value can be obtained, which is the overall silhouette coefficient obtained by training with the current number of clusters K. The value of the overall silhouette coefficient is between [-1, 1]. The larger the value, the more compact the clusters are and the more dispersed the clusters are, resulting in better clustering results. Therefore, the K value corresponding to the largest overall silhouette coefficient should be selected as the optimal number of clusters K.

[0074] For example, when selecting the optimal number of clusters K, we can choose any integer between 2 and 10 for K, and use each K value to train the dataset using the K-means algorithm. We start with an initial overall silhouette coefficient of -1 and an initial K value of 0. After each training round, we calculate the current overall silhouette coefficient and compare it with the overall silhouette coefficient saved from the previous training round. We take the maximum of the two as the new overall silhouette coefficient and update the corresponding K value. After all iterations are complete, we return the final K value as the optimal number of clusters K.

[0075] According to the embodiments of this disclosure, the script automatically calculates the index value based on the clustering results, finds a new number of clusters N1 based on the index value, and then uses the new number of clusters to perform clustering again and calculate the index value. After multiple rounds of iteration, the optimal number of clusters is determined. Thus, without providing any additional parameters, the script can be executed directly from the first test result to automatically complete the entire process of generating the coverage report, extracting feature vectors, and generating the final clustering results.

[0076] According to embodiments of this disclosure, related methods typically involve manually setting the number of clusters based on experience. Since the number of clusters set based on experience may not be accurate enough, it can lead to suboptimal clustering results and a lack of objective evaluation criteria. However, by automatically finding the optimal number of clusters during the clustering process based on clustering quality evaluation indicators, a more accurate determination of the number of clusters can be achieved compared to methods that manually preset parameters based on experience. Furthermore, by embedding the clustering quality evaluation indicators into an iterative closed loop, the number of clusters can be automatically optimized, and the final optimal number of clusters can be output. This realizes an autonomous optimization mechanism of evaluation, updating, and re-clustering, eliminating not only the manual parameter input step but also constructing a fully automated processing flow from raw test data to clustering results, significantly improving the clustering quality and efficiency in circuit testing scenarios.

[0077] According to an embodiment of this disclosure, performing an operation to determine N second target test cases from N sets of candidate test cases based on sorting priority includes: if the test results corresponding to the N first target test cases and the N second target test cases determined in the Kth iteration operation do not reach the preset value of functional coverage, for each set of candidate test cases, selecting second target test cases whose sorting priority meets the preset value condition from the candidate test case sets, to obtain the N second target test cases determined in the K+1th operation, where K is a positive integer; determining whether the test results corresponding to the N first target test cases, the N second target test cases determined in the Kth iteration operation, and the N second target test cases determined in the K+1th operation reach the preset value of functional coverage; wherein, the sorting priority of the N second target test cases determined in the K+1th operation is less than that of the N second target test cases determined in the Kth iteration operation.

[0078] For example, the overall functional coverage of multiple functions to be tested can be determined by N first target test cases and N second target test cases determined by the Kth iteration operation. When the overall functional coverage reaches the preset value of functional coverage, that is, when the test results corresponding to the N first target test cases and the N second target test cases determined by the Kth iteration operation reach the preset value of functional coverage, the N first target test cases and the N second target test cases determined by the Kth iteration operation can be directly used as the final typical test cases. When performing tests on multiple functions to be tested of the circuit under test in the subsequent process, only the N first target test cases and the N second target test cases determined by the Kth iteration operation can be used for testing.

[0079] For example, if the overall functional coverage of multiple functions to be tested by the N first target test cases and the N second target test cases determined in the Kth iteration operation does not reach the preset value of functional coverage, that is, if the test results corresponding to the N first target test cases and the N second target test cases determined in the Kth iteration operation do not reach the preset value of functional coverage, the K+1th operation can be executed. For example, the sorting priority of the N second target test cases determined in the K+1th operation can be second only to the N second target test cases determined in the Kth iteration operation. For example, the N second target test cases determined in the Kth iteration operation correspond to the second sorting priority, and the N second target test cases determined in the K+1th operation correspond to the third sorting priority.

[0080] For example, it is possible to determine the overall functional coverage of multiple functions to be tested by the N first target test cases, the N second target test cases determined by the Kth iteration operation, and the N second target test cases determined by the K+1th operation, and to determine whether the overall functional coverage meets the functional coverage requirements. That is, it is possible to determine whether the test results corresponding to the N first target test cases, the N second target test cases determined by the Kth iteration operation, and the N second target test cases determined by the K+1th operation reach the preset value of functional coverage. When it is determined that the preset value of functional coverage has been reached, the N first target test cases, the N second target test cases determined by the Kth iteration operation, and the N second target test cases determined by the K+1th operation can be used as the final typical test cases.

[0081] According to embodiments of this disclosure, in the field of large-scale integrated circuits, the test case set is large in scale and the number of functional points is numerous. Therefore, it is difficult to write targeted tests for each functional point, and test vectors can only be randomly generated under certain constraints. As a set of specific constraints, even if the same test case is used, it may not cover the same functional point due to different random values. However, through iterative operations, a small number of typical test cases can be selected from M candidate test cases. Subsequent tests only need to use the typical test cases. The other candidate test cases in the M candidate test cases, except for the typical test cases, are redundant test cases and can be excluded in all subsequent regression tests. Thus, more functional points can be covered with fewer test cases, thereby reflecting the coverage of core functions with a small number of tests. This avoids the time wasted by executing redundant tests, ensures that regression testing can be completed in the shortest possible time after each code update, shortens the verification time, and accelerates the verification process.

[0082] According to embodiments of this disclosure, the circuit testing method may further include: upon determining that the second test result reaches a preset value for functional coverage, performing subsequent tests on multiple functions to be tested based on N first target test cases.

[0083] For example, executing tests on multiple functions under test using N first target test cases yields N second test results. If the second test results meet the preset functional coverage value, it indicates that the coverage of the multiple functions under test by the N first target test cases has met the preset requirements. Therefore, there is no need to select more candidate test cases for testing. Thus, subsequent tests on the multiple functions under test can be performed based solely on the N first target test cases. This reduces the number of test cases used and saves testing time.

[0084] According to embodiments of this disclosure, determining N first target test cases from N sets of candidate test cases based on ranking priority includes: for each set of candidate test cases, selecting the candidate test case with the highest ranking priority as the first target test case.

[0085] For example, for a set of candidate test cases, the candidate test case with the highest ranking priority has the highest contribution to functional coverage. It performs best in terms of functional coverage and test execution time, so it can be used as the first target test case.

[0086] According to embodiments of this disclosure, preprocessing the M feature vectors includes at least one of the following: deduplication, removal of invalid test feature vectors, wherein invalid test feature vectors include at least one of the following: format anomaly test feature vectors, functional coverage anomaly test feature vectors; and clustering the preprocessed test feature vectors.

[0087] For example, the M candidate test cases may include duplicate test cases or abnormal test cases, but the number of pre-built candidate test cases is usually large and difficult to screen manually. Therefore, the screening can be done by script during the construction of the M feature vectors.

[0088] For example, three test feature vectors are generated: (1,1,0), (1,1,0), and (0,1,0). A value of 1 indicates that the test case covers the corresponding function to be tested; a value of 0 indicates that the test case does not cover the corresponding function to be tested. Different positions in the vector correspond to different functions to be tested. For example, for the test feature vector (1,1,0), its corresponding test case can cover the first and second functions to be tested, but cannot cover the third function to be tested. Since two (1,1,0) test feature vectors are generated, deduplication can be performed.

[0089] For example, the format anomaly test feature vector can include vectors that do not conform to the preset format rules, such as incorrect vector length or incorrect numerical value. The preset format rules can be manually defined in advance. For example, the preset is a 3-bit vector (corresponding to 3 functions to be tested), but a certain test feature vector is 4 bits; for example, the preset uses "1" and "0" to represent coverage or non-coverage, but a certain vector has an invalid value such as 2.

[0090] For example, a feature vector for abnormal functional coverage test can include vectors whose numerical values ​​are in the correct format, but whose coverage does not conform to logic. For example, a test feature vector is (0,0,0), which means that the test case corresponding to the vector does not cover any functional point and is an invalid test case.

[0091] According to embodiments of this disclosure, by performing preprocessing such as deduplication during the feature vector construction stage, duplicate and outlier vectors can be removed, thereby improving the accuracy of subsequent clustering.

[0092] According to embodiments of this disclosure, the preset evaluation metrics include at least one of the following: distance from candidate test cases to the centroid, average functional coverage, historical functional coverage gain, and test execution time.

[0093] For example, the distance from a candidate test case to the centroid can be calculated by taking the average of the feature vectors of all test cases within that cluster. The smaller the distance from the candidate test case to the centroid, the closer the candidate test case is to the central feature vector of that cluster.

[0094] For example, for any candidate test case, the average functional coverage can include the ratio of the number of functions to be tested covered by that candidate test case to the total number of functions to be tested. The average functional coverage can reflect the overall coverage capability of the candidate test cases.

[0095] For example, historical functional coverage gain can characterize how much additional improvement in overall coverage is brought after adding a candidate test case. For instance, for a newly added candidate test case, we can determine the coverage of multiple functions under test by existing candidate test cases before adding the candidate test case, and then determine the increase in coverage after adding the candidate test case. If the coverage was 66% before adding the candidate test case and 80% after adding it, then the historical functional coverage gain of that candidate test case is 14%. If a test case has a low historical functional coverage gain, it indicates that the function under test it covers may have already been covered by other existing candidate test cases, making the candidate test case redundant. For example, test execution time can include the total time required for a candidate test case to execute from the start to the output of the verification result; the shorter the test execution time, the higher the testing efficiency of the candidate test case.

[0096] According to the embodiments of this disclosure, candidate test cases can be comprehensively evaluated from multiple dimensions by using indicators such as the distance from the centroid of the candidate test case, average functional coverage, historical functional coverage gain, and test execution time. This avoids the limitations of evaluating a single indicator and ultimately allows for the selection of test cases with strong coverage, low redundancy, and high execution efficiency.

[0097] Figure 4 A flowchart illustrating a circuit testing method according to another embodiment of this disclosure is shown schematically. Figure 4 As shown, the circuit testing method of this embodiment includes operations S410 to S440.

[0098] When operating the S410, execute M candidate test cases, analyze the execution results using a coverage analysis tool, and generate a coverage report. For example, a coverage group can be added to a predefined verification platform, and initial values ​​for each candidate test case can be specified. Each of the M candidate test cases is executed once, and the coverage analysis tool is used to analyze the test results of the M candidate test cases for multiple functions of the circuit under test. The coverage of the M candidate test cases is collected, and a coverage report is generated. The coverage analysis tool can be determined according to actual needs and is not limited here. The coverage report may include the functional coverage of each coverage group when each candidate test case is executed individually. The coverage group can be defined based on the interface protocol, state machine transition rules, or data path of the functional module under test and is not limited here.

[0099] When operating S420, extract the test feature vector corresponding to each candidate test case from the coverage report, and use the test feature vector as the input dataset for the clustering algorithm.

[0100] For example, test coverage information for multiple functions to be tested can be extracted from the coverage report, and test feature vectors corresponding to M candidate test cases can be generated based on the test coverage information. The test feature vectors can be preprocessed, such as deduplication, and used as the input dataset for a clustering algorithm. Each data point in the dataset represents a candidate test case, and the feature vector of each candidate test case is composed of the coverage rate of each coverage group. For example, if test feature vectors (1,1,0) and (0,1,0) both cover the second function point, one of the test feature vectors is removed during clustering to avoid duplicate features affecting the cluster boundary partitioning effect.

[0101] In operation S430, a clustering algorithm is used to classify the M feature vectors corresponding to the M candidate test cases, resulting in N clusters with distinct features and their respective centroids.

[0102] For example, clustering algorithms can be executed, where candidate test cases in each cluster have similar coverage characteristics, meaning they contribute significantly to one or more specific coverage groups.

[0103] For example, predetermined clustering quality evaluation criteria can be selected to evaluate and iteratively optimize the clustering effect, ultimately returning the optimal clustering result and the centroid of each cluster. Data points within each cluster exhibit high feature similarity, while data points between different clusters show low feature similarity. This corresponds to a high degree of overlap in the functional points covered by candidate test cases within each cluster, and a low degree of overlap in the functional points covered by candidate test cases between different clusters. The centroid of each cluster is the average of the coordinates of all data points within that cluster, possessing the ability to represent the entire cluster. The clustering algorithm and clustering quality evaluation metrics can be set according to actual needs and are not limited here.

[0104] In operation S440, the sorting priority of each feature vector in each cluster is determined, and typical test cases corresponding to the typical feature vectors are selected in descending order of sorting priority until the coverage of typical test cases reaches the expected level.

[0105] For example, the feature vector with the highest ranking priority in each cluster can be selected sequentially, and the test cases corresponding to that feature vector can be used as typical test cases for that cluster, until a set of typical test cases is selected with the same number as the number of clusters. The selected typical test cases are then executed, and the overall functional coverage is calculated after executing these typical test cases. If the overall functional coverage does not reach the expected value, that is, it does not reach the preset value for functional coverage, test cases can be added sequentially according to the ranking priority until the overall functional coverage reaches the expected value.

[0106] For each cluster, test cases are ranked according to their priority. Then, representative test cases with the highest priority in each cluster are selected as the first test case set, the second highest priority as the second test case set, and so on. Through automated script control, the first test case set is executed first to collect coverage data, allocating the maximum number of random attempts to each test case in this set. If the coverage does not meet expectations, the second test case set is executed to collect coverage data and allocate the second-highest number of random attempts, until coverage converges. The remaining test cases after coverage convergence are considered redundant and can be excluded from all subsequent regression tests.

[0107] For example, to address the issue of random testing failing to identify which functionalities it corresponds to, we can link constrained random tests with functionalities to create a functionality coverage profile. If a certain type of constraint consistently hits a specific functionality, we can use this constraint to generate a test vector only once in subsequent iterations; if a certain type of constraint covers a functionality less frequently, we can use tests containing this type of constraint multiple times to generate random tests in subsequent regressions.

[0108] According to embodiments of this disclosure, by determining the sorting priority of each test case, fewer test cases can be used to fully cover the function to be tested, and the clustering process only needs to be executed once initially, and the previous clustering results can be used in subsequent iterations.

[0109] According to embodiments of this disclosure, in related circuit testing methods, all test cases are typically executed without any execution priority. This results in a small number of typical test cases covering most functionalities, while the remaining test cases are highly repetitive, covering almost no additional functionalities, yet requiring significant additional execution time. The circuit testing method of this disclosure, by classifying test cases using a clustering algorithm, provides a clearer understanding of which functionalities each test case covers. This facilitates engineers in optimizing the test structure, reducing the execution frequency of redundant test cases, and effectively shortening verification time. By selecting typical test cases based on ranking priority, the method ensures that test cases with the greatest coverage improvement are executed first, thereby increasing functional coverage to the expected level in the shortest time. Limited time resources are then focused on developing test cases for uncovered functionalities, improving the completeness of functional verification. Compared to related circuit testing methods, this approach allows for more rational allocation of test resources, effectively shortening testing time while ensuring test completeness.

[0110] Figure 5 A flowchart illustrating the iterative process of a clustering algorithm according to an embodiment of this disclosure is shown schematically. Figure 5 As shown, the circuit testing method of this embodiment includes operations S510 to S580.

[0111] When operating S510, input the optimal number of clusters K and the dataset.

[0112] For example, the number of clusters K and the dataset are required inputs. Optional inputs include the maximum number of iterations, the number of times the algorithm is run with different initial centroids, and the method for choosing the initial centroids. If these options are not specified, the default values ​​are used.

[0113] When operating S520, K data points are randomly selected as the initial centroids.

[0114] For example, a selection principle could be that the distance between initial centroids should be as far as possible. One approach is to first randomly select a data point from the dataset as the first initial centroid, and then calculate the distance between each of the remaining data points and the nearest initial centroid. . The calculation can be performed using Euclidean distance. Assume we need to calculate n-dimensional data points. and The Euclidean distance between them is calculated using the following formula (4).

[0115] (4)

[0116] For example, after calculating the Euclidean distance of all data points Then, the probability of each data point being selected as the next initial centroid can be determined based on its numerical value. The larger the value, the greater the probability of selection. The probability of each data point being selected is calculated. The formula is shown in (5).

[0117] (5)

[0118] Here, X is the dataset. Operation S520 can be performed iteratively until K initial centroids are found.

[0119] In operation S530, the distance between each data point and the K centroids is calculated, and the data point is added to the cluster containing the nearest centroid. For example, the distance between each data point and the K centroids can be calculated using Euclidean distance, and then a new column can be added to the dataset matrix as a label to store the cluster number to which each data point belongs.

[0120] In operation S540, the mean of all data points in each cluster is calculated as the new centroid, and the calculation method is shown in the following formula (6).

[0121] (6)

[0122] Among them, c k Center of mass, C k For the center of mass c k The set of data points in the corresponding cluster.

[0123] When operating the S550, calculate the target distance function.

[0124] For example, when the clustering algorithm is K-means, the target distance function of K-means can be calculated using the following formula (7).

[0125] (7)

[0126] For example, by calculating the target distance function, the distance from each data point in a cluster to the cluster centroid can be minimized. Since it is a monotonically decreasing function with a lower bound, the convergence of the results can be ensured.

[0127] When operating the S560, determine whether the expected convergence effect of the algorithm has been achieved.

[0128] When operating S570, if the expected convergence effect of the algorithm is achieved, the iteration ends and the clustering results and evaluation metrics are output.

[0129] In operation S580, if the expected convergence effect of the algorithm is not achieved, it is determined whether the upper limit of the number of iterations has been reached. For example, if the expected convergence effect is achieved, the iteration can be terminated and the clustering results and evaluation metrics can be output. If the expected convergence effect is not achieved, operation S530 can be returned to calculate the distance between each data point and the K centroids, and the data point can be added to the cluster containing the nearest centroid. By having the clustering algorithm and the clustering effect evaluation algorithm work together and perform multiple iterations until the optimal number of clusters and clustering results are found, and the evaluation metric values ​​are given, this method can be applied to large-scale random testing scenarios with fuzzy cluster boundaries and high requirements for clustering quality.

[0130] Based on the above circuit testing method, this disclosure also provides a circuit testing apparatus. The following will be combined with... Figure 6 The device is described in detail. Figure 6 A schematic block diagram of a circuit testing apparatus according to an embodiment of the present disclosure is shown.

[0131] like Figure 6 As shown, the circuit testing device 600 of this embodiment includes a first testing module 610, a clustering module 620, a first determining module 630, a second determining module 640, and an iteration module 650.

[0132] The first test module 610 is used to execute tests on multiple functions of the circuit under test using M candidate test cases, and obtain M first test results, where M is an integer greater than 1. In one embodiment, the first test module 610 can be used to execute the operation S210 described above, which will not be repeated here.

[0133] The clustering module 620 is used to cluster the M first test results to obtain a set of N candidate test cases. In one embodiment, the clustering module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0134] The first determining module 630 is used to determine the ranking priority of each candidate test case in the N sets of candidate test cases according to preset evaluation indicators. In one embodiment, the first determining module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0135] The second determining module 640 is used to determine N first target test cases from the N sets of candidate test cases according to their sorting priority, and execute the N first target test cases to obtain a second test result. In one embodiment, the second determining module 640 can be used to execute the operation S240 described above, which will not be repeated here.

[0136] The iteration module 650 is used to iteratively execute the operation of determining N second target test cases from the N sets of candidate test cases according to the sorting priority when it is determined that the second test result does not reach the preset value of functional coverage, until the second test result reaches the preset value of functional coverage, wherein the sorting priority of the second target test cases is lower than that of the first target test cases. In one embodiment, the iteration module 650 can be used to execute the operation S250 described above, which will not be repeated here.

[0137] The iteration module may include a first selection submodule and a judgment submodule. The first selection submodule is used to select second target test cases whose sorting priority meets a preset value condition from each set of candidate test cases if the N second target test cases determined in the Kth iteration operation do not reach the preset value of functional coverage, thereby obtaining the N second target test cases determined in the K+1th operation, where K is a positive integer; the judgment submodule is used to determine whether the N second target test cases determined in the K+1th operation reach the preset value of functional coverage; wherein, the sorting priority of the N second target test cases determined in the K+1th operation is less than that of the N second target test cases determined in the Kth iteration operation.

[0138] The circuit testing apparatus also includes a follow-up testing module. This follow-up testing module is used to perform subsequent tests on multiple functions under test based on N first target test cases, provided that the second test result meets the preset functional coverage value.

[0139] The clustering module 620 includes: a coverage report generation submodule, used to generate M coverage reports for M first test results, wherein each coverage report includes P coverage groups and P functional coverage rates corresponding to the P coverage groups, wherein the coverage group includes one type of functional to be tested from multiple functional to be tested, and P is a positive integer; a feature vector construction submodule, used to construct a feature vector for each coverage report, taking the P functional coverage rates as P features, and obtaining M feature vectors; a preprocessing submodule, used to preprocess the M feature vectors; and a clustering submodule, used to cluster the preprocessed M feature vectors using the target number of clusters.

[0140] The clustering module 620 may further include an iterative submodule, configured to: perform a first clustering on the preprocessed M feature vectors using the initial clustering count, generating a first clustering result, wherein the first clustering result includes at least one cluster, and the number of clusters is equal to the initial clustering count; generate a next clustering count for the next clustering based on the index value of the clustering result relative to a preset clustering quality evaluation index, and perform a next clustering on the preprocessed M feature vectors using the next clustering count to generate a next clustering result; iteratively execute the generation of a next clustering count for the next clustering based on the index value of the clustering result relative to the preset clustering quality evaluation index, and perform the next clustering operation, until a preset iteration condition is met, and output the target clustering count. According to an embodiment of this disclosure, the second determining module includes a second selection submodule. The second selection submodule is configured to select the candidate test case with the highest ranking priority from each set of candidate test cases as the first target test case.

[0141] According to embodiments of this disclosure, any plurality of modules among the first test module 610, clustering module 620, first determination module 630, second determination module 640, and iteration module 650 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first test module 610, clustering module 620, first determination module 630, second determination module 640, and iteration module 650 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the first test module 610, clustering module 620, first determination module 630, second determination module 640, and iteration module 650 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0142] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a circuit testing method according to an embodiment of the present disclosure. Figure 7As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure. Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. The processor 701 performs various operations of the method flow according to an embodiment of the present disclosure by executing programs in the ROM 702 and / or RAM 703. It should be noted that the program may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 can also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0143] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0144] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure. The computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0145] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the methods provided in the embodiments of this disclosure. When the computer program is executed by processor 701, it performs the functions defined in the system / apparatus of the embodiments of this disclosure. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules. The computer program can rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program can also be transmitted and distributed in the form of signals over a network medium and downloaded and installed via communication section 709, and / or installed from removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of the embodiments of this disclosure. The systems, devices, apparatuses, modules, and units described above can be implemented using computer program modules. Program code for executing the computer programs provided in the embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0147] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A circuit testing method, wherein, The method includes: M candidate test cases are used to perform tests on multiple functions of the circuit under test, resulting in M ​​first test results, where M is an integer greater than 1; the M first test results are then clustered to obtain N sets of candidate test cases, where N is an integer greater than or equal to 1. Based on preset evaluation indicators, determine the ranking priority of each candidate test case in the N sets of candidate test cases; Based on the sorting priority, N first target test cases are determined from the N sets of candidate test cases, and the N first target test cases are used to perform tests on the multiple functions to be tested to obtain a second test result. If it is determined that the second test result does not reach the preset value of functional coverage, the operation of determining N second target test cases from the N sets of candidate test cases according to the sorting priority is performed iteratively until the second test result reaches the preset value of functional coverage, wherein the sorting priority of the second target test cases is less than the sorting priority of the first target test cases.

2. The method according to claim 1, wherein, The clustering using the M first test results includes: M coverage reports are generated for the M first test results, wherein each coverage report includes P coverage groups and P function coverages corresponding to the P coverage groups, wherein the coverage group includes one type of function to be tested from the plurality of functions to be tested, and P is a positive integer; For each of the coverage reports, the P functional coverage rates are used as P features, and a feature vector is constructed for the P features to obtain M feature vectors; The M feature vectors are preprocessed; The preprocessed M feature vectors are clustered using the target clustering number.

3. The method according to claim 2, wherein, The method further includes: The preprocessed M feature vectors are clustered for the first time using a preset initial number of clusters to generate the first clustering result, wherein the first clustering result includes at least one cluster, and the number of clusters is equal to the initial number of clusters. Based on the clustering results and the index values ​​of the preset clustering quality evaluation index, the next number of clusters is generated for the next clustering, and the next number of clusters is used to perform the next clustering on the preprocessed M feature vectors to generate the next clustering result. The iterative process generates the next cluster count based on the clustering results and the values ​​of preset clustering quality evaluation indicators, and performs the next clustering operation until the preset iteration conditions are met, and then outputs the target cluster count.

4. The method according to claim 1, wherein, Performing an operation to determine N second target test cases from the N sets of candidate test cases according to the sorting priority includes: If the test results corresponding to the N first target test cases and the N second target test cases determined by the Kth iteration operation do not reach the preset value of functional coverage, for each set of candidate test cases, select the second target test cases whose sorting priority meets the preset numerical condition from the candidate test case sets to obtain the N second target test cases determined by the K+1th operation, where K is a positive integer; Determine whether the test results corresponding to the N first target test cases, the N second target test cases determined by the Kth iteration operation, and the N second target test cases determined by the K+1th operation reach the preset value of functional coverage; The sorting priority of the N second target test cases determined in the (K+1)th operation is lower than that of the N second target test cases determined in the Kth iteration operation.

5. The method according to claim 1, wherein, The method further includes: If the second test result is determined to reach the preset value of functional coverage, subsequent tests on the multiple functions to be tested are performed based on the N first target test cases.

6. The method according to claim 1, wherein, The step of determining N first target test cases from the N sets of candidate test cases according to the sorting priority includes: For each set of candidate test cases, the candidate test case with the highest ranking priority is selected as the first target test case.

7. The method according to claim 2, wherein, The preprocessing of the M feature vectors includes at least one of the following: deduplication, removal of invalid test feature vectors. The invalid test feature vector includes at least one of the following: format anomaly test feature vector and functional coverage anomaly test feature vector.

8. The method according to claim 1, wherein, The preset evaluation metrics include at least one of the following: distance from candidate test cases to the centroid, average functional coverage, historical functional coverage gain, and test execution time.

9. A circuit testing device, wherein, The device includes: The first test module is used to execute multiple functions of the circuit under test using M candidate test cases, and obtain M first test results, where M is an integer greater than 1. The clustering module is used to cluster the M first test results to obtain a set of N candidate test cases. The first determining module is used to determine the sorting priority of each candidate test case in the N sets of candidate test cases according to preset evaluation indicators; The second determining module is used to determine N first target test cases from the N sets of candidate test cases according to the sorting priority, and use the N first target test cases to perform tests on the multiple functions to be tested to obtain a second test result; An iteration module is used to iteratively execute the operation of determining N second target test cases from the N sets of candidate test cases according to the sorting priority when it is determined that the second test result does not reach the preset value of functional coverage, until the second test result reaches the preset value of functional coverage, wherein the sorting priority of the second target test cases is lower than that of the first target test cases.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images