Regression testing method and device

Abstract

The embodiment of the invention provides a regression test method and device, and the method comprises the steps: determining an error case of a test error from a historical regression record; all the error use cases are sequenced, sequenced error use cases are obtained, and error use case information of the sequenced error use cases at least comprises one of use case names of the error use cases, seed numbers of the error use cases, the error number of the error use cases and use case numbers of the error use cases; and generating a new regression test sequence by using the error case information, and performing regression test by using the new regression test sequence in the verification environment. Through the method and the device, the technical problem of low regression test efficiency in a chip verification process in related technologies is solved, so that the regression test efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular, to a regression testing method and device. Background Art

[0002] With the rapid development of related technologies of large-scale integrated circuit systems, the scale of integrated circuits is getting larger and larger, and the verification operation of circuit chips accounts for an increasingly important proportion in the chip development process. The purpose of verification is to test as many circuit designs as possible within a certain period of time and find defects. At present, in the verification process, regression testing of use cases is an important part of improving convergence efficiency throughout the entire verification life cycle. After the verification personnel develop the verification environment, prepare the stimulus test, and develop the check component for automatic comparison results, they will perform regression testing of the use cases, that is, execute all existing test sequences.

[0003] As chip complexity increases, verification becomes increasingly difficult, the number of cases that require regression is increasing, and the verification time is also increasing. During the development process, both the RTL (Register Transfer Level Design) code and the verification environment require multiple iterations, and each regression test will have erroneous cases caused by various reasons. Current regression tools can only simply obtain the results of test pass or test failure. When repairing based on the results of test failure, in order to avoid new problems caused by the repair, a large number of logs must be manually queried and analyzed to obtain more detailed failure details. However, the manual analysis process is very cumbersome, error-prone, time-consuming, and inefficient. It is impossible to quickly formulate new test sequences and provide appropriate optimization plans.

[0004] Therefore, in the related art, there is a technical problem of low efficiency of regression testing during the chip verification process.

[0005] There is currently no effective solution to this problem. Summary of the invention

[0006] The embodiments of the present application provide a regression testing method and device to at least solve the technical problem of low efficiency of regression testing in the chip verification process in the related art.

[0007] According to one embodiment of the present application, a regression testing method is provided, which is applied to a regression testing platform, wherein the regression testing platform is built with a verification environment, and the regression testing platform is pre-installed with a regression testing module, and the regression testing platform is used to control the regression testing module to perform regression testing under the verification environment, including: determining error cases of test errors from historical regression records; sorting all error cases to obtain sorted error cases, wherein error case information of the sorted error cases includes at least one of the following: a case name of the error case, a seed number of the error case, the number of errors in the error case, and a case number of the error case; using the error case information to generate a new regression test sequence, and using the new regression test sequence to perform regression testing under the verification environment.

[0008] In an exemplary embodiment, the method of determining the erroneous use cases of test errors from the historical regression records includes: extracting log information from the use case simulation log in the historical regression records according to the extraction fields, wherein the extraction fields include at least one of the following: the user name of the test case, the use case name of the test case, the seed number of the test case, the number of errors in the test case, the timeout information of the test case, and the use case number of the test case; classifying the log information according to the test results of the test case to obtain the correct use cases of the test correctness and the erroneous use cases of the test errors, wherein the test results include the test correctness and the test errors.

[0009] In an exemplary embodiment, all error cases are sorted to obtain sorted error cases, including: sorting all error cases in ascending order of the number of errors in the error cases to obtain the sorted error cases; or sorting all error cases in ascending order of the case numbers of the test cases to obtain the sorted error cases; or classifying all error cases according to the case numbers of the test cases to obtain multiple groups of first error cases, wherein each group of first error cases in the multiple groups of first error cases corresponds to a case number; sorting the multiple groups of first error cases in ascending order of the case numbers of the test cases to obtain the sorted multiple groups of first error cases; and sorting the sorted multiple groups of first error cases according to the case numbers of the test cases. For each group of first error cases in the error cases, sort the error cases in each group of first error cases in ascending order according to the number of errors in the error cases, to obtain a group of second error cases corresponding to each group of first error cases; determine the sorted error cases based on multiple groups of second error cases; or, sort all the error cases in ascending order according to the number of errors in the error cases, to obtain sorted intermediate error cases; classify the sorted intermediate error cases according to the case number of the test case, to obtain multiple groups of third error cases after classification, wherein each group of third error cases corresponds to a case number; sort the multiple groups of third error cases in ascending order according to the case number of the test case, to obtain the sorted error cases.

[0010] In an exemplary embodiment, the error case information is used to generate a new regression test sequence, including: determining a sequence generation condition pre-set for a target object based on the error case information; splitting the sorted error case sequences into two groups of case sequences according to the sequence generation condition; and when it is determined that the number of error case sequences included in the two groups of case sequences is less than or equal to a preset value, merging the two groups of case sequences to obtain the new regression test sequence.

[0011] In an exemplary embodiment, the error case information includes the number of errors of the error case, and using the error case information to generate a new regression test sequence includes: determining the debugging priority of the error case using the number of errors of the error case, wherein the greater the number of errors, the higher the debugging priority; determining target debugging information corresponding to the error type of the error case from preset debugging information, wherein the target debugging information includes at least debugging time and debugging plan; generating an initial regression test sequence corresponding to the error case using the debugging time and the debugging plan, wherein the debugging level of the initial regression test sequence is a default level; and updating the debugging level of the initial regression test sequence to the debugging level corresponding to the debugging priority to obtain the new regression test sequence.

[0012] In an exemplary embodiment, before using the error case information to generate a new regression test sequence, the method further includes: determining multiple error cases with the same case name from the sorted error cases; and when it is determined that the seed numbers, error quantities and case numbers of the multiple error cases are all consistent, retaining any one error case from the multiple error cases and deleting other error cases.

[0013] In an exemplary embodiment, after performing regression testing using the new regression test sequence, the method further includes: obtaining regression test results of the regression testing using the new regression test sequence; recursively analyzing multiple regression test results using a recursive algorithm to obtain common error cases in the multiple regression test results; and determining repair results of repairing the error cases corresponding to the multiple regression test results; and sending a notification message generated based on the common error case and the repair result to a target object.

[0014] According to another embodiment of the present application, a regression testing device is provided, including: a determination module, used to determine an error case of a test error from historical regression records; an acquisition module, used to sort all error cases to obtain sorted error cases, wherein the error case information of the sorted error case includes at least one of the following: the case name of the error case, the seed number of the error case, the number of errors of the error case, and the case number of the error case; a generation module, used to generate a new regression test sequence using the error case information, and perform regression testing using the new regression test sequence in the verification environment.

[0015] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0016] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0018] Through the present application, a verification environment is pre-built in the regression test platform, and a regression test module is pre-installed in the regression test platform. The regression test platform is used to control the regression test module to perform regression testing under the verification environment, and the error case of the test error is determined from the historical regression record. Specifically, all error cases are sorted to obtain the sorted error cases, wherein the error case information of the sorted error cases includes at least one of the following: the case name of the error case, the seed number of the error case, the number of errors of the error case, and the case number of the error case; a new regression test sequence is generated using the error case information, and regression testing is performed using the new regression test sequence under the verification environment. That is, by automatically extracting the error case information such as the case name, seed number, and number of errors from the historical regression records that have been obtained, and then generating a new regression case list (i.e., regression test sequence) according to the error case information, so as to use the new regression case list for regression testing and continuously optimize the regression test design. Therefore, the technical problem of low efficiency of regression testing in the chip verification process in the related art is solved, thereby improving the efficiency of regression testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a hardware structure block diagram of a server device of a regression testing method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a regression testing method according to an embodiment of the present application;

[0021] Figure 3 is a flowchart of a regression test according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a regression use case execution process according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of the structure of a regression testing device according to an embodiment of the present application;

[0024] Figure 6It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a regression testing method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the regression test method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] Next, some technical terms in this application are explained:

[0031] UVM, Universal Verification Methodology, Universal Verification Methodology. UVM refers to a methodology for hardware verification of electronic systems applied to complex integrated circuits. It can be specifically a class library and framework of SystemVerilog, which uses a set of standardized methods and interfaces to build reusable verification components.

[0032] SV, System Verilog, is a hardware description language for electronic system design and verification. SystemVerilog extends the Verilog language, adding object-oriented programming, enhanced testing capabilities, data structure support, and transaction-level modeling capabilities, making it very suitable for designing complex integrated circuits and systems.

[0033] TB, TestBench, is a verification environment built to simulate the hardware design environment, including the code and logic used to generate test vectors, monitor outputs, and verify the design. When the testbench is updated, it may be necessary to rerun the test to ensure that the test covers all new or modified functions.

[0034] SoC, System on a Chip, System on a Chip. System on a chip refers to integrating most or all functions of a computer or other electronic system into a single integrated circuit (chip). This integrated system is usually used to reduce system cost, power consumption, and improve efficiency.

[0035] Regression testing is a method in software testing. After each software change (such as patching, functional enhancement, etc.), relevant tests need to be re-executed to ensure that the change has not introduced new errors. In hardware design and verification, regression testing ensures that the modified code or design still meets all expected standards and functions.

[0036] It should be noted that in the field of chip design and verification, regression testing is an automated testing process used to verify whether the code after the design modification meets the expectations and no new errors are introduced. When performing regression testing, it is very important to ensure that all relevant design files (including RTL code and test platforms) are up to date and the test results are reliable. This helps maintain the quality and reliability of the design.

[0037] RTL design, hardware design at the register transfer level, is a stage in integrated circuit design. At this level, the design is described as a series of registers and the logical operations between them, as well as the transfer of data between these registers. RTL is usually written in a hardware description language (such as Verilog or VHDL) and is the intermediate step from hardware design concept to actual circuit implementation.

[0038] RTL_UPDATE refers to updates made to the RTL code. In the context of regression testing, this can mean that the RTL code was updated or modified and the tests need to be rerun to verify those changes.

[0039] DUT, design under test, refers to the hardware design or component whose functionality, performance, and reliability need to be verified during the testing process. During regression testing, the test team runs a series of test cases against the DUT to verify that the design meets the specifications and no new issues have been introduced due to recent code changes.

[0040] MSA, Merge Sort, Merge Sort Algorithm, is an efficient sorting algorithm that uses divide-and-conquer. It divides the array into two halves, recursively sorts each half, and then merges the sorted halves together. Merge sort has a time complexity of O(n log n) in the best, worst, and average cases, so it is very effective when processing large data sets.

[0041] In this embodiment, a regression testing method is provided. Figure 2 is a flowchart of a regression testing method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0042] Step S202, determining the error case of the test error from the historical regression record;

[0043] Step S204, sorting all the error cases to obtain sorted error cases, wherein the error case information of the sorted error cases includes at least one of the following: the case name of the error case, the seed number of the error case, the number of errors of the error case, and the case number of the error case;

[0044] Among them, for the above error use case information, for example, the error quantity is sorted and other processing is performed, and the error use case information is effectively managed subsequently.

[0045] Step S206: Generate a new regression test sequence using the error case information, and perform regression testing using the new regression test sequence in the verification environment.

[0046] The purpose of each regression test may include ensuring that old problems are fixed, ensuring that changes do not introduce new problems, and ensuring that new functions work correctly. Therefore, the new regression test sequence generated is generally based on the previous regression test sequence.

[0047] Through the above steps, a verification environment is pre-built on the regression test platform, and a regression test module is pre-installed in the regression test platform. The regression test platform is used to control the regression test module to perform regression testing under the verification environment, and the error case of the test error is determined from the historical regression record. Specifically, all error cases are sorted to obtain sorted error cases, wherein the error case information of the sorted error case includes at least one of the following: the case name of the error case, the seed number of the error case, the number of errors of the error case, and the case number of the error case; a new regression test sequence is generated using the error case information, and regression testing is performed using the new regression test sequence under the verification environment. That is, by automatically extracting the error case information such as the case name, seed number, and number of errors from the historical regression records that have been obtained, and then generating a new regression case list (i.e., regression test sequence) according to the error case information, so as to use the new regression case list for regression testing and continuously optimize the regression test design. Therefore, the technical problem of low efficiency of regression testing in the chip verification process in the related art is solved, thereby improving the efficiency of regression testing.

[0048] The execution subject of the above steps may be a server, terminal, etc. where the regression test platform is located, but is not limited thereto.

[0049] In an exemplary embodiment, the implementation process of determining the error case of the test error from the historical regression record in step S202 specifically includes: extracting log information from the case simulation log in the historical regression record according to the extraction field, wherein the extraction field includes at least one of the following: the user name of the test case, the case name of the test case, the seed number of the test case, the number of errors of the test case, the timeout information of the test case, and the case number of the test case; classifying the log information according to the test result of the test case to obtain the correct case of the test correct and the error case of the test error, wherein the test result includes the test correct and the test error. This embodiment uses automation instead of manual work to realize the automatic extraction of the error case of the test error, and the user does not need to perform additional operations for a certain project, with low maintenance cost and strong reusability, which is very friendly to the management of error problems and project planning. It can also reduce the time required for subsequent chip verification convergence, thereby shortening the verification cycle.

[0050] In an exemplary embodiment, a technical solution for sorting all error cases to obtain sorted error cases is further provided, and the specific steps include: sorting all error cases in ascending order according to the number of errors in the error cases to obtain the sorted error cases; or sorting all error cases in ascending order according to the case number of the test case to obtain the sorted error case; or classifying all error cases according to the case number of the test case to obtain multiple groups of first error cases, wherein each group of first error cases in the multiple groups of first error cases corresponds to a case number; sorting the multiple groups of first error cases in ascending order according to the case number of the test case to obtain multiple groups of sorted first error cases; for For each group of the sorted multiple groups of first error cases, sort the error cases in each group of the first error cases in the order of the number of errors in the error cases from small to large, and obtain a group of second error cases corresponding to each group of the first error cases; determine the sorted error cases based on multiple groups of second error cases; or, sort all the error cases in the order of the number of errors in the error cases from small to large, and obtain sorted intermediate error cases; classify the sorted intermediate error cases according to the case number of the test case, and obtain multiple groups of third error cases after classification, wherein each group of third error cases corresponds to a case number; sort the multiple groups of third error cases in the order of the case number of the test case from small to large, and obtain the sorted error cases. This embodiment is based on the principle of the merge classification algorithm, and by sorting the number of errors, it is convenient to output the corresponding error information and regression list according to the sorted error cases and actual configuration requirements.

[0051] It should be noted that the merge sorting algorithm is a sorting algorithm suitable for large data volume sorting tasks. Specifically, it uses a divide-and-conquer strategy to divide a sequence into two shorter subsequences, sort the subsequences separately, and then merge the sorted subsequences into an ordered sequence. The principles of the merge sorting algorithm can include:

[0052] , Sequence decomposition: Divide the original data sequence into two parts. If the size of the sequence is already less than or equal to, then return the sequence directly because they are already ordered.

[0053] 2. Recursive decomposition: Continue to perform the decomposition operation on the two subsequences obtained until each subsequence contains only one element.

[0054] 3. Sequence merging: Merge the ordered subsequences obtained by decomposition to form a new ordered sequence.

[0055] 4. Recursive merging: Continue to perform the merging operation on the merged ordered sequence until the final ordered sequence is obtained.

[0056] Compared with other decomposition and sorting algorithms, the merge classification algorithm can quickly and orderly generate error information files and corresponding error regression lists based on a large number of error cases.

[0057] Optionally, this embodiment can also obtain an error case with a single-point error or an error case with multiple-point errors according to the sorting result of the error quantity, wherein a single-point error indicates that there is only one error type, and multiple-point errors indicate that at least two error types occur.

[0058] In an exemplary embodiment, the implementation process of using the error use case information to generate a new regression test sequence in step S206 is described through the following steps: determining a sequence generation condition pre-set for the target object based on the error use case information; splitting the sorted error use cases into two groups of use case sequences according to the sequence generation condition; and when it is determined that the number of error use cases included in the two groups of use case sequences is less than or equal to a preset value, merging the two groups of use case sequences to obtain the new regression test sequence.

[0059] Optionally, when it is determined that the number of erroneous use cases contained in the two groups of use case sequences is greater than the preset value, for the two groups of use case sequences, continue to split the two groups of use case sequences into new sub-use case sequences according to the sequence generation condition until the number of erroneous use cases contained in the new sub-use case sequences is less than the preset value; and merge all the new sub-use case sequences to obtain the new regression test sequence.

[0060] Optionally, for sequence generation conditions, users can set them based on previous error cases and case seed numbers to maximize the guarantee that the same stimulus can occur.

[0061] Among them, the process of pre-setting the sequence generation conditions of the target object based on the erroneous use case information can be understood as the user pre-defining the information that needs to be collected, such as the type of error, the design module corresponding to the first error, the use case seed number and other information. Then use this information to obtain the sequence generation conditions. For example, for the condition that includes all erroneous use cases and use case seed numbers, a sequence including all erroneous use cases and use case seed numbers can be generated. Alternatively, a sequence generation condition is set to more accurately classify the error information. Of course, the user can generate an error regression list based on different actual needs, so that the erroneous use cases can be efficiently analyzed, and the regression list can be selected for regression according to the changes.

[0062] In an exemplary embodiment, for the case where the error case information includes the number of errors of the error case, the process of generating a new regression test sequence using the error case information is described by the following scheme: determining the debugging priority of the error case using the number of errors of the error case, wherein the more the number of errors, the higher the debugging priority; determining the target debugging information corresponding to the error type of the error case from the preset debugging information, wherein the target debugging information at least includes the debugging time and the debugging plan; generating an initial regression test sequence corresponding to the error case using the debugging time and the debugging plan, wherein the debugging level of the initial regression test sequence is the default level; updating the debugging level of the initial regression test sequence to the debugging level corresponding to the debugging priority, and obtaining the new regression test sequence. Based on this embodiment, the user can obtain a more accurate debugging time and debugging plan according to the error type of the test case, generate an initial regression test sequence using the debugging time and debugging plan, analyze the priority order of the case debugging according to the sorting result of the number of errors, and then update the debugging level of the initial regression test sequence to the debugging level corresponding to the debugging priority, and then obtain the regression test sequence that needs to be tested after the modification, thereby improving the efficiency and accuracy of the test.

[0063] In an exemplary embodiment, before the error case information is used to generate a new regression test sequence, multiple error cases with the same case name can be further determined from the sorted error cases; when the seed numbers, error numbers and case numbers of the multiple error cases are determined to be consistent, any one error case is retained from the multiple error cases and the other error cases are deleted. This embodiment can delete repeated errors and only count the number of different errors, thereby improving the accuracy of the regression test sequence.

[0064] Optionally, in other embodiments, multiple error cases with the same seed number can be determined from the sorted error cases; when it is determined that the case names, number of errors and case numbers of the multiple error cases are consistent, any one error case is retained from the multiple error cases and the other error cases are deleted.

[0065] In addition, this application can select different error regression list use cases according to parameter definitions, configuration files, etc. It is not limited to a single project and can be inherited and used between different projects, with strong scalability and continuity.

[0066] Based on the above embodiment, a minimum error regression use case set can also be generated according to the classification and sorting results of the error use cases. According to the design changes, the repeated iterations of the passed use cases can be reduced, and the simulation results can be quickly obtained to determine whether the changes are effective, thereby improving the verification efficiency and facilitating project iteration.

[0067] In an exemplary embodiment, after the regression test is performed using the new regression test sequence, a technical solution is further provided, which specifically includes: obtaining the regression test results of the regression test using the new regression test sequence; recursively analyzing multiple regression test results using a recursive algorithm to obtain common error cases in the multiple regression test results; and determining repair results of repairing the error cases corresponding to the multiple regression test results; and sending a notification message generated based on the common error case and the repair result to a target object.

[0068] Optionally, in one embodiment, a corresponding error repair curve chart may be drawn for the repair results of the error use cases corresponding to the above-mentioned multiple regression test results, so as to be used as a further risk report for project management.

[0069] Optionally, this embodiment may also select one or more sorting results to generate a corresponding regression list (i.e., the above-mentioned regression test sequence), thereby effectively reducing the frequent manual search operations on the execution logs of the regression use cases. At the same time, recursive regression may be performed on the error use cases multiple times to analyze the error use cases that have not been fixed.

[0070] Further, combined with Figure 3 The process shown in FIG. 1 illustrates the process of regression testing of this application. Figure 3 As shown, the following steps are included:

[0071] Step S1, analyze the regression test case log, find the error log in all simulation logs to extract the test case information. Specifically, in the regression test platform, extract the following information from the log: user name, test case seed number, number of errors, test case number, timeout information.

[0072] The error number includes, for example, the number of UVM_ERROR, the number of UVM_FATAL, the number of ERROR, and the number of FATAL. The timeout information is, for example, the TIMEOUT information, and the use case number can be understood as the ID information corresponding to the predefined error information printing format, namely, ERR_ID.

[0073] In the UVM (Universal Verification Methodology) environment, UVM provides a series of macros and methods to represent different error levels and information. The following are common error and information types in UVM and their meanings:

[0074] 1.UVM_ERROR: An error-level macro used to report minor problems, that is, problems that do not affect the continuation of testing. For example, expected non-compliant behavior under a certain condition or minor problems.

[0075] 2.UVM_FATAL: A higher level error used to report serious problems that may prevent the test from continuing or make the test results unreliable. Usually, once the UVM_FATAL error is recorded, the test stops.

[0076] 3.ERROR: An alias for UVM_ERROR, used for backward compatibility. In some cases, it may be used to indicate an error, but the specific meaning may depend on the UVM version used or the specific implementation of the team.

[0077] 4.FATAL: An alias for UVM_FATAL, also used for backward compatibility. It indicates that a serious error was encountered during the test, which usually causes the test to stop.

[0078] 5.TIMEOUT: Usually refers to a timeout event, which means that an operation or waiting for a condition takes too long and exceeds the preset threshold. In UVM, this may be reported through a specific macro or method to indicate that an expected event or response did not occur within the expected time.

[0079] In the UVM environment, the correct use of these macros and methods is essential to ensure the accuracy and reliability of testing. They help verification engineers identify and deal with problems encountered in testing, as well as ensure the stability of the test environment and the repeatability of test results.

[0080] Exemplarily, the simulation log sim.log contains the following information about the UVM environment:

[0081] Report counts by severity.

[0082] UVM_INFO: 1572.

[0083] UVM_WARNING: 6.

[0084] UVM_ERROR: 14.

[0085] UVM_FATAL: 0.

[0086] And the following print information is included in the custom C use case:

[0087] ERROR: sync2_cell_chk, input data not stable.

[0088] ERROR: data compare mismatch.

[0089] If the above print information is triggered, it will be printed once or multiple times.

[0090] And also includes display printing information for debugging convenience. For example: "SimulError: at 15680ps, "Null pipe: RxVld must be low when Pwron is low."".

[0091] Alternatively, it can also include timeout information generated when the use case is not completed within the specified time.

[0092] Among them, the seed number of the use case can be extracted from the log, such as: "NOTE: automatic random seedused: 701709163".

[0093] Exemplarily, the ID number (case number) of the first error can also be extracted from the following information:

[0094] "UVM_ERROR@0.0ns: uvm_test_top.env.master_agent.seqr.rsp_export[Connection Error]connection count of 0 does not meet required minimum of".

[0095] For the UVM_INFO, UVM_WARNING and other information mentioned above, the above information can be extracted through the following script:

[0096] “import re;

[0097] from openpyxl import Workbook;

[0098] log_file_path=′simrun.log′;

[0099] excel_file_path=′error_data_ori.xlsx′;

[0100] log_line_pattern=re.compile(r′^(.+?),(\d+),(\d+),(\d+)$′);

[0101] workbook=Workbook();

[0102] sheet=workbook.active;

[0103] titles=[′Use Case Name′,′Seed′,′Error_num′,′ERR_ID′]sheet.append(titles);

[0104] with open(log_file_path,′r′)as file:

[0105] for line in file:

[0106] match=log_line_pattern.match(1ine.strip());

[0107] if match:

[0108] use_case_name,seed,error_num,err_id=match.groups();

[0109] sheet.append([use_case_name,seed,error_num,id]);

[0110] workbook.save(excel_file_path);

[0111] printf(Error_Infor has been extracted and saved to{excel_file_path}′”。

[0112] The representation format of the extracted error information is shown in Table 1 below, for example, including the error use case name (Use_Case_Name), seed number (Seed), error number (Error_num) and error number (ERR_ID).

[0113] Table 1

[0114]

[0115]

[0116] Furthermore, Table 1 can be sorted by the following script:

[0117]

[0118]

[0119] By stacking and sorting the number of use cases of all erroneous use cases according to the above code script, a new erroneous use case information table from 1 to n can be formed. Exemplarily, the erroneous use case information table is shown in Table 2 below. Based on Table 2, it is easy to judge the difficulty of use case debugging and the overall effort and time required.

[0120] Table 2

[0121]

[0122]

[0123] Step S2, sort the use cases according to the error data, generate an error information table and an error regression list. The use cases can be stacked and sorted according to the number of use cases to form a new error use case information table from 1 to n, and automatically generate an error regression list. The error regression list is represented as error.hvp.

[0124] For example, you can use the following script to generate a new error regression list:

[0125]

[0126]

[0127] Step S3, generating regression ID subsets for different IDs according to the erroneous use case ID.

[0128] Specifically, the following script can be used to more accurately generate error regression subsets based on the same ID for different error ID information, so as to more accurately know which type of problem can be fixed.

[0129] The script looks like this:

[0130]

[0131] Among them, the code script for the error regression subset with ID connect is as follows:

[0132]

[0133]

[0134] Among them, the subset of regression lists with error number CONNECT is shown in Table 3 below.

[0135] Table 3

[0136] Use Case Name Seed Error num ERR ID timer1 underflow 1 1 CONNECT timer1 overflow 146287890 2 CONNECT timer2 overflow 23848208 3 CONNECT timer1 zero 12345677 4 CONNECT

[0137] For example, it can also be combined with Figure 4 The following steps are shown to illustrate the regression testing process based on the new regression test sequence:

[0138] Step 1, get the new RTL code and update the test platform.

[0139] Step 2: Generate test cases.

[0140] Step 3: Determine whether the test case passes the test. If yes, go to step 1. Otherwise, go to step 6.

[0141] Step 4, get the wrong test case from the test case.

[0142] Step 5, determine whether the erroneous test case passes the test. If yes, proceed to step 2. Otherwise, proceed to step 6.

[0143] Step 6, obtain the corrected RTL code and update the test platform.

[0144] Based on the above steps, it can be seen that after updating the code or environment, only a small-scale regression of the error regression list subset or the entire error regression list is used to determine whether the modification is correct. If the error is still reported, update again, or modify the use case and update the environment again. If it is correct, you can start the full regression again. In this way, a small-scale regression test can be performed within the regression subset according to the newly modified design code or verification environment code, avoiding the reopening of a large-scale invalid regression test, reasonably allocating time and deploying resources, greatly saving the time of verification engineers, and reducing the cost of using verification tool licenses. In addition, by quickly identifying the wrong use cases, seed numbers, error numbers, and error information IDs, etc., they are sorted to facilitate developers to effectively manage error information. The generated error regression list is also convenient for secondary regression after modification.

[0145] Based on the above embodiments, the present application makes the following improvements to the existing regression method: First, the extraction of the error use case list is automated, and the error use case information is automatically extracted through the existing regression results, and a new error use case regression list is generated. Second, through a changeable configuration, an error use case regression list that meets the requirements is generated, and the regression plan is restarted according to the latest changes. Users can customize the configuration based on the error use case information to generate an error use case regression list that meets the requirements, without the need for additional operations for a single project, with low maintenance costs, greatly reducing the time required for verification convergence, thereby shortening the verification cycle.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] In this embodiment, a regression testing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0148] Figure 5 is a structural block diagram of a regression testing device according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0149] A determination module 52, used to determine the error case of the test error from the historical regression records;

[0150] The obtaining module 54 is used to sort all the error cases to obtain sorted error cases, wherein the error case information of the sorted error cases includes at least one of the following: the case name of the error case, the seed number of the error case, the number of errors of the error case, and the case number of the error case;

[0151] The generating module 56 is used to generate a new regression test sequence using the error case information, and perform regression testing using the new regression test sequence in the verification environment.

[0152] Through the above device, a verification environment is pre-built on the regression test platform, and a regression test module is pre-installed in the regression test platform. The regression test platform is used to control the regression test module to perform regression testing under the verification environment, and the error case of the test error is determined from the historical regression record. Specifically, all error cases are sorted to obtain sorted error cases, wherein the error case information of the sorted error case includes at least one of the following: the case name of the error case, the seed number of the error case, the number of errors of the error case, and the case number of the error case; a new regression test sequence is generated using the error case information, and regression testing is performed using the new regression test sequence under the verification environment. That is, by automatically extracting the error case information such as the case name, seed number, and number of errors from the historical regression records that have been obtained, and then generating a new regression case list (i.e., regression test sequence) according to the error case information, so as to use the new regression case list for regression testing and continuously optimize the regression test design. Therefore, the technical problem of low efficiency of regression testing in the chip verification process in the related art is solved, thereby improving the efficiency of regression testing.

[0153] In an exemplary embodiment, the determination module is also used to: extract log information from the use case simulation log in the historical regression record according to the extraction field, wherein the extraction field includes at least one of the following: the user name of the test case, the use case name of the test case, the seed number of the test case, the number of errors of the test case, the timeout information of the test case, and the use case number of the test case; classify the log information according to the test results of the test case to obtain the correct case of the test correctly and the incorrect case of the test error, wherein the test results include the test correct and the test error.

[0154] In an exemplary embodiment, the obtaining module is further used to: sort all the error cases in ascending order according to the number of errors in the error cases to obtain the sorted error cases; or sort all the error cases in ascending order according to the case number of the test cases to obtain the sorted error cases; or classify all the error cases according to the case number of the test cases to obtain multiple groups of first error cases, wherein each group of first error cases in the multiple groups of first error cases corresponds to a case number; sort the multiple groups of first error cases in ascending order according to the case number of the test cases to obtain the sorted multiple groups of first error cases; for each of the sorted multiple groups of first error cases A first group of error cases, sorting the error cases in each group of first error cases in ascending order according to the number of errors in the error cases, to obtain a group of second error cases corresponding to each group of first error cases; determining the sorted error cases based on multiple groups of second error cases; or, sorting all the error cases in ascending order according to the number of errors in the error cases, to obtain sorted intermediate error cases; classifying the sorted intermediate error cases according to the case number of the test case, to obtain multiple groups of third error cases after classification, wherein each group of third error cases corresponds to a case number; sorting the multiple groups of third error cases in ascending order according to the case number of the test case, to obtain the sorted error cases.

[0155] In an exemplary embodiment, the generation module is also used to: determine the sequence generation conditions pre-set for the target object based on the error use case information; split the sorted error use cases into two groups of use case sequences according to the sequence generation conditions; and when it is determined that the number of error use cases included in the two groups of use case sequences is less than or equal to a preset value, merge the two groups of use case sequences to obtain the new regression test sequence.

[0156] In an exemplary embodiment, the generation module is also used to: determine the debugging priority of the error case using the number of errors in the error case, wherein the more errors there are, the higher the debugging priority; determine the target debugging information corresponding to the error type of the error case from the preset debugging information, wherein the target debugging information at least includes the debugging time and the debugging plan; generate an initial regression test sequence corresponding to the error case using the debugging time and the debugging plan, wherein the debugging level of the initial regression test sequence is a default level; update the debugging level of the initial regression test sequence to the debugging level corresponding to the debugging priority to obtain the new regression test sequence.

[0157] In an exemplary embodiment, the regression testing device also includes a deletion module, which is used to determine multiple error cases with the same use case name from the sorted error cases before using the error case information to generate a new regression test sequence; when it is determined that the seed numbers, error quantities and use case numbers of the multiple error cases are all consistent, retain any one error case from the multiple error cases and delete other error cases.

[0158] In an exemplary embodiment, the regression testing device also includes a sending module, which is used to obtain a regression test result of the regression test using the new regression test sequence after the regression test is performed using the new regression test sequence; use a recursive algorithm to recursively analyze multiple regression test results to obtain common error cases in the multiple regression test results; and determine a repair result of repairing the error cases corresponding to the multiple regression test results; and send a notification message generated based on the common error case and the repair result to a target object.

[0159] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0160] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0161] Optionally, in this embodiment, the computer program may be configured to perform the following steps by means of the computer program:

[0162] S 1, identify the wrong use cases of test errors from historical regression records;

[0163] S2, sorting all the error use cases to obtain sorted error use cases, wherein the error use case information of the sorted error use cases includes at least one of the following: the use case name of the error use case, the seed number of the error use case, the number of errors of the error use case, and the use case number of the error use case;

[0164] S3, using the error case information to generate a new regression test sequence, and performing regression testing using the new regression test sequence in the verification environment.

[0165] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0166] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0167] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0168] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0169] The embodiment of the present application also provides an electronic device, such as Figure 6 As shown, the electronic device includes a memory 502 and a processor 504. The memory 502 stores a computer program, and the processor 504 is configured to execute the steps in any of the above method embodiments through the computer program.

[0170] Optionally, in this embodiment, the processor 504 may be configured to perform the following steps through a computer program:

[0171] S 1, identify the wrong use cases of test errors from historical regression records;

[0172] S2, sorting all the error use cases to obtain sorted error use cases, wherein the error use case information of the sorted error use cases includes at least one of the following: the use case name of the error use case, the seed number of the error use case, the number of errors of the error use case, and the use case number of the error use case;

[0173] S3, using the error case information to generate a new regression test sequence, and performing regression testing using the new regression test sequence in the verification environment.

[0174] Alternatively, a person skilled in the art may understand that: Figure 6The structure shown is for illustration only. Figure 6 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 6 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 6 Different configurations are shown.

[0175] Among them, the memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the regression testing method and device in the embodiment of the present application. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, that is, to implement the above-mentioned regression testing method. The memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include a memory remotely located relative to the processor 504, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 502 may specifically be used, but is not limited to, to store information such as distributed storage system configuration files. As an example, such as Figure 6 As shown, the memory 502 may include but is not limited to all modules in the regression test device. In addition, it may also include but is not limited to other module units in the regression test device, which will not be described in detail in this example.

[0176] Optionally, the transmission device 506 is used to receive or send data via a network. Specific examples of the network may include wired networks and wireless networks. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 506 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0177] In addition, the electronic device further includes: a display 508; and a connection bus 510, which is used to connect various module components in the electronic device.

[0178] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0179] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0180] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0181] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0182] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0183] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A regression testing method, characterized in that: Applied to a regression test platform, the regression test platform is built with a verification environment, the regression test platform is pre-installed with a regression test module, and the regression test platform is used to control the regression test module to perform regression testing under the verification environment. include: Identify test cases that failed from historical regression records; Sorting all the error cases to obtain sorted error cases, wherein the error case information of the sorted error cases includes at least one of the following: a case name of the error case, a seed number of the error case, the number of errors of the error case, and a case number of the error case; A new regression test sequence is generated using the error case information, and regression testing is performed using the new regression test sequence in the verification environment.

2. The method according to claim 1, characterized in that The error cases of the test errors determined from the historical regression records include: Extracting log information from the use case simulation log in the historical regression record according to the extraction field, wherein the extraction field includes at least one of the following: a user name of the test case, a use case name of the test case, a seed number of the test case, a number of errors of the test case, timeout information of the test case, and a use case number of the test case; The log information is classified according to the test results of the test cases to obtain correct test cases with correct tests and incorrect test cases with incorrect tests, wherein the test results include correct tests and incorrect tests.

3. The method according to claim 2, characterized in that Sort all error cases to get sorted error cases, including: Sorting all the error cases in ascending order according to the number of errors in the error cases to obtain the sorted error cases; Alternatively, all the error cases are sorted in ascending order according to the case numbers of the test cases to obtain the sorted error cases; Alternatively, all the error cases are classified according to the case numbers of the test cases to obtain multiple groups of first error cases, wherein each group of first error cases in the multiple groups of first error cases corresponds to a case number; Sorting the multiple groups of first error cases according to the case numbers of the test cases in ascending order to obtain the sorted multiple groups of first error cases; For each group of first error cases in the sorted multiple groups of first error cases, sort the error cases in each group of first error cases in ascending order of the number of errors in the error cases, to obtain a group of second error cases corresponding to each group of first error cases; Determine the sorted error cases based on multiple groups of second error cases; Alternatively, all the error cases are sorted in ascending order according to the number of errors in the error cases to obtain sorted intermediate error cases; Classifying the sorted intermediate error cases according to the case numbers of the test cases to obtain a plurality of groups of third error cases after classification, wherein each group of third error cases corresponds to a case number; The multiple groups of third error cases are sorted in ascending order according to the case numbers of the test cases to obtain the sorted error cases.

4. The method according to claim 1, characterized in that: Generate a new regression test sequence using the error case information, including: Determine a sequence generation condition pre-set by the target object based on the error use case information; Splitting the sorted error use cases into two groups of use case sequences according to the sequence generation condition; When it is determined that the number of erroneous use cases included in the two groups of use case sequences is less than or equal to a preset value, the two groups of use case sequences are merged to obtain the new regression test sequence.

5. The method according to claim 1, characterized in that The error case information includes the number of errors in the error case, and generating a new regression test sequence using the error case information includes: Determine the debugging priority of the error case by using the number of errors of the error case, wherein the greater the number of errors, the higher the debugging priority; Determine target debugging information corresponding to the error type of the error use case from preset debugging information, wherein the target debugging information at least includes debugging time and debugging plan; Generate an initial regression test sequence corresponding to the error case by using the debugging time and the debugging plan, wherein the debugging level of the initial regression test sequence is a default level; The debugging level of the initial regression test sequence is updated to the debugging level corresponding to the debugging priority to obtain the new regression test sequence.

6. The method according to claim 1, characterized in that Before generating a new regression test sequence using the error case information, the method further includes: Determine multiple error use cases with the same use case name from the sorted error use cases; When it is determined that the seed numbers, the number of errors and the case numbers of the multiple error cases are all consistent, any one error case is retained from the multiple error cases and the other error cases are deleted.

7. The method according to claim 1, characterized in that After performing regression testing using the new regression test sequence, the method further includes: Obtaining a regression test result of a regression test performed using the new regression test sequence; Recursively analyzing the multiple regression test results using a recursive algorithm to obtain common error cases in the multiple regression test results; and, determining a repair result for repairing the erroneous use cases corresponding to the plurality of regression test results; A notification message generated based on the common error case and the repair result is sent to a target object.

8. A regression testing device, characterized in that: include: A determination module is used to determine the error cases of test errors from historical regression records; A module is obtained, which is used to sort all the error cases to obtain sorted error cases, wherein the error case information of the sorted error cases includes at least one of the following: the case name of the error case, the seed number of the error case, the number of errors of the error case, and the case number of the error case; A generation module is used to generate a new regression test sequence using the error case information, and perform regression testing using the new regression test sequence in a verification environment.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images