A regression test method, device, computer equipment and storage medium
By performing batch simulations and real-time monitoring of multiple test cases, and automating the handling of simulation failure cases, the inefficiency of existing regression testing methods is solved, achieving efficient SoC chip design verification and improving the automation and accuracy of testing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 海光信息技术(成都)有限公司
- Filing Date
- 2026-04-13
- Publication Date
- 2026-07-14
AI Technical Summary
Existing regression testing methods are inefficient in the development of ultra-large-scale SoC chips. They cannot effectively monitor and process a large number of test cases, resulting in time-consuming and labor-intensive manual intervention, insufficient utilization of computing resources, delays in the discovery of design defects, and the inability to achieve automated error identification and processing.
By performing batch simulation runs on multiple test cases, monitoring the simulation status in real time, processing failed simulation cases in parallel, automating reruns and repairs, adopting differentiated strategies for different error types, and using automatic restart test case simulation tools for automated monitoring and processing.
It significantly improves the efficiency and reliability of regression testing, shortens the design convergence cycle, reduces manual intervention, enables continuous execution of the simulation process and closed-loop processing of failed test cases, supports multi-round parallel testing, and improves the smoothness and completeness of testing.
Smart Images
Figure CN122389772A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of chip testing technology, specifically to a regression testing method, apparatus, computer equipment, and storage medium. Background Technology
[0002] With the rapid development of integrated circuit technology, the scale and complexity of System-on-a-Chip (SoC) are constantly increasing, and the SoC verification environment is becoming increasingly complex, making verification work more difficult. To ensure the correctness and completeness of chip functionality, SoC regression testing is implemented throughout the entire chip design process. As a crucial step in the chip verification process, its importance is increasingly prominent. Regression testing refers to the method of re-executing existing test cases after software or hardware system modifications to ensure that the modifications have not introduced new defects, while verifying that the original functions still work properly. In the field of chip design verification, regression testing typically requires simulating and running a large number of test cases to verify the correctness and stability of the design.
[0003] In existing technologies, a round of regression testing is first initiated to determine whether all test cases have passed. If all test cases pass, the current round of regression testing ends. If any test cases fail, the verification personnel manually determine the running status of the test cases and the reasons for the failure based on the simulation logs. After the problem is fixed, a new round of regression testing is initiated to verify the effectiveness of the defect fix.
[0004] Figure 1 This is a flowchart illustrating a regression testing method in the prior art, such as... Figure 1 As shown, the method includes: Step 101: Start a round of regression testing; Step 102: Determine whether all test cases have passed; Step 103: If all have passed, end the current round of testing; Step 104: If not all have passed, fix the failed test cases and start a new round of regression testing.
[0005] However, existing regression testing methods still have significant technical shortcomings when facing the development of ultra-large-scale SoC chips. First, when the number of test cases reaches tens of thousands, existing methods struggle to monitor the running status of all test cases in a timely and effective manner. Verification personnel need to manually perform tasks such as test case status monitoring, error analysis, and resubmitting simulations. This manual intervention is not only time-consuming and labor-intensive, but also highly repetitive and prone to human error, seriously affecting the efficiency and reliability of regression testing.
[0006] Secondly, existing regression testing methods cannot fully utilize the computational resources of a single round of regression testing. For simulation failures caused by server malfunctions, insufficient computing resources, or defects in the test cases themselves, the processing efficiency is low, and automated error identification and handling cannot be achieved. This affects the timely discovery of design defects, prolongs the entire design convergence cycle, and reduces chip development efficiency.
[0007] Furthermore, errors caused by issues with the verification environment and test cases need to be identified and fixed as soon as possible to ensure that defects causing mismatches between functionality and design code are discovered earlier. Achieving overall functional coverage more quickly within limited time and computing resources presents a significant challenge for SoC verification personnel. Summary of the Invention
[0008] To address the technical problems of monitoring difficulties, low efficiency of manual operation, and insufficient utilization of computing resources caused by the large number of test cases in the development of ultra-large-scale SoC chips in existing regression testing processes, embodiments of this application provide a regression testing method, apparatus, computer equipment, and storage medium.
[0009] According to a first aspect of the embodiments of this application, a regression testing method is provided, comprising: batch simulating multiple test cases; monitoring the simulation running status of each test case in real time; when a test case fails to run in simulation but other test cases continue to run in simulation, processing the failed test case and re-simulating the processed test case; and completing the regression test after all test cases have successfully simulated. This method can accelerate the convergence speed of regression testing, monitor the simulation running status of each test case in real time during a round of regression testing, and automatically process and re-simulate failed test cases in parallel.
[0010] Preferably, batch simulation runs of multiple test cases include compiling and generating executable files according to the requirements of the multiple test cases; and performing simulation runs of the multiple test cases based on the executable files. This ensures that each test case starts the simulation with an executable file that matches its own requirements, improving the accuracy and stability of the simulation run.
[0011] Preferably, test cases that fail during simulation are processed, and the processed test cases are re-run in simulation alongside other test cases that have not yet completed simulation. If a test case fails during simulation and other test cases continue to run, the error type of the failed test case is identified; the test case is then processed according to the error type; and the processed test case is re-run in simulation while other test cases continue to run. This allows for differentiated processing strategies based on different failure causes, avoiding the resource waste and inefficiency of a one-size-fits-all approach of rerunning.
[0012] Preferably, the error types include server or computing resource anomalies, test case stimulus or checker errors, and waveforms that need to be recorded during the simulation process. This enables accurate classification of failed test cases, providing a clear basis for subsequent automated processing and rapid recovery.
[0013] Preferably, the test cases are processed according to the error type, and the processed test cases are re-run in simulation. This includes: for simulation failures caused by server or computing resource anomalies, directly returning to the simulation phase of the test case and rerunning; for simulation failures due to stimulus or checker errors in the test case, repairing the errors first, then incrementally compiling the stimulus and checker parts, and then returning to the simulation phase of the test case and rerunning; for simulation failures caused by recording waveforms of the simulation process, adding the command option requiring the waveforms, and then returning to the simulation phase of the test case and rerunning. This allows for the most efficient repair and rerun strategies for different error types, significantly reducing invalid compilation and repetitive operations, and significantly improving the recovery speed of failed test cases.
[0014] Preferably, the simulation running status of each test case is monitored in real time. If a test case fails to run in simulation while other test cases continue to run, the failed test case is processed and re-run. This includes running an automatic restart test case simulation tool based on a command. This tool automatically starts monitoring the simulation running status of each test case and, if a test case fails to run in simulation while other test cases continue to run, automatically processes the failed test case and re-runs the simulation. This fully automates the process of manual monitoring, manual judgment, and manual rerunning, reducing human intervention, improving the reliability of regression testing, making fuller use of regression test resources, and more efficiently resolving simulation failures caused by computational resources, test case defects, etc. It also allows for faster discovery of design flaws and shortens the design convergence cycle. A simple command can analyze the current regression test running status and restart test cases with specific error types, greatly improving the efficiency of regression testing, saving manpower, and avoiding basic errors.
[0015] Preferably, the automatic restart test case simulation tool includes: extracting and saving the running status and error information of the test cases based on the simulation logs; extracting the running commands and the storage path of the executable files for each test case from the simulation logs; obtaining files to fix the stimulus or checker errors of the test cases; filtering out a list of test cases that need to be re-simulated and run as needed; determining and saving the running commands for the test cases that need to be re-simulated and run from the command list; running the saved test cases that need to be re-simulated and run in parallel, and recording the updated files and the updated list of test cases to be re-simulated and run. This enables automatic parsing of simulation information, automatic filtering of failed test cases, and parallel rerunning, further improving rerun efficiency and ensuring process traceability.
[0016] According to a second aspect of the embodiments of this application, a regression testing apparatus is provided. The apparatus includes a simulation module, a monitoring module, and a processing module. The simulation module performs batch simulation runs on multiple test cases. The monitoring module monitors the simulation running status of each test case in real time. When the monitoring module detects a test case that fails to run in simulation while other test cases continue to run in simulation, the processing module processes the test case that failed to run in simulation, and the simulation module re-runs the processed test case in simulation. After all test cases successfully simulate, the regression test is completed.
[0017] According to a third aspect of the embodiments of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.
[0018] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described regression testing method.
[0019] According to a fifth aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, implements the above-described regression testing method.
[0020] The beneficial effects of this application are as follows: By monitoring the simulation running status of each test case in real time, and processing the failed test cases while keeping other test cases running in simulation, the convergence speed of regression testing can be accelerated. In a round of regression testing, the simulation running status of each test case can be monitored in real time, and the automatic processing and re-simulation of failed test cases can be performed in parallel. In addition, it can realize the continuous execution of regression tests and the closed-loop processing of failed test cases, avoid the interruption of the simulation process, and improve the smoothness and completeness of the overall regression test. In addition, it supports parallel use of multiple rounds and multiple projects in regression testing without interference. It can be used multiple times in different projects and different versions of regression testing, which greatly improves the efficiency of regression testing.
[0021] Furthermore, based on the requirements of the multiple test cases, an executable file is compiled and generated; the simulation of the multiple test cases is then performed based on the executable file. This ensures that each test case starts the simulation with an executable file that matches its own requirements, improving the accuracy and stability of the simulation.
[0022] Furthermore, test cases that fail during simulation are processed according to the error type, and the processed test cases are re-run in simulation and executed in parallel with other test cases that have not yet completed simulation. This allows for differentiated processing strategies for different failure reasons, avoiding the resource waste and inefficiency of a one-size-fits-all approach of rerunning. Furthermore, the error types include server or computing resource anomalies, test case stimulus or checker errors, and waveforms that need to be recorded during the simulation process. This enables precise classification of failed test cases, providing a clear basis for subsequent automated processing and rapid recovery. Furthermore, the test cases are processed according to the error type, and the processed test cases are re-run in the simulation. This allows for the most efficient repair and rerun strategies for different error types, significantly reducing invalid compilation and repetitive operations, and dramatically improving the recovery speed of failed test cases. Furthermore, the automatic restart test case simulation tool is used to automatically start real-time monitoring of the simulation running status of each test case. When a test case fails to run in simulation, while other test cases continue to run, the tool automatically processes the failed test case and restarts the simulation. This fully automates the process of manual monitoring, manual judgment, and manual rerunning, reducing human intervention, improving the reliability of regression testing, making fuller use of regression test resources, more efficiently resolving simulation failures caused by computational resources, test case defects, etc., discovering design flaws faster, and shortening the design convergence cycle. A simple command can analyze the current regression test running status and restart test cases with specific error types, greatly improving the efficiency of regression testing, saving manpower, and avoiding low-level errors. Furthermore, the test case simulation tool is run to automatically restart, including extracting and saving the running status and error information of the test cases based on the simulation logs. This enables automatic parsing of simulation information, automatic filtering of failed test cases, and parallel reruns, further improving rerun efficiency and ensuring process traceability. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating the regression testing method in the prior art; Figure 2 This is a block diagram of the regression testing method according to an embodiment of this application; Figure 3 This is a flowchart illustrating the regression testing method according to an embodiment of this application; Figure 4This is a schematic diagram of the workflow of the automatic restart test case simulation tool according to an embodiment of this application; Figure 5 This is a schematic diagram of the input and output of the automatic restart test case simulation tool according to an embodiment of this application; Figure 6 This is a block diagram of a regression testing apparatus according to an embodiment of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0026] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.
[0027] The term "and / or" as used in this document includes any or all of the items mentioned.
[0028] The regression testing method provided in this application includes, but is not limited to, a computer as the execution subject.
[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0030] First aspect of the embodiments
[0031] For ultra-large-scale SoC systems, each round of regression testing is time-consuming and consumes significant computing resources such as server memory and hard drive capacity, resulting in very high runtime costs. As SoC designs increase in size, verification environments become increasingly complex, making it difficult to reproduce some test cases that fail during regression simulation locally. Some test cases require even more computing resources for simulation, placing higher demands on the stability and reliability of the cluster servers. With increasing simulation time, test case failures due to server anomalies become more frequent. It is hoped that unreasonable test cases or other test case-related defects can be fixed and verified as soon as possible.
[0032] The existing regression testing process analyzes and fixes failed test cases, and then starts the next round of regression testing after the current round is completed. For SoC verification, this method has the following problems: how to handle test case failures caused by abnormal computing resources; how to demonstrate whether the defects related to the current test case have been correctly fixed; and how to reproduce failed test cases.
[0033] Furthermore, existing regression testing methods have very long iteration cycles, significantly reducing design convergence speed. These problems become even more pronounced in the early stages of regression testing when the design code, verification environment, and test cases are not yet stable.
[0034] To address the aforementioned technical problems, an embodiment of the first aspect of this application provides a regression testing method.
[0035] The regression testing method of this application can be applied to various scenarios where regression testing is used, such as verification and testing scenarios on a SoC system.
[0036] Figure 2 This is a block diagram of the regression testing method according to an embodiment of this application, as shown below. Figure 2 As shown, the method includes: Step 201: Perform batch simulation runs on multiple test cases; Step 202: Monitor the simulation running status of each test case in real time; Step 203: If a test case fails to run during simulation, while other test cases continue to run during simulation, process the test case that failed to run during simulation and rerun the simulation for the processed test case. Step 204: After all test cases have been successfully simulated, complete the regression test.
[0037] This approach accelerates the convergence speed of regression testing, monitors the simulation status of each test case in real time during a round of regression testing, and automatically processes and re-runs simulation failure test cases in parallel. For example, when the regression testing method of this embodiment is applied to ultra-large-scale SoC regression testing, it can significantly improve the convergence speed of ultra-large-scale SoC regression testing. In addition, it enables continuous execution of regression testing and closed-loop processing of failed test cases, avoiding interruption of the simulation process and improving the overall smoothness and completeness of regression testing. In addition, it supports parallel use of multiple rounds and multiple projects in regression testing without interference. It can be used multiple times in different projects and different versions of regression testing, which greatly improves the efficiency of regression testing.
[0038] The following is a detailed explanation of each step of the method. This explanation is intended to provide a better understanding of this application, and not to limit it.
[0039] In step 201, multiple test cases are simulated and run in batches; according to the requirements of the test cases, an executable file is compiled and generated; and the test cases are simulated and run based on the executable file.
[0040] In addition, it ensures that each test case starts the simulation with an executable file that matches its own requirements, thereby improving the accuracy and stability of the simulation.
[0041] In step 202, the simulation running status of each test case is monitored in real time. Throughout the entire simulation process, simulation logs, running parameters, error messages, and other data are continuously collected to determine whether the simulation is executing normally and whether any abnormalities such as interruptions, errors, or timeouts occur. Real-time monitoring allows for the timely detection of failed test cases, providing a basis for rapid subsequent processing and preventing the accumulation of problems.
[0042] In step 203, if a test case fails to run during simulation and other test cases continue to run during simulation, the test case that fails to run during simulation is processed and the processed test case is re-run during simulation. For example, when a test case simulation fails, corresponding repair, adjustment, or rerun operations are performed based on the cause of the failure. After processing, the test case is simulated again until the simulation succeeds. Test cases undergoing re-simulation are executed in parallel with other test cases that have not yet completed their simulations, ensuring that the testing process is not interrupted or skipped, thus improving the completeness and accuracy of regression testing.
[0043] Figure 3 This is a flowchart illustrating the regression testing method according to an embodiment of this application. Figure 3As shown, the process can implement the method in step 203, which includes: identifying the error type of the simulation failure for test cases that fail to run; processing the test cases according to the error type; and re-running the simulation for the processed test cases. Figure 3 As shown, the method includes: Step 301: Identify the error types that caused the simulation to fail; In some embodiments, the error type includes: server or computing resource anomalies; test case stimulus or checker errors; and the need to record waveforms of the simulation process.
[0044] Among them, server or computing resource anomalies refer to operational anomalies caused by insufficient resources, service anomalies, hardware failures, or system scheduling anomalies in the simulation runtime environment, which are not caused by the design or test cases themselves; test case stimulus or checker errors refer to defects in the test cases themselves, including incorrect stimulus signal generation, timing configuration errors, logical errors in expected result judgment, and incorrect checker assertion condition settings, which lead to simulation failure; recording waveforms during the simulation process means that in order to locate the cause of simulation failure, it is necessary to re-execute the simulation and enable waveform recording to obtain complete process information on signal timing, data interaction, and module behavior, which is a necessary operation for debugging and problem localization.
[0045] This enables accurate classification of failed test cases, providing a clear basis for judgment in subsequent automated processing and rapid recovery.
[0046] Step 302: Process the test cases according to the error type; For simulation failures caused by server or computing resource anomalies, the simulation process will be returned directly to the test case's simulation phase (e.g., ...). Figure 2 Step 201) and run it again; If a simulation fails due to a test case stimulus or checker error, first repair the issue, then incrementally compile the stimulus and checker parts, and finally return to the test case simulation phase (e.g., ...). Figure 2 Step 201) and run it again; For simulation failures caused by the need to record waveforms during the simulation process, add the command option that requires the waveform, and then return to the simulation phase of the test case (e.g.) Figure 2 (Step 201) and run it again.
[0047] This enables the most efficient repair and rerun strategies for different error types, significantly reducing invalid compilations and repetitive operations, and dramatically improving the recovery speed of failed test cases.
[0048] Step 303: Rerun the simulation on the processed test cases; Therefore, in this round of regression testing, all problems caused by abnormal computing resources were dealt with in a timely manner, the defects of the test cases themselves were verified as soon as possible, and the test cases that failed in simulation were reproduced.
[0049] Furthermore, for ultra-large-scale SoC regression testing, it can make fuller use of the resources of a single regression test, more efficiently resolve simulation failures caused by computing resources, defects in the test cases themselves, etc., discover design defects faster, and shorten the design convergence cycle.
[0050] To automate the monitoring and handling of failed test cases, the system automatically restarts the test case simulation tool based on commands. This automatically restarted tool monitors the simulation status of each test case, automatically handles failed test cases, and re-runs the simulation. Furthermore, the re-running test cases are executed in parallel with other test cases that have not yet completed their simulations.
[0051] This fully automates the processes of manual monitoring, judgment, and rerunning, reducing human intervention, improving the reliability of regression testing, making fuller use of regression resources, resolving simulation failures caused by computational resources or test case defects more efficiently, discovering design flaws faster, and shortening the design convergence cycle. A simple command can analyze the current regression test status and restart test cases with specific error types, greatly improving the efficiency of regression testing, saving manpower, and avoiding basic errors.
[0052] Figure 4 This is a schematic diagram of the workflow of the automatic restart test case simulation tool according to an embodiment of this application, such as... Figure 4 As shown, the method includes: Step 401: Based on the simulation logs of the test cases, extract and save the runtime status and error information of the test cases. For example, based on the test case simulation logs, extract the runtime status, random seed, failure error information, simulation log path, etc., and store them in the text file case_status.txt and the table file case_status.xlsx. The text file serves as the input file for subsequent processing, while the table file provides a more concise summary of the test case runtime status, making it easier to view the types of errors that have occurred and the runtime status of various test cases. Step 402: Extract the running command and the storage path of the executable file for each test case from the simulation log; for example, extract the running command, random seed, and location of the executable file for each test case from the simulation log and store them in the text file cmd.txt. Step 403: Obtain the files for fixing the stimulus or checker errors of the test cases, and obtain the files for fixing the defects related to the test cases; Step 404: Filter out the list of test cases that need to be simulated again as needed; for example, input one or more error types as needed to filter out the list of test cases that need to be simulated again. Step 405: Determine the execution commands for the test cases that need to be simulated again from the command list and save them; for example, find the execution command for each test case from the text file cmd.txt and save it into the script file rerun.csh; Step 406: Run the saved test cases that need to be simulated again in parallel, and record the updated files and the updated list of test cases to be simulated again; for example, run the test cases that need to be re-executed in parallel and record the updated files and the list of re-run test cases.
[0053] Steps 401 to 406 can be executed in stages as needed, or multiple steps can be executed at once.
[0054] This enables automatic parsing of simulation information, automatic screening of failed test cases, and parallel reruns, further improving rerun efficiency and ensuring process traceability.
[0055] In this way, the improved regression testing method of this application can automatically handle various types of simulation failures, ensuring that all test cases are correctly verified. Regression testing is completed once all test cases have successfully simulated. This method significantly improves the automation and reliability of regression testing, reduces the need for manual intervention, and ensures the accuracy of test results.
[0056] Figure 5 This is a schematic diagram of the input and output of the automatic restart test case simulation tool according to an embodiment of this application, as shown below. Figure 5 As shown, the input and output of this tool are as follows: Input parameters include the regression test run path 501, the name of this round of regression tests 502, the test case list 503, one or more error types that need to be rerun 504, the random seed 505, the waveform level 506, etc. Some of these input parameters are required and some are not, and can be passed according to the need for rerun.
[0057] The output includes a text file case_staus.txt showing the test case running status and a table file (containing a list of test cases, running status, random seed, simulation log directory, error types, etc.) 507, a text file cmd.txt containing the command to run each test case 508, and a script file rerun.csh containing the command to run test cases with specified error types 509, all executed automatically in parallel.
[0058] By building an automatic test case simulation tool and optimizing the regression testing process, it is possible to accelerate the convergence speed for ultra-large-scale SoC regression testing, make fuller use of the resources of a regression round, solve simulation failures caused by computing resources, defects in the test cases themselves, etc. more efficiently, discover design defects faster, and shorten the design convergence cycle. In addition, it can automate the rerun of each round of regression testing. Regression test maintenance personnel only need to run a simple command to analyze the current regression test running status and restart test cases of specific error types, which greatly improves the efficiency of regression testing, saves manpower, and avoids low-level errors. In addition, it supports parallel use of multiple rounds and multiple projects in regression testing without interference. It can be used multiple times in different projects and different versions of regression testing, which greatly improves the efficiency of regression testing.
[0059] Furthermore, for specific types of test cases that absolutely need to be rerun, such as errors caused by machine problems, the tool can be set to run periodically, achieving a completely automatic rerun effect without manual intervention; and the tool can be implemented in different computer programming languages.
[0060] Second aspect of the embodiments
[0061] The second aspect of this application also provides a regression testing apparatus, which corresponds to the regression testing method described in the first aspect of this application. That is, the regression testing apparatus is a regression testing apparatus that implements the regression testing method of the first aspect embodiment. The function of the regression testing apparatus can be referred to the description in the first aspect embodiment, and the repeated content will not be specifically described.
[0062] Figure 6 This is a block diagram of the regression testing apparatus according to an embodiment of this application, such as... Figure 6 As shown, the regression testing apparatus 600 may include a simulation module 601, a monitoring module 602, and a processing module 603. It is worth noting that this figure is exemplary; other types of structures may be used to supplement or replace this structure to achieve testing functions or other functions.
[0063] In one implementation, the simulation module 601 can be configured to: perform batch simulation runs on multiple test cases; for example, compile and generate an executable file according to the requirements of the test cases; and perform simulation runs on the test cases based on the executable file.
[0064] The monitoring module 602 can be configured to monitor the simulation running status of each test case in real time. For example, throughout the entire simulation running process, it continuously collects data such as simulation logs, running parameters, and error messages to determine in real time whether the simulation is executing normally, whether interruptions, errors, or timeouts occur, etc. Through real-time monitoring, test cases that fail to simulate can be identified in a timely manner, providing a basis for rapid subsequent processing and preventing the accumulation of problems.
[0065] The processing module 603 can be configured to process test cases that fail during simulation.
[0066] The simulation module 601 can also be configured as follows: when a test case fails to run in simulation and other test cases continue to run in simulation, the simulation module re-runs the processed test case, and completes the regression test after all test cases have successfully run in simulation.
[0067] For example, when a test case simulation fails, corresponding repair, adjustment, or rerun operations are performed based on the cause of the failure. After processing, the test case is simulated again until the simulation succeeds. Test cases undergoing re-simulation are executed in parallel with other test cases that have not yet completed their simulations, ensuring that the testing process is not interrupted or skipped, thus improving the completeness and accuracy of regression testing.
[0068] In the embodiments of this application, the implementation of the functions of the simulation module 601, the monitoring module 602 and the processing module 603 can refer to the description of the relevant steps in the embodiments of the first aspect of this application, and will not be repeated here.
[0069] By building an automatic test case simulation tool and optimizing the regression testing process, it is possible to accelerate the convergence speed for ultra-large-scale SoC regression testing, make fuller use of the resources of a regression round, solve simulation failures caused by computing resources, defects in the test cases themselves, etc. more efficiently, discover design defects faster, and shorten the design convergence cycle. In addition, it can automate the rerun of each round of regression testing and automatically handle simulation failure test cases in parallel. Regression test maintenance personnel only need to run a simple command to analyze the current regression test running status and restart test cases with specific error types, which greatly improves the efficiency of regression testing, saves manpower, and avoids low-level errors. In addition, it supports parallel use of multiple rounds and projects in regression testing without interference. It can be used multiple times simultaneously in different projects and different versions of regression testing, greatly improving the efficiency of regression testing. Third aspect of the embodiments A third aspect of this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.
[0070] Fourth aspect of the embodiment
[0071] An embodiment of the fourth aspect of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the regression testing method described above.
[0072] Fifth aspect of the embodiment
[0073] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the regression testing method described above.
[0074] Finally, it should be noted that the above description is only a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0075] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.
[0076] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0077] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed in the order shown in the embodiments or drawings or in parallel (e.g., in a parallel processor or multi-threaded processing environment).
[0078] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0081] This application uses specific embodiments to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A regression testing method, characterized in that, The method includes: Perform batch simulation runs on multiple test cases; Real-time monitoring of the simulation running status of each test case; If a test case fails to run during simulation, while other test cases continue to run during simulation, the test case that failed to run during simulation is processed and the processed test case is re-run during simulation. After all test cases have been successfully simulated, regression testing is completed.
2. The method according to claim 1, characterized in that, Batch simulation runs of multiple test cases, including: Based on the requirements of the multiple test cases, compile and generate an executable file; The simulation of the multiple test cases is performed based on the executable file.
3. The method according to claim 1, characterized in that, If a test case fails to run during simulation, but other test cases continue to run, the failed test case is processed and the simulation is re-run on the processed test case, including: If a test case fails to run during simulation, while other test cases continue to run during simulation, the error type of the failed test case is identified. The test cases are processed according to the error type. While other test cases continue to run in the simulation, the processed test cases are run in the simulation again.
4. The method according to claim 3, characterized in that, The error types include: The server or computing resources are malfunctioning; The test case's stimulus or checker is faulty; The waveforms during the simulation process need to be recorded.
5. The method according to claim 4, characterized in that, The test cases are processed according to the error type, and the processed test cases are re-run in simulation, including: If the simulation fails due to server or computing resource anomalies, return directly to the simulation phase of the test case and rerun it; If the simulation run fails due to a test case stimulus or checker error, first repair it, then incrementally compile the stimulus and checker parts, and then return to the simulation phase of the test case and rerun it. If the simulation fails due to the need to record the waveform of the simulation process, add the command option that requires the waveform, then return to the simulation stage of the test case and rerun it.
6. The method according to any one of claims 1 to 5, characterized in that, Real-time monitoring of the simulation execution status of each test case; when a test case fails to run in simulation but other test cases continue to run, processing the failed test case and re-running the simulation on the processed test case, including: The command-based automatic restart test case simulation tool is used to automatically start monitoring the simulation running status of each test case, and when a test case fails to run, while other test cases continue to run, it automatically processes the failed test case and restarts the simulation.
7. The method according to claim 6, characterized in that, Run the test case simulation tool to automatically restart, including: Based on the simulation logs of the test cases, extract and save the running status and error information of the test cases; Extract the running commands and the storage path of the executable file for each test case from the simulation log; Obtain the file that fixes the stimulus or checker error in the test case; Filter out the list of test cases that need to be simulated again as needed; In the command list, identify the command to run the test cases that need to be simulated again and save it; Run the saved test cases that need to be simulated again in parallel, and record the updated files and the updated list of test cases to be simulated again.
8. A regression testing device, characterized in that, The device includes a simulation module, a monitoring module, and a processing module: The simulation module performs batch simulation runs on multiple test cases; The monitoring module monitors the simulation running status of each test case in real time. If the monitoring module detects a test case that fails to run in simulation while other test cases continue to run in simulation, the processing module processes the test case that failed in simulation, and the simulation module re-runs the simulation of the processed test case. After all test cases have successfully run in simulation, the regression test is completed.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the regression testing method according to any one of claims 1 to 7.
11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the regression testing method according to any one of claims 1 to 7.