Automatic testing method and device, equipment and storage medium

Abstract

The invention relates to an automatic test method and device, equipment and a storage medium, and the method comprises the steps: obtaining a test script template, and recognizing a parameter tag in the test script template; obtaining a plurality of input test cases, replacing corresponding parameter labels in the test script template with path parameters and data parameters based on parameter categories of the path parameters and the data parameters in the test cases, and generating an executable test script file; distributing an independent file path for each test case; and based on the file path corresponding to the test case, executing the test script file, and generating a plurality of test result files. According to the method and the device, the problem that the test efficiency is low due to the fact that the repetition rate of the test script is high and the use cases cannot be executed in parallel is solved, the flexibility of the test framework is kept, and efficiency and reliability are improved through structural design.

Description

Technical Field

[0001] This application relates to the field of semiconductor testing technology, and in particular to an automated testing method, apparatus, device, and storage medium. Background Technology

[0002] In semiconductor chip development, Design for Testability (DFT) technology is crucial for improving chip testing efficiency. DFT technology enhances test observability and controllability by embedding test structures into the chip design. As process nodes continue to evolve, chip design complexity increases exponentially, making traditional testing methods insufficient for the demands of modern chip development. Currently, the industry widely adopts script-based testing frameworks, but these frameworks suffer from the following problems: In related technologies, test scripts (such as dofile format scripts) and hardware netlists (such as Case.v format netlists) need to be written manually one by one. There is a lot of duplicate code between the two, and they have a many-to-many dependency relationship, which leads to high maintenance costs and low testing efficiency. It also means that when design rules or processes are updated, all related scripts need to be modified synchronously, which can easily introduce human error and make it impossible to quickly adapt to new test scenarios.

[0003] Currently, no effective solution has been proposed to address the problem of low testing efficiency due to high repetition rates in test scripts in related technologies. Summary of the Invention

[0004] This application provides an automated testing method, apparatus, system, device, and storage medium to at least solve the problem of low testing efficiency caused by high repetition rate of test scripts in related technologies.

[0005] In a first aspect, embodiments of this application provide an automated testing method, including: Obtain a preset test script template and identify the parameter tags in the test script template; Multiple input test cases are obtained. Based on the parameter categories of path parameters and data parameters in the test cases, the corresponding parameter tags in the test script template are replaced with the path parameters and data parameters to generate an executable test script file. The preset testing tool is invoked, and the test script file is executed based on the file path corresponding to the test case to generate multiple test result files.

[0006] In some embodiments, the step of replacing the corresponding parameter tags in the test script template with the path parameters and data parameters based on the parameter categories of the path parameters and data parameters in the test case to generate an executable test script file includes: Based on the parameter categories of path parameters and data parameters in the test cases, the path parameters and data parameters are replaced with the corresponding parameter tags in the test script template to generate test parameters; and the test parameters are validated to generate validation results. If the verification result is successful, the test script file is generated based on the test parameters; otherwise, the generation of the test script file is terminated and the test case is marked as failed.

[0007] In some embodiments, after invoking a preset testing tool, executing the test script file based on the file path corresponding to the test case, and generating multiple test result files, the method further includes: Based on the execution process of the test script file, an execution log is generated; The test result file is compared with a pre-stored standard file to generate a content comparison result. Keywords are extracted from the execution log. Based on the content comparison result and the keywords, the test result file is verified and a target verification result is generated. The test result file corresponding to the target verification result is then output.

[0008] In some embodiments, the step of verifying the test result file and generating a target verification result based on the content comparison result and the keywords includes: Based on the content comparison results and the keywords, the test result file is verified and a first verification result is generated; The same test result file is verified by calling a preset standard tool and a second verification result is generated. The target verification result is generated based on the first verification result and the second verification result.

[0009] In some embodiments, the file path includes an iteration identifier, a module identifier, and a test case identifier; before invoking a preset testing tool to execute the test script file based on the file path corresponding to the test case and generate multiple test result files, the method further includes: assigning an independent file path to each test case. Based on each test case, obtain the corresponding iteration cycle number, test module name, and test case number; generate the iteration identifier, module identifier, and test case identifier according to the iteration cycle number, test module name, and test case number. The iteration identifier, the module identifier, and the test case identifier are concatenated sequentially to generate the file paths corresponding to each of the test cases; Copy each of the test cases, the test script files, the corresponding netlist files, and the process library files to the specified file path.

[0010] In some embodiments, obtaining the preset test script template includes: Based on historical test script files, historical test script templates are encapsulated into the base class module; Call the base class module to obtain the test script template.

[0011] In some embodiments, the testing tool includes at least two versions of the toolkit; the step of invoking the preset testing tool and executing the test script file based on the file path corresponding to the test case includes: Based on the version identifier corresponding to the test case, the version toolkit is selected as the current test tool; The current testing tool is invoked to execute the test script file based on the file path corresponding to the test case.

[0012] Secondly, embodiments of this application provide an automated testing apparatus, comprising: The template acquisition module is used to acquire test script templates and identify parameter tags in the test script templates. The script generation module is used to obtain multiple input test cases, and based on the parameter categories of path parameters and data parameters in the test cases, replace the corresponding parameter tags in the test script template with the path parameters and data parameters to generate an executable test script file. The test execution module is used to call a preset test tool, execute the test script file based on the file path corresponding to the test case, and generate multiple test result files.

[0013] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automated testing method described in the first aspect above.

[0014] Fourthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the automated testing method described in the first aspect above.

[0015] Compared to related technologies, the automated testing method, device, and storage medium provided in this application solve the problem of low testing efficiency caused by high test script repetition rate by first obtaining a template and identifying parameter tags, then generating script files by replacing tags based on test case parameters, and finally allocating an independent path and executing the test script files according to the allocated independent path. This maintains the flexibility of the testing framework and achieves a dual improvement in efficiency and reliability through structured design.

[0016] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a hardware structure block diagram of the terminal of the automated testing method according to an embodiment of the present invention; Figure 2 This is a flowchart of an automated testing method according to an embodiment of this application; Figure 3 This is a flowchart of an automated testing method according to a preferred embodiment of this application; Figure 4 This is a flowchart of the script generation process for an automated testing method according to a preferred embodiment of this application; Figure 5 This is a schematic diagram of the file structure of the automated testing method according to a preferred embodiment of this application; Figure 6 This is a flowchart of the execution script of the automated testing method according to a preferred embodiment of this application; Figure 7 This is an automated testing framework diagram of an automated testing method according to a preferred embodiment of this application; Figure 8 This is a flowchart of the verification results of the automated testing method according to a preferred embodiment of this application; Figure 9 This is a structural block diagram of an automated testing apparatus according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0019] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0020] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0021] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the terminal of the automated testing method according to an embodiment of the present invention. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1The different configurations shown.

[0022] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the automated testing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0023] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0024] This embodiment provides an automated testing method. Figure 2 This is a flowchart of an automated testing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain a preset test script template and identify the parameter labels in the test script template.

[0025] In an automated testing framework, the first step is to design or select a generic test script template (i.e., a base dofile). This template contains the basic structure and parameter placeholders (i.e., parameter labels) required to execute tests. The test script template serves as a baseline file to eliminate duplicate code, retaining only common test logic. Pre-defined test script templates (such as base.dofile) contain a fixed command flow (repeated pre- / post-commands) and parameter label placeholders (such as {netlist_path}, {scan_length}). Parameter labels are placeholders marking inputs that need to be dynamically injected (e.g., {netlist_path} corresponds to the netlist path). Parameter labels are displayed in the template in a specific format (e.g., ...). <tag>The `<parameter>` tag is used for subsequent parameter replacement operations. Regular expressions are used to scan the test script template to identify and locate parameter tags. An example test script template is shown below: # Base dofile template source<lib_path> / lib.tcl set_design<netlist_path> / design.v run_test in,<lib_path> and<netlist_path> This refers to parameter labels. This step, through template-based design, reduces the workload of repeatedly writing similar scripts and improves script maintainability.

[0026] Step S202: Obtain multiple input test cases. Based on the parameter categories of path parameters and data parameters in the test cases, replace the corresponding parameter tags in the test script template with the path parameters and data parameters to generate an executable test script file.

[0027] The path parameters include the netlist path (Case.v), the technology library path (Lib), etc.; the path parameters are the paths to the input files (netlist describes the circuit structure, and the technology library defines the design rules); the data parameters include the scan chain length (scan_length), the number of scan chains (scan_count), etc.; the data parameters are variables that control the test behavior (e.g., scan chain length = 100). Multiple test cases are obtained from external sources (such as databases, Excel spreadsheets, or APIs), each containing specific path parameters (e.g., library file path, netlist file path) and data parameters. Each line in the test script template is traversed, and when a parameter label is identified, the corresponding actual parameter value is found in the test case and replaced. After the replacement is complete, a complete, executable test script file (dofile) is generated. For example, a test case with path parameters lib_path= / path / to / libs and netlist_path= / path / to / netlists would have a dofile after the replacement as follows: # Generated dofile source / path / to / libs / lib.tcl set_design / path / to / netlists / design.v run_test This step improves the flexibility and reusability of test scripts by dynamically generating them, while reducing the complexity of script writing.

[0028] Step S203: Invoke the preset testing tool, execute the test script file based on the file path corresponding to the test case, and generate multiple test result files.

[0029] Through the above steps, firstly, addressing the issue of high test script repetition rate, step S201 designs test script templates and identifies parameter tags. This is achieved by using common elements such as core test commands, netlist paths, and library paths as template structures, while reserving differentiated path and data parameters in the form of tags. Step S202 further establishes a parameter replacement mechanism, automatically filling in specific values ​​based on the parameter categories of the input test cases, dynamically generating customized test scripts. This combination of template-based and parameter-based approaches transforms repetitive similar scripts into template reuse and parameter configuration modes, effectively solving the problem of duplicate code and significantly reducing repetitive writing work. Compared to traditional technologies, this application solves the script redundancy problem through "template decoupling + parameter injection." This end-to-end optimization from script generation to execution management maintains the flexibility of the testing framework while achieving a dual improvement in efficiency and reliability through structured design, effectively solving the long-standing inefficiency problem in the semiconductor chip testing field.

[0030] In some other embodiments, the process of executing the above-described test script file may also include the following steps: Each test case is assigned an independent file path, and the testing tool is invoked to execute multiple test scripts in parallel based on the independent file paths using multi-threading technology.

[0031] The design involves creating a file structure where each test case is assigned an independent file path to ensure isolation between test cases. This independent file path can include an iteration identifier, a module identifier, and a test case identifier, used for physical resource isolation. A three-level independent path is generated for each test case (e.g., 202504-basic_chain-case_14), and test script files are executed based on their respective paths. For example, testing tools (such as the DFT tool) can be invoked, and multi-threading techniques (such as the pytest-xdist plugin) can be used to execute multiple test script files in parallel. Each thread independently starts a test tool instance and executes the corresponding test script according to the assigned file path. After test execution, the test results for each test case are saved in its independent file path, forming multiple test result files. For instance, if two test cases are assigned to the paths / results / 202504 / basic_chain / case_14 / and / results / 202504 / advanced_chain / case_25 / respectively, after multi-threaded parallel execution of these two test cases, corresponding test result files are generated in both paths. This step avoids mutual interference between test cases by isolating them, thus improving the stability and reliability of the test. Multi-threaded parallel execution significantly improves test execution efficiency and shortens the overall test cycle. Furthermore, the independent file path design facilitates the management and traceability of test results, improving the maintainability of the test work.

[0032] In related technologies, test cases need to be executed serially and share global resource paths. A complete test of thousands of test cases can take weeks, slowing down iteration progress. Resource contention leads to interference between test cases (such as file overwriting and tool lock conflicts). This application addresses the efficiency bottleneck caused by the inability to execute test cases in parallel by designing independent file paths (such as iteration cycle / module / test case ID) and allocating independent execution space for each test case. Physical isolation ensures that resources do not interfere with each other during multi-threaded execution, and combined with multi-threading technologies such as the pytest-xdist plugin, true parallel testing is achieved. Compared to traditional serial execution methods, this design significantly improves execution speed, while the resource isolation mechanism avoids coupling conflicts between test cases. By adopting "space isolation + multi-threaded scheduling," the obstacle to parallel execution is overcome, thereby significantly improving test stability.

[0033] In some embodiments, based on the parameter categories of path parameters and data parameters in the test cases, the corresponding parameter tags in the test script template are replaced with the path parameters and data parameters to generate an executable test script file, including: Based on the parameter categories of path parameters and data parameters in the test cases, the corresponding parameter tags in the test script template are replaced with the path parameters and data parameters to generate test parameters; and the test parameters are validated to generate validation results. If the verification result is successful, a test script file is generated based on the test parameters; otherwise, the generation of the test script file is terminated and the test case execution is marked as failed.

[0034] The system extracts path parameters (such as library file paths and netlist paths) and data parameters from test cases, and uses tag matching to replace the actual values ​​in the test script template to generate test parameters. For example, the `source` parameter in the test script template...<lib_path> ` / lib.tcl` is replaced with `source / actual / path / lib.tcl`. The system performs a validity check on the generated test parameters, including whether the path exists and whether the parameter format meets the requirements (such as path separators and file extensions). If the validation passes, the validated test parameters are written to the test script template (base dofile) for subsequent execution. If the validation fails, script generation is terminated, the test case is marked as "execution failed," and the error reason is recorded, such as "invalid path." This embodiment avoids manually writing a large number of similar scripts through template design, greatly improving code reusability. Furthermore, the parameter validation mechanism eliminates invalid parameters during the script generation stage, preventing interruptions during execution due to parameter errors, reducing resource waste, and ensuring that the dynamically generated scripts fully meet execution requirements, thus improving the accuracy of test results.

[0035] In some embodiments, after invoking a preset testing tool and executing the test script file in parallel using multiple threads based on the file path corresponding to the test case, generating multiple test result files, the process further includes: Generate execution logs based on the execution process of the test script file; The test result file is compared with the pre-saved standard file to generate a content comparison result. Keywords are extracted from the execution log. Based on the content comparison result and keywords, the test result file is verified and a target verification result is generated. The target verification result is the test result file that has passed.

[0036] During the execution of the test script file (dofile), the system captures the output information of testing tools (such as DRC tools) in real time and generates detailed execution logs. The execution logs record real-time information output by the tools during the execution of the test script, including operation steps, status feedback, and error messages, providing process data for result verification and assisting in troubleshooting. The generated test result file is compared with a pre-saved standard file, using text difference analysis algorithms (such as line-by-line comparison and hash verification) to generate content comparison results. The standard file is a manually verified test result file, serving as the benchmark for subsequent comparisons. Pre-defined keywords (such as "ERROR," "PASS," and "WARNING") are extracted from the execution logs and combined with the content comparison results to determine whether the test passes.

[0037] If the content comparison results are consistent and there are no erroneous keywords in the execution log, the target verification result is marked as "passed". Only the test result file with a target verification result of "passed" is output, filtering out invalid or erroneous data.

[0038] On the other hand, if the comparison is inconsistent or the log contains erroneous keywords (such as "ERROR"), the target verification result is marked as "failed". At this time, a file can also be output. In subsequent processes, the output file can be compared with the correct file, and the differences can be recorded to accurately locate the error type and root cause.

[0039] This embodiment uses dual verification of content comparison and execution log keywords to reduce omissions (such as timing issues) from a single comparison method, improve the accuracy of test results, and quickly locate the cause of failure (such as path errors or tool malfunctions) through comparison results and log keywords, shortening debugging time. Only files that pass verification are output to ensure the reliability of data used in subsequent processes.

[0040] In some embodiments, based on content comparison results and keywords, the test result file is verified and a target verification result is generated, including: Based on the content comparison results and keywords, the test result file is verified and the first verification result is generated; The system invokes a pre-defined standard tool to verify the same test result file and generate a second verification result. Based on the first and second verification results, the target verification result is generated.

[0041] The process involves comparing the test result file line-by-line or structured with a pre-saved standard file to generate a content comparison result (match / inconsistency). Preset keywords (such as "ERROR" and "PASS") are then extracted from the execution log. If no error keywords are found in the log and the content comparison result matches, a "pass" first verification result is generated; otherwise, it is marked as "fail." Further, a pre-set standard tool (such as an industry-leading DRC tool) is invoked to perform a second verification on the same test result file, generating an independent second verification result (pass / fail). The standard tool is an industry-recognized high-accuracy testing tool, serving as the benchmark for the second verification to eliminate the limitations of a single tool and ensure the objectivity of the result. If both the first and second verification results are "pass," the target verification result is "pass," and the test result file is output. If either verification result is "fail," the target verification result is "fail," and the differences are recorded (such as specific fields where the content comparison is inconsistent, and error messages from the standard tool). This embodiment reduces the false positive rate of a single verification method by using dual verification (content + tool), further improving the accuracy of test results. Furthermore, the standard tool can be replaced with tools from different manufacturers or versions to adapt to diverse testing needs. It can also record the reasons for verification failure in detail (such as content differences or tool errors), shortening the time for problem localization.

[0042] In some embodiments, the file path includes an iteration identifier, a module identifier, and a test case identifier; before invoking a preset testing tool to execute the test script file based on the file path corresponding to the test case and generate multiple test result files, the method further includes: assigning an independent file path to each test case. Based on each test case, obtain the corresponding iteration cycle number, test module name, and test case number; generate the iteration identifier, module identifier, and test case identifier according to the iteration cycle number, test module name, and test case number. Sequentially concatenate the iteration identifier, module identifier, and test case identifier to generate the file path corresponding to each test case; copy each test case, test script file, corresponding netlist file, and process library file to the file path.

[0043] The iteration cycle number is obtained from the test case's metadata, indicating the current iteration cycle to which the test belongs (e.g., 202504 represents the 4th iteration in 2025); the test module name is the functional module to which the test case belongs (e.g., basic_chain represents the basic chain test module); and the test case number is a unique ID assigned to each test case (e.g., case_14). The iteration cycle number, module name, and test case number are mapped to the iteration identifier, module identifier, and test case identifier, respectively. For example, the iteration cycle number 202504 is mapped to the iteration identifier 202504; the module name basic_chain is mapped to the module identifier basic_chain; and the test case number case_14 is mapped to the test case identifier case_14.

[0044] Next, the identifiers are concatenated in a fixed order to generate a hierarchical file path. The file path structure can be iteration identifier-module identifier-test case identifier, such as / results / <iteration identifier> / <module identifier> / <test case identifier> / . Finally, all test cases, test script files, and related files are copied to the above file path. The related files mainly include the netlist file representing the chip design under test itself and the process library file providing timing and power consumption information for the process node. In this way, by copying various files and placing them in independent folders according to the generated file path, the netlist in the folder is executed during execution, instead of the original netlist, thus ensuring test case isolation and independent execution.

[0045] In this embodiment, each test case has an independent path, avoiding file overwriting or resource conflicts during parallel execution and ensuring that test results do not interfere with each other. The path contains three levels of information: iteration, module, and test case, which can quickly locate the source of the problem (such as tracing the impact of design changes through iteration identifiers). The structured path facilitates the script to automatically clean up historical data and collect test results, improving automation efficiency. Moreover, when executed in multiple threads, each thread operates on the file independently according to the path, without the need for locking mechanisms, thus improving execution efficiency.

[0046] In some embodiments, obtaining a preset test script template includes: Based on historical test script files, historical test script templates are encapsulated into the base class module; Call the base class module to obtain the test script template.

[0047] The base class module is the parent class containing common test logic and template resources, and subclasses reuse its functionality through inheritance. Historical test script templates are standardized script frameworks abstracted from past tests, containing parameter label placeholders and fixed processes, used to dynamically generate specific test script files. Common logic (such as tool initialization, file path configuration, result verification, etc.) is extracted from historical test scripts, and generic code unrelated to specific test cases is stripped away. The BaseTest base class module is created, encapsulating the common logic into base class methods (such as setup_tool(), generate_report()). Abstract methods (such as execute_test()) are defined, forcing subclasses to implement specific test case logic. Abstract methods are methods defined in the base class but not implemented; subclasses are required to implement them to ensure the completeness of the test case logic. Historical test script templates (such as base_dofile.tcl containing parameter placeholders) are stored as base class attributes or accompanying resource files and associated with base class methods. When creating new test case classes, they inherit from the BaseTest base class, allowing direct calling of its encapsulated methods and templates. Subclasses obtain templates through interfaces provided by the base class (such as `load_template()`), and generate specific scripts by replacing parameters. This embodiment uses encapsulation of common logic, allowing new test case development to focus only on the differentiated parts, reducing repetitive code and script writing. Templates and base class logic are centrally managed, eliminating the need to traverse all test case scripts during modifications. For example, when updating tool initialization logic, only the base class needs modification. Similarly, when adding new test scenarios, only a subclass needs to be created and its specific logic populated, without requiring a complete refactoring of the infrastructure, thus adapting to rapid iteration requirements.

[0048] In some embodiments, the testing tool includes at least two versions of the toolkit; it invokes a preset testing tool and executes test script files in parallel using multiple threads based on the file paths corresponding to the test cases, including: Based on the version identifier corresponding to the test case, select the version toolkit as the current test tool; Invoke the current testing tool and execute the test script file based on the file path corresponding to the test case.

[0049] It's important to note that the testing framework's default testing tools are not single entities, but rather a collection of toolkits containing at least two versions. These different versions of the toolkit refer to different tool forms developed to achieve the same testing objective, differing in architecture and invocation methods, and requiring parallel verification. Since the test case metadata includes version identifiers—tags specifying the required toolkit type and version (e.g., standalone_v2.1)—obtaining the version identifier from the test case metadata allows for the identification of the required toolkit type and version, thus enabling the selection of the appropriate testing tool.

[0050] The following example illustrates the selection process for testing tools, using standalone and integrated versions of the tool package as examples. Standalone tool packages are separately installed testing tools (e.g., ` / opt / drc_tool_v2.1 / bin / drc`), requiring explicit path specification. They are suitable for scenarios with multiple versions coexisting, avoiding version conflicts by loading independent tool instances in each thread. Integrated tool packages are tool libraries embedded in the testing framework (e.g., Python packages installed via `pip install drc_tool`), accessed through API calls. They are suitable for unified version management and isolate tool contexts using thread-local storage (ThreadLocal). If the version identifier includes a standalone version identifier (i.e., this test case is a standalone version), the standalone tool package is selected. If the version identifier includes an integrated version identifier (i.e., this test case is an integrated version), the integrated tool package is selected.

[0051] Thus, during the execution of the test script file, a thread pool (such as ThreadPoolExecutor) is created. The number of threads is dynamically configured based on the number of CPU cores. An execution task is generated for each test case. The task includes the current test tool path (such as standalone / integrated version) selected in the previous steps and the file path of the test script file. The task is then submitted to the thread pool, which automatically schedules and executes it. This embodiment dynamically selects the toolkit through version identifiers, supports the coexistence of multiple versions of tools, avoids test failures due to version conflicts, and can fully utilize the physical characteristics of different versions of toolkits. For example, standalone version toolkits can avoid global installation conflicts, and integrated version toolkits can reduce the overhead of repeated loading, thereby balancing flexibility and performance. In some embodiments, an independent file path can be assigned to each test case, and multi-threaded parallel testing can greatly shorten the testing time. Furthermore, the thread pool automatically manages task distribution and exception handling, reducing manual scheduling costs.

[0052] The embodiments of this application will be described and illustrated below through preferred embodiments.

[0053] A chip design company is conducting iterative testing on the Basic_chain module (basic scan chain function). It needs to verify 100 test cases, covering different scan chain lengths (10-500), scan chain counts (300-500), and process libraries (28nm / 5nm).

[0054] Figure 3 This is a flowchart of an automated testing method according to a preferred embodiment of this application, such as... Figure 3 As shown, this automated testing method includes the following steps: Step S301: Obtain the test script template and identify the parameter labels. The test script template (base.dofile) is as follows: read_netlist {netlist_path} # tag {netlist_path} load_library {library_path} # tag {library_path} Set_scan_cfg -max_length {scan_length} #Tag{scan_length} run_scan_test The tag identification is achieved by scanning the test script template using the framework to locate the tags for the three parameters {netlist_path}, {library_path}, and {scan_length}.

[0055] Step S302: Inject parameters and generate script.

[0056] Test case configuration (case_14): params = { "netlist_path": "case_14 / Case.v", # Netlist path "library_path": "lib / 28nm.lib", # 28nm process library "scan_length": 100 # Scan depth } Inject the parameters into the template to generate a custom test script file, dofile. read_netlist case_14 / Case.v load_library lib / 28nm.lib Set_scan_cfg -max_length 100 run_scan_test More specifically, in combination Figure 4 The script generation process of step S302 above will be described in detail. Figure 4 This is a flowchart of the script generation process for an automated testing method according to a preferred embodiment of this application, such as... Figure 4 As shown, the parameter categories of path parameters and data parameters in the test cases, the test cases to be replaced, and the parameter labels in the test script template are input into the automated replacement engine. The path parameters can include netlist paths (Case.v), process library paths (Lib), etc.; Case.v is the gate-level netlist of the test cases; Lib is the process library file that defines the design rules for the manufacturing process; and the Tcl file is the tool's execution command flow file. Since only a few commands in the dofile need to be considered during testing, while other preconditions and postconditions remain largely unchanged, the core test commands, netlists, and used libraries are extracted and decoupled from the preconditions and postconditions. This allows for the configuration of netlist and library addresses through parameter passing, generating a large number of dofiles in a short time to support test cases. A module replacement approach is used: a base dofile is set up, tagged with location labels. The addresses of test commands, netlists, and library files are replaced with the corresponding labels, and the module and iteration cycle of the test cases are concatenated to form a storage address for the test cases, which is then written into the dofile to create a complete dofile. Based on the version identifier of the test cases, the version toolkit is selected as the current test tool, and the current test is executed, with the execution results stored at that address.

[0057] Next, parameter validation is performed, checking the existence of the Case.v and 28nm.lib files. If successful, the validation continues to verify that scan_length=100 is a positive integer. If successful, a valid test script file, case_14.dofile, is generated. If validation fails (e.g., due to an incorrect path), generation is immediately terminated and the test case is marked as failed. If the parameters are incorrect, the generated dofile will be invalid, and the test case will fail. There is no need to specifically focus on parameter conflicts; only the test case results need to be checked.

[0058] Step S303: Execute the test script file.

[0059] Specifically, each test case can be assigned an independent file path, and the test script file can be executed based on the corresponding file path. Figure 5 This is a schematic diagram of the file structure of the automated testing method according to a preferred embodiment of this application, such as... Figure 5 As shown, strict consistency is ensured through file structure. The generated test cases are stored and run under the iteration-module-case_id level. During multi-threaded execution, an independent tool is launched, executing from the file path within this level. It executes independently, without depending on other test cases, and the generated files are also stored in this path, ensuring test cases remain independent throughout the project. For example, the test case path `insertion_730-basic_chain-case_12` represents the 12th case under the `basic_chain` module of iteration `insertion_730`. When this case is executed, the execution output will be stored in the `result` folder following this path structure. Similarly, the test case path `202504-basic_chain-case_14` means the 14th case under the `basic_chain` module of iteration `202504`, and this case will be executed in this path. The multi-threaded execution discussed in this article utilizes the `pytest-xdist` component of the Python testing framework, which implements a path generation rule of "iteration identifier-module identifier-test case identifier," as follows: iteration_id = "202504" # Iteration identifier (April 2025) module_id = "basic_chain" # Module identifier case_id = "case_14" # Case ID file_path = f"{iteration_id} / {module_id} / {case_id}" # The full path is 202504 / basic_chain / case_14 Next, we will continue to combine... Figure 6 The execution process of the test script file in step S303 above will be explained in detail. Figure 6 This is a flowchart of the execution script of the automated testing method according to a preferred embodiment of this application, such as... Figure 6 As shown, the flowchart includes Figure 4 The process for generating test scripts shown also includes the following steps: To execute the test script file in parallel, pytest-xdist is used to start 10 threads for parallel execution, with each thread running in an independent path: dft_tool execute --path 202504 / basic_chain / case_14 / case_14.dofile.

[0060] Output results file (202504 / basic_chain / case_14 / wagnbiao.json, scan chain connection data) and log file (202504 / basic_chain / case_14 / execution.log).

[0061] The toolkit version is automatically selected. For example, if test case_15 requires testing the integrated version of the toolkit, the integrated toolpath is selected based on the parameters. During execution, based on the passed package name, the corresponding standalone or integrated version of the toolkit is automatically selected to execute the test case. The standalone and integrated versions are two forms of an iteration, and the execution commands will differ slightly. Both versions need to be tested simultaneously.

[0062] To better understand this embodiment, it is combined with... Figure 7 The data formats involved in the aforementioned automated testing are systematically explained. Figure 7 This is an automated testing framework diagram of the automated testing method according to a preferred embodiment of this application, such as... Figure 7 As shown, this embodiment encapsulates the execution and verification methods into a common method `common`, and extracts them into the `base.py` base class, achieving method decoupling. All test scripts call the common execution and verification methods by inheriting from this base class. Furthermore, the automated testing framework in this embodiment also includes the following modules: test item configuration (`config`), test library (`liberty`), framework execution log (`log`), standard file for comparing execution results (`reference`), input file for the current execution result (`result`), execution report (`report`), automated test scripts (`testcases`) (including `insertion_630`, `insertion_730`, `integtation`, and `base.py`), test case data (`testdata`), utility class `untils`, full execution script `run.py`, and global processing hook function `conftest.py`.

[0063] Step S304: Multi-dimensional result verification to determine whether the test case was executed successfully. If not, analyze the reason for the test case execution failure; if yes, indicate that the test case execution was completed.

[0064] Compare wagnbiao.json line by line with the pre-stored standard file reference / basic_chain / case_14_wagnbiao.json. If the number of chain connections is the same, the test passes. Extract keywords from the execution log; if there are no ERROR keywords, the test passes. Combine file content comparison and log keyword analysis to obtain the first verification result. Call the standard tool to generate the second verification result. If the second verification result is FAIL (the main tool did not report this sequence violation), combine the first and second verification results for a comprehensive judgment to generate the target verification result.

[0065] Figure 8 This is a flowchart illustrating the verification results of the automated testing method according to a preferred embodiment of this application, as shown below. Figure 8 As shown, the output after execution consists of the log content after executing `dofile` and the result file after executing `dofile`. Therefore, the output needs to be validated. During test case execution, the first execution result is manually checked. Once verified, the result is stored in the `reference` folder as a standard comparison file. In subsequent development versions, the execution result is compared with the first manual check. If there are functional changes, the `reference` is updated. The execution log, execution result file, and `reference` are then input into an automated validation engine. This automated validation engine ensures the correctness of the test case results by verifying the consistency of error and warning information in the log, the consistency of data in the chained execution result files (such as netlists, tcl, json, and def files), and verifying the correctness of the def output using the `dizo` command.

[0066] This embodiment also provides an automated testing device for implementing the above embodiments and preferred embodiments, which will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0067] Figure 9 This is a structural block diagram of an automated testing apparatus according to an embodiment of this application, such as... Figure 9 As shown, the device includes: Template acquisition module 10 is used to acquire test script templates and identify parameter tags in the test script templates; The script generation module 20 is used to obtain multiple input test cases, and based on the parameter categories of path parameters and data parameters in the test cases, replace the corresponding parameter tags in the test script template with path parameters and data parameters to generate an executable test script file. The test execution module 30 is used to call the preset test tools, execute the test script files in parallel using multiple threads based on the file paths corresponding to the test cases, and generate multiple test result files.

[0068] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0069] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0070] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0071] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1, obtain the preset test script template and identify the parameter labels in the test script template.

[0072] S2: Obtain multiple input test cases, and based on the parameter categories of path parameters and data parameters in the test cases, replace the corresponding parameter tags in the test script template with the path parameters and data parameters to generate an executable test script file; S3 calls the preset testing tool, executes the test script file based on the file path corresponding to the test case, and generates multiple test result files.

[0073] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0074] Furthermore, in conjunction with the automated testing methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the automated testing methods in the above embodiments.

[0075] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.< / tag>

Claims

1. An automated testing method, characterized in that, include: Obtain a preset test script template and identify the parameter tags in the test script template; Multiple input test cases are obtained. Based on the parameter categories of path parameters and data parameters in the test cases, the corresponding parameter tags in the test script template are replaced with the path parameters and data parameters to generate an executable test script file. The preset testing tool is invoked, and the test script file is executed based on the file path corresponding to the test case to generate multiple test result files.

2. The automated testing method according to claim 1, characterized in that, The step of replacing the corresponding parameter tags in the test script template with the path parameters and data parameters based on the parameter categories in the test case to generate an executable test script file includes: Based on the parameter categories of path parameters and data parameters in the test cases, the path parameters and data parameters are replaced with the corresponding parameter tags in the test script template to generate test parameters; and the test parameters are validated to generate validation results. If the verification result is successful, the test script file is generated based on the test parameters; otherwise, the generation of the test script file is terminated and the test case is marked as failed.

3. The automated testing method according to claim 1, characterized in that, After invoking a preset testing tool, executing the test script file based on the file path corresponding to the test case, and generating multiple test result files, the process further includes: Based on the execution process of the test script file, an execution log is generated; The test result file is compared with a pre-stored standard file to generate a content comparison result. Keywords are extracted from the execution log. Based on the content comparison result and the keywords, the test result file is verified and a target verification result is generated. The test result file corresponding to the target verification result is then output.

4. The automated testing method according to claim 3, characterized in that, The process of verifying the test result file and generating a target verification result based on the content comparison results and the keywords includes: Based on the content comparison results and the keywords, the test result file is verified and a first verification result is generated; The same test result file is verified by calling a preset standard tool and a second verification result is generated. The target verification result is generated based on the first verification result and the second verification result.

5. The automated testing method according to claim 1, characterized in that, The file path includes an iteration identifier, a module identifier, and a test case identifier; before invoking the preset testing tool, executing the test script file based on the file path corresponding to the test case, and generating multiple test result files, the method further includes: assigning an independent file path to each test case. Based on each test case, obtain the corresponding iteration cycle number, test module name, and test case number; generate the iteration identifier, module identifier, and test case identifier according to the iteration cycle number, test module name, and test case number. The iteration identifier, the module identifier, and the test case identifier are concatenated sequentially to generate the file paths corresponding to each of the test cases; Copy each of the test cases, the test script files, the corresponding netlist files, and the process library files to the specified file path.

6. The automated testing method according to claim 1, characterized in that, The process of obtaining the preset test script template includes: Obtain historical test script files and historical test script templates; based on the historical test script files, encapsulate the historical test script templates into the base class module; Call the base class module to obtain the test script template.

7. The automated testing method according to claim 1, characterized in that, The testing tool includes at least two versions of the toolkit; the step of invoking the preset testing tool and executing the test script file based on the file path corresponding to the test case includes: Based on the version identifier corresponding to the test case, the version toolkit is selected as the current test tool; The current testing tool is invoked to execute the test script file based on the file path corresponding to the test case.

8. An automated testing device, characterized in that, include: The template acquisition module is used to acquire test script templates and identify parameter tags in the test script templates. The script generation module is used to obtain multiple input test cases, and based on the parameter categories of path parameters and data parameters in the test cases, replace the corresponding parameter tags in the test script template with the path parameters and data parameters to generate an executable test script file. The test execution module is used to call a preset test tool, execute the test script file based on the file path corresponding to the test case, and generate multiple test result files.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the automated testing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the automated testing method according to any one of claims 1 to 7 when it is run.

Images