Python-based algorithm tool automatic testing system and method

Abstract

This invention discloses an automated testing system and method for algorithm tools based on Python. Implemented in Python, the system employs a three-layer architecture, including a script layer, an execution layer, and a test case layer. The script layer interacts with user input and manages and schedules the automated testing process, including parsing input information and outputting automated test results. The execution layer calls the corresponding test case set and testing tools in the test case layer based on the parsing results from the script layer to execute the test cases. The test case layer is a collection of test cases describing different characteristics. In this invention, test cases are automatically generated; the invocation of different algorithms under test, test case sets, and the updating of the association between test case sets and testing tools are all achieved by adjusting configuration files. The three-layer architecture is well-separated, with low coupling, strong scalability, low user threshold, high testing efficiency, and significantly reduced labor costs.

Description

Technical Field

[0001] This invention belongs to the fields of semiconductors and computer software, and specifically relates to an automated testing system and method based on Python algorithm tools. Background Technology

[0002] Field-Programmable Gate Arrays (FPGAs) are a further development of programmable devices such as Programmable Array Logic (PAL) and General Purpose Array Logic (GAL). Emerging as a semi-custom circuit in the field of Application-Specific Integrated Circuits (ASICs), they address the shortcomings of custom circuits while overcoming the limited gate count of earlier programmable devices. They can be field-programmed to achieve the user's desired logic design.

[0003] When designers use FPGAs for early-stage logic verification, if the design is too large for a single FPGA to accommodate and thus cannot meet the designer's needs, the circuit design needs to be broken down into multiple sub-circuits and mapped onto multiple FPGAs. Multiple FPGAs are then used to form a platform for logic verification and simulation. Therefore, designers need to design partitioning algorithms to divide the original circuit design so that multiple FPGAs can be used for logic verification and simulation without affecting the original logic structure of the design.

[0004] To meet the needs of the designers mentioned above, Electronic Design Automation (EDA) objects are required to implement this function. EDA objects play a crucial role in Hardware Description Language (HDL) digital logic design. The HDL (registers) written by the designer needs to be converted into an RTL (Register Transfer Level) circuit design, and related design operations and FPGA resource modeling and estimation are performed through software. Then, the RTL-level design is converted from HDL language to a hypergraph structure, facilitating subsequent partitioning of the original design. After partitioning the hypergraph using an algorithm, the partitioned hypergraph structure is obtained. To verify the quality and performance of the partitioning algorithm, a large number of hypergraph structures of different designs and sizes need to be tested. Only after the partitioning algorithm passes the test can the partitioning results be used for subsequent logic synthesis, placement and routing, bitstream generation, and writing to the FPGA. Therefore, testing the partitioning algorithm is crucial. Thus, how to make the testing of the partitioning algorithm faster, more efficient, and more cost-effective in the face of numerous hypergraph design examples becomes even more important.

[0005] To ensure the correct partitioning of the circuit design, the designed partitioning algorithm needs to be thoroughly tested. Testing should cover a large number of cases with different characteristics and scales to guarantee the stability of the algorithm's functionality, quality, and performance. Current technologies for testing partitioning algorithms mostly rely on manual testing. Manual testing allows for logical judgment of the correctness of each step and result; furthermore, manual testing allows for some leaps in execution steps because the logic is clearly understood, enabling precise problem localization.

[0006] However, manual testing is time-consuming, labor-intensive, and not very accurate. While manual testing can barely handle a small number of cases, it is clearly not the best choice for a large number of cases. Moreover, manual testing has high manpower costs, so it is necessary to consider introducing automated testing.

[0007] Automated testing technology executes scripts, allowing human logic to verify the correct implementation of each step. Test case steps are highly interconnected, unlike the often disjointed nature of manual test cases. Automated testing makes better use of resources, executing test cases even at night, without time constraints. Currently, automated testing technology is continuously developing across various industries, especially in the internet industry, where it has become an essential testing tool and is gradually replacing manual testing. Furthermore, automated testing can be used to ensure the correctness and completeness of core product functions, freeing testers from tedious workloads.

[0008] Existing automation technologies include API automation, web automation, and so on, but most are designed to automate repetitive testing steps for specific functions or scenarios, limiting their applicability and making them difficult to extend to other business areas. Furthermore, traditional automated testing requires personnel to undergo rigorous training and even possess specialized programming skills, resulting in a high barrier to entry. Finally, existing automated testing frameworks are generally highly coupled, with scripts, execution commands, and test cases interdependent; updating one requires changes to all dependencies. Moreover, their scalability is limited; adding a new feature may necessitate modifications to the entire framework, leading to high maintenance costs and requiring dedicated teams for updates and maintenance. Therefore, existing automated testing solutions struggle to meet the current demands for automated testing of segmented algorithms. Summary of the Invention

[0009] To address the aforementioned problems in the existing technology, this invention provides an automated testing system and method for algorithm tools based on Python. The technical problem to be solved by this invention is achieved through the following technical solution:

[0010] In a first aspect, embodiments of the present invention provide an automated testing system for algorithm tools based on Python, implemented using the Python language. The system includes a script layer, an execution layer, and a test case layer, wherein...

[0011] The script layer interacts with user input and manages and schedules the automated testing system's workflow. This includes receiving the input, initiating the automated testing process, parsing the input to obtain results, and passing these results to the execution layer; and calling a result detection tool to process the execution layer's data and output the automated testing results. The script layer contains automated testing scripts. The input information includes the directory of the algorithm tool under test, and the name and directory of the test case set configuration file. The algorithm tool under test includes either the algorithm itself or the software tool under test, including partitioning algorithms in circuit design. Calling different algorithm tools is achieved by adjusting their location configuration. Multiple test cases used for testing the algorithm tool are specified through the test case set configuration file.

[0012] The execution layer is used to call the corresponding test case set and matching test tool in the case layer according to the parsing result, and to complete the test on the algorithm tool to be tested, which is the execution target.

[0013] The test case layer includes multiple test case sets; each test case set describes a type of test function; one test function corresponds to at least one test case; each test case set consists of a test execution configuration file and at least one test case; the test execution configuration file sets the test type representing the test function, the matching test tool, the test tool execution flow description, the target execution target, and the corresponding execution command; the association update between the test case set and the test tool and execution target is achieved by modifying the test execution configuration file in the test case set; the multiple test case sets are formed by automatically generating and classifying several test cases in the test case layer in advance.

[0014] In one embodiment of the present invention, the script layer parses the input information to obtain a parsing result, including:

[0015] Read the input information;

[0016] The input information is parsed using a parser to obtain the directory of the algorithm tool under test, the name and directory of the test case set configuration file;

[0017] The algorithm to be tested is located in the directory of the algorithm to be tested, and its type and version are obtained as the parsing result of the algorithm to be tested.

[0018] Based on the name and directory of the test case set configuration file, each test case set is located, and the test execution configuration file of each test case set is parsed to obtain the test execution configuration file parsing result.

[0019] In one embodiment of the present invention, the process of generating the plurality of test case sets for the partitioning algorithm includes:

[0020] Repeatedly execute the test case generation steps to generate multiple test cases; wherein, the test case generation steps include: automatically generating a hypergraph design file; designing a test constraint file based on the hypergraph design file and the test constraint file; and combining the hypergraph design file and the test constraint file to form a test case;

[0021] The generated test cases are categorized according to the test functions to form different test case sets.

[0022] In one embodiment of the present invention, the automated generation of hypergraph design files includes automatically generating a node list file and a netlist file for test cases;

[0023] The automatic generation process of the node list file includes:

[0024] Step a1: Define a node generation function, where the parameters are the total number of nodes M and the resource ratio of each node P;

[0025] Step a2: Calculate the number of resource nodes N that need to be brought according to N = M * P;

[0026] Step a3: Set the value range of the total node, and randomly generate N resource nodes within that range;

[0027] Step a4: Set the range of node weight changes [a, b];

[0028] Step a5: Iterate through the nodes. If a node is a node with resources, randomly generate a weight value for the node within the range [a, b]. If a node is a node without resources, set the weight value of the node to 0.

[0029] Step a6: Put all the generated nodes and their weight values ​​into a list, and write the list back to a file to obtain a node list file;

[0030] The automatic generation process of the netlist file includes:

[0031] Step b1, define the netlist connection generation function, where the parameters are node file, node percentage q, and number of connections L;

[0032] Step b2: Read all nodes in the node file line by line and store them in a node list;

[0033] Step b3: Randomly select M*q nodes from the node list obtained in step b2 as driving nodes; where M is the total number of nodes.

[0034] Step b4: Remove the driver node from the node list obtained in step b2, randomly select L load nodes and form a connection relationship with the driver node in step b2, and save it to the list file;

[0035] Step b5: Iterate through the list of all driver nodes in step b3 until each driver node has a connection with a load node, then exit the loop.

[0036] Step b6: Write the generated connection list file back to the netlist file.

[0037] In one embodiment of the present invention, the test constraint file includes: FPGA resource constraints, topological constraints between FPGAs, and fixed node constraints;

[0038] The description format of the FPGA resource constraints includes the FPGA number and the upper limit of K types of resource values ​​on the FPGA represented by K groups of numbers after the FPGA number.

[0039] The description format of the topology constraints between the FPGAs includes the FPGA numbers of the two connected FPGAs and the physical connection resource values ​​between them.

[0040] The description format of the fixed node constraint includes the FPGA number and the node number pre-assigned to the FPGA.

[0041] In one embodiment of the present invention, the step of calling the result detection tool to process the result data of the execution layer includes: calling the result detection tool to detect and verify the result data of the execution layer, and performing visualization analysis and statistics;

[0042] The methods used to detect and verify the result data of the execution layer include: file comparison verification, keyword extraction verification, and verification by calling third-party tools; the methods used for keyword extraction verification include: regular expression extraction and line-by-line parsing extraction.

[0043] In one embodiment of the present invention, the system is provided with a running display interface for writing input information and displaying prompts during and after the test.

[0044] In one embodiment of the present invention, the input information is set according to a predetermined parameter format and includes, in sequence: an entry script, an input identifier for the algorithm tool to be tested set with a first preset character, a directory of the algorithm tool to be tested, a test case set configuration file identifier set with a second preset character, a directory of the test case set configuration file, and the name of the test case set configuration file.

[0045] In one embodiment of the present invention, the test case set configuration file is in config format, and the content of each test case set includes the test case set name, test case set directory and test case set description; the execution of the test case set supports multi-level search and is implemented using a depth-first traversal algorithm.

[0046] Secondly, embodiments of the present invention provide an automated testing method for algorithm tools based on Python, implemented using an automated testing system, which includes a script layer, an execution layer, and a test case layer; the method includes:

[0047] The script layer interacts with user input information and manages and schedules the workflow of the automated testing system using scripts. This includes receiving the input information, initiating the automated testing process, parsing the input information to obtain the parsing results, and passing them to the execution layer. The script layer contains automated test scripts. The input information includes the directory of the algorithm tool under test, and the name and directory of the test case set configuration file. The algorithm tool under test includes the algorithm or software tool under test, and the algorithm under test includes partitioning algorithms in circuit design. Calling different algorithm tools under test is achieved by adjusting the position configuration of the algorithm tools under test. Multiple test case sets used for testing the algorithm tools under test are specified through the test case set configuration file.

[0048] Based on the parsing results, the execution layer calls the corresponding test case set and matching test tool in the case layer to perform tests on the algorithm tool under test, which is the execution target. The case layer includes multiple test case sets; each test case set describes a type of test function; one test function corresponds to at least one test case; each test case set consists of a test execution configuration file and at least one test case; the test execution configuration file sets the test type representing the test function, the matching test tool, the test tool execution flow description, the target execution target, and the corresponding execution command; the association update between the test case set and the test tool / target execution target is achieved by modifying the test execution configuration file in the test case set; the multiple test case sets are formed by automatically generating and classifying several test cases in the case layer beforehand.

[0049] The script layer calls the result detection tool to process the result data of the execution layer and outputs automated test results.

[0050] The beneficial effects of this invention are:

[0051] 1) It transforms a large number of repeatable test steps into automated execution, which can be executed with one click without human intervention, thereby improving testing efficiency and reducing labor costs;

[0052] 2) Different types of functions or features of the algorithm tool under test are classified and executed, and test cases can be categorized; multiple test case sets used for testing the algorithm tool under test are specified through the test case set configuration file. The association and update between the test case set and the test tool and execution target are achieved by modifying the test execution configuration file in the test case set. Test cases can be easily added, searched, and deleted, and changes to test cases do not affect changes to other content.

[0053] 3) To test different types or versions of the algorithm tool under test, only the directory of the algorithm tool under test and the relevant configuration file parameters in the input information need to be modified. This avoids the problem of hard coding for a specific requirement, has high scalability, and can be extended to test different types of algorithms or software tools.

[0054] 4) It can automatically perform visual statistical analysis of test results, helping testers evaluate the quality and performance of the algorithm tools under test;

[0055] 5) Implemented using Python, it has good platform portability and can be built on different types of operating system platforms, with wide coverage. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the structure of an automated testing system based on Python algorithm tools provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram illustrating the process of obtaining parsing results from input information in the script layer according to an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram illustrating the decoupling of the script layer and the execution layer in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram illustrating the categorizability of test cases in the test case set of this invention embodiment;

[0060] Figure 5 (a)~ Figure 5 (f) is a schematic diagram illustrating the understanding of DFS directory traversal in the embodiments of the present invention;

[0061] Figure 6 This is a schematic diagram illustrating the decoupling of the execution layer and the case layer in an embodiment of the present invention;

[0062] Figure 7 This is a flowchart illustrating an automated testing method based on Python algorithms provided in an embodiment of the present invention.

[0063] Figure 8 This is a schematic diagram illustrating the specific process of the automated testing method based on Python algorithm tools provided in this embodiment of the invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Python is a programming language that, thanks to its simplicity, readability, and extensibility, has become one of the most popular programming languages. Python provides efficient high-level data structures and enables simple and effective object-oriented programming. Its syntax, dynamic typing, and interpreted language nature make it a popular choice for scripting and rapid application development on most platforms. Furthermore, the development and setup of Python automated testing frameworks offer greater flexibility.

[0066] Therefore, in a first aspect, embodiments of the present invention provide an automated testing system for algorithm tools based on Python, implemented using the Python language. The system includes a script layer, an execution layer, and a test case layer. For example... Figure 1 As shown, this Python-based algorithm tool automated testing system may include:

[0067] The script layer is used for script management and overall scheduling of the automated testing system's processes, and to interact with user input information. This includes receiving input information, starting the automated testing process, parsing the input information to obtain the parsing results, and passing them to the execution layer; as well as calling the result detection tool to process the result data from the execution layer and output the automated testing results.

[0068] The script layer contains automated test scripts; the input information includes the directory of the algorithm tool under test, the name and directory of the test case set configuration file; the algorithm tool under test includes the algorithm under test or the software tool under test, and the algorithm under test includes the partitioning algorithm in circuit design; the invocation of different algorithm tools under test is achieved by adjusting the position configuration of the algorithm tool under test; multiple test cases used for testing the algorithm tool under test are specified through the test case set configuration file.

[0069] The execution layer is used to call the corresponding test case set and matching test tools in the case layer based on the parsing results, and to complete the test of the algorithm tool to be executed.

[0070] The case layer includes multiple test case sets;

[0071] Each test case set describes a type of test function; one test function corresponds to at least one test case; each test case set consists of a test execution configuration file and at least one test case; the test execution configuration file sets the test type representing the test function, the matching test tool, the execution process description of the test tool, the execution target, and the corresponding execution command; the association update between the test case set and the test tool and execution target is achieved by modifying the test execution configuration file in the test case set; multiple test case sets are formed by automatically generating and classifying several test cases in the case layer in advance.

[0072] Figure 1 In this context, m and n are merely illustrative of the total number of test cases and the total number of test cases in the test case sets, respectively. Both values ​​are natural numbers greater than 0, and it is not necessary for each test case set to have the same number of test cases. The following provides a detailed description of the embodiments of this invention.

[0073] This invention relates to an automated testing system for algorithm tools based on Python, hereinafter referred to as "this testing system." This system is designed to automate testing of algorithms or software tools under test, including hypergraph partitioning algorithms. For ease of understanding, these algorithms or software tools under test are collectively referred to as "algorithm tools under test," and in actual testing, these algorithm tools serve as the execution targets. In specific implementations, the algorithm tools under test can be reasonably configured and extended as needed, without specific limitations. In designing this testing system, in addition to considering convenience, speed, and automation during the execution phase, the ease of use of the testing framework and the convenience of organizing and adding test cases are also considered. The former determines the efficiency of a testing system's test execution, while the latter determines the completeness of the test case set. If the steps for adding test cases are cumbersome or require high costs, such as writing unfamiliar scripts, it may directly affect the cost of adding and using test cases. To facilitate understanding of the solution, each layer is described below.

[0074] 1) Script layer

[0075] This testing system adopts a three-layer architecture, with the script layer completely separated from the execution layer and the test case layer. When running this testing system, users only need to focus on locating the algorithm tool under test and the test case configuration file required for testing it. Specifically, the script layer is the implementation part of this testing system, containing automated test scripts and multiple scripts as execution tools. The script layer manages the entire system, specifically managing and scheduling the automated test process, and is responsible for the execution, verification, report statistics, and output of all tests. First, the script layer interacts with the user, receives input information from the user, and parses it to obtain the results. This input information is in string format, including the directory of the algorithm tool under test, and the name and directory of the test case configuration file.

[0076] In one optional implementation, the system is provided with a running display interface for writing input information and displaying prompts during and after the test.

[0077] Users can directly enter information on the running display interface. Furthermore, this testing system can also include an input box on the running display interface to clearly indicate where users should enter information. After pressing Enter or the corresponding confirmation key, the automated test begins. Any prompts during the test can be displayed on the running display interface, and a test completion message will be output to prompt the user to view the automated test results in the corresponding location. For example, after receiving the test completion message, the user can view the automatically generated automated test result file in the directory where the results are stored. This automated test result file can be in Excel or other formats, and users can use appropriate tools to open and analyze the automated test result file.

[0078] The input information is set according to a predetermined parameter format and includes, in sequence: entry script, input identifier of the algorithm tool to be tested set with a first preset character, directory of the algorithm tool to be tested, identifier of the test case set configuration file set with a second preset character, directory of the test case set configuration file, and name of the test case set configuration file.

[0079] For example, if the algorithm to be tested is a partitioning algorithm, the input information could be:

[0080] run_qc_regress.py–exe alg_part= / xxx / xxx / alg_part-set / xxx / xxx / set.conf

[0081] Among them, run_qc_regress.py is the entry script; the entry script is the entry point of the test system in this embodiment of the invention. Here, the user can input the directory where the algorithm tool to be tested is located and the directory and name of the configuration file of the test case set. In addition, the entry script is also used to start the automated test process.

[0082] In this example, "-exe" is the first preset character, indicating that the algorithm tool under test inputs an identifier to specify the execution target. Of course, the first preset character in this embodiment of the invention is not limited to the above-mentioned character and can be set appropriately as needed.

[0083] `part = / xxx / xxx / alg_part` represents the directory of the algorithm tool under test, indicating its address path for location. ` / xxx / xxx / ` is only an example and can be set according to actual needs. Multiple algorithm tools can be specified depending on the specific test settings. For example, for testing partitioning algorithms, the directory after `alg_part` can be specified. If testing other algorithm tools, other characters besides `alg_` can be used to specify the directory; for example, `part = / xxx / xxx / part` can specify only one algorithm tool's directory. If multiple algorithm tools are specified, their directories need to be specified separately.

[0084] -set is a second preset character, representing the test case set configuration file identifier to specify the test case set required when executing the target test. Of course, the second preset character in this embodiment of the invention is not limited to the above characters and can be set appropriately as needed.

[0085] In ` / xxx / xxx / set.conf`, ` / xxx / xxx / ` represents the directory of the test case configuration file. ` / xxx / xxx / ` is just an example and may not be the same as ` / xxx / xxx / ` in the directory of the algorithm tool under test. It can be set as needed to represent the address path of the test case configuration file for location. `set.conf` represents the name of the test case configuration file. The specification of the test cases depends on the testing requirements. For example, if the user wants to test the parsing function of the algorithm tool under test, `-set` will specify the directory of the parsing test cases in the test case configuration file. If the user wants to test the performance of the algorithm tool under test, `-set` will specify the directory of the performance test cases in the test case configuration file. If multiple functions need to be tested simultaneously, the directories of the test cases corresponding to each function will be configured in the test case configuration file. Therefore, the user only needs to add the directory of the test cases to the test case configuration file to complete the specification.

[0086] In one alternative implementation, the test case configuration file and the entry script can be in the same directory. Therefore, it is not necessary to specify the directory of the test case configuration file separately. In this case, the above input information can be simplified to:

[0087] run_qc_regress.py-exe alg_part=part= / xxx / xxx / alg_part-set set.conf

[0088] In one optional implementation, the script layer parses the input information to obtain the parsing result, which may include the following steps:

[0089] ① Read input information;

[0090] ② Use a parser to parse the input information to obtain the directory of the algorithm tool under test, the name and directory of the test case set configuration file;

[0091] Since the input information is described in string form, it needs to be converted into a machine-readable language after reading. Therefore, keyword parsing can be performed using parsers such as configparser.

[0092] Specifically, the input information parameter format can be parsed first. If the input information meets the predetermined parameter format, the parsing is successful, and the directory of the algorithm tool under test and the name and directory of the test case set configuration file are read. If the input information does not meet the predetermined parameter format, the parsing fails. In this case, a prompt message can be output to inform the user that the parameter format is incorrect and to write the input information according to the standard predetermined parameter format.

[0093] ③ Locate the algorithm tool to be tested in the directory of the algorithm tool to be tested, and obtain the type and version of the algorithm tool to be tested as the parsing result of the algorithm tool to be tested;

[0094] The specific parsing process for this step is as follows: If the directory of the algorithm tool to be tested is read from the input information, then the algorithm tool to be tested is searched according to the directory. If it can be found, the found algorithm tool to be tested is further parsed to obtain its type and version. The type of the algorithm tool to be tested indicates whether it is an algorithm to be tested or a software tool to be tested, and further, it specifies which type of algorithm to be tested or which type of software tool to be tested. Different versions represent updates to the same algorithm tool to be tested. In this embodiment of the invention, version numbers such as V1, V2, etc., can be used, with larger numbers indicating newer versions.

[0095] If the algorithm tool to be tested is found but its type and version cannot be parsed, a message can be output indicating that the configuration is incorrect and needs to be reconfigured. If the algorithm tool to be tested cannot be found, a message can be output indicating that the algorithm tool to be tested cannot be found and that the user should check and verify the input information.

[0096] ④ Based on the name and directory of the test case set configuration file, locate each test case set, parse the test execution configuration file of each test case set, and obtain the test execution configuration file parsing result.

[0097] The specific parsing process for this step is as follows: If the name and directory of the test case set configuration file are read from the input information, the test case set is searched according to the directory and name of the test case set configuration file. If it is found, the test execution configuration file of the found test case set is further parsed. From the various contents of the parsed test execution configuration file, such as the test type, the parsing result of the test execution configuration file is obtained. If the test case set is found but the contents of the test execution configuration file cannot be parsed, a prompt message can be output to inform the user that the configuration is incorrect and needs to be reconfigured. If the test case set cannot be found, a prompt message can be output to inform the user that the test case set cannot be found and to check the input information.

[0098] Steps ③ and ④ above can be executed in parallel or sequentially, both of which are reasonable. Prompt messages can be displayed on the running interface and can be further integrated with voice prompts, etc.

[0099] In this embodiment of the invention, the script layer passes the parsing results of the algorithm tool under test and the parsing results of the test execution configuration file as the final parsing results to the execution layer.

[0100] For details on the process of parsing input information at the script layer to obtain the parsing result, please refer to [link / reference]. Figure 2 Understanding the process, among which, Figure 2 The flowchart above provides a simplified illustration of the above content, and will not be explained in detail here.

[0101] In this embodiment of the invention, the script layer manages the entire automated testing system. When a specific algorithm tool needs to be tested, the script layer calls it through the directory of the algorithm tool in the input information. In other words, calling different algorithm tools is achieved by adjusting their directories, i.e., adjusting their location configurations. To test different algorithm tools, only the directory of the algorithm tool in the input information needs to be modified; there is no need to rewrite different calling scripts for different algorithm tools. Therefore, the script layer and execution layer can be separated and decoupled. For more details on this functionality, please refer to... Figure 3Understand, where V1 and V2 represent versions of the algorithm under test.

[0102] In this embodiment of the invention, multiple test case sets used for testing the algorithm tool under test are specified through a test case set configuration file. If the test case sets used for testing the algorithm tool under test need to be modified, only the name and directory of the test case sets in the test case set configuration file need to be modified to complete the specification again.

[0103] In one optional implementation, the test case configuration file is in config format, and the content of each test case includes the test case name, test case directory, and test case description.

[0104] Specifically, in the test case configuration file, each line represents a test case or a test case set to be executed. To facilitate understanding of the test case configuration file's content, the following is an example of a test case configuration file, set.conf, specifying two test cases: tdmtest and looptest. It should be noted that the following example of the set.conf format is merely one specific example. The characters used in the test case name, test case directory, and test case description, such as tdm test and loop test, can be modified or set as needed and are not limited to the character format shown in the example. That is, this example is not the only form of the set.conf format.

[0105]

[0106]

[0107] The test case set description explains the specific test case set, such as what function it tests and its characteristics. This description allows unfamiliar users to easily understand the function being tested when performing automated testing. For example, if functional testing is required, the user can quickly identify the corresponding test cases based on the description and understand how to use them. This lowers the barrier to entry and makes it easy for third-party personnel to quickly get started. This section can be understood in conjunction with the subsequent explanation of the test case layer.

[0108] Optionally, when a test case set is not needed for the time being, characters such as "#" can be used to comment out the test case set configuration file. When the test system parses the test case set configuration file, it will not execute the test case set when it encounters "#".

[0109] After the script layer passes the parsed results to the execution layer, the execution layer will obtain the result data after completing the automated test. The script layer will then call the result detection tool to process the result data from the execution layer and output the automated test results.

[0110] This includes calling the result detection tool to process the result data of the execution layer, including calling the result detection tool to detect and verify the result data of the execution layer, and performing visualization analysis and statistics.

[0111] Automated testing generates a series of files, including log files and result files. Result verification tools are then used to check and validate the execution layer's results, primarily by comparing the results of each test case with the pre-designed expected results to verify whether the test results meet expectations. Visual analysis and statistics involve extracting, analyzing, and visualizing key information from the result data to help testers evaluate the quality and performance of the testing algorithm.

[0112] The result verification tool checks the execution layer's results by evaluating each test case. For example, if an automated test case generates three files, and this is the expected result, the test case passes. If it generates only two files, the test case fails. The error message can be displayed as: "The number of files does not match the expected number of files. The expected number is three files, but the actual number is two." This allows users to quickly identify the cause of the error: one file is missing.

[0113] The methods used to detect and verify the results data of the execution layer include: file comparison verification, keyword extraction verification, and verification by calling third-party tools.

[0114] Specifically, the first type is document comparison verification, which means that after a test case is executed, the result data of the test case is compared with the expected result designed in the case. If the comparison results are consistent, the verification is correct and the test case is executed successfully; otherwise, it fails.

[0115] The second method is keyword extraction verification, which involves verifying the correctness of the result by extracting keywords from the result data after the test case is executed. These keywords represent the information relevant to the user. For example, for a partitioning algorithm, the following can be extracted: partition size, algorithm score, number of FPGAs used, number of violations during partitioning, and maximum TDM ratio. This facilitates user analysis and judgment of the partitioning algorithm tool and allows for comparison with similar algorithms. Keyword extraction verification employs methods including regular expression extraction and line-by-line parsing extraction. This embodiment of the invention combines these two keyword extraction verification methods.

[0116] There are several ways to capture keywords, such as (1) whether the keyword exists. For example, in the test result log, if the keyword "Finish Successfully" exists, it means that the automated test was executed successfully. If the keyword "Finish Successfully" does not exist, it means that the automated test was not executed successfully. You can judge whether the automated execution was successful based on whether the keyword "Finish Successfully" exists. (2) the number of times the keyword appears. For example, after the automated test for the partitioning algorithm is completed, the cutsize will appear twice in the output. There will be one cutsize for the initial partition and one cutsize for the iterative improvement. The appearance of two cutsizes means that the algorithm has been executed completely. If one cutsize is missing, it means that part of the algorithm was not executed. Therefore, you can also judge the result based on the number of times the keyword appears. (3) the value of the keyword. For example, in the automated test result for the partitioning algorithm, the cutsize value of the iterative improvement is always less than or equal to the cutsize value of the initial partition. When the cutsize value of the iterative improvement stage is greater than the cutsize value of the initial partition stage in the automated test result, it means that the algorithm has a problem and the partitioning result is incorrect. This is the process of judging the result by the value of the keyword.

[0117] The characteristics of regular expressions in regular expression extraction are: (1) whether the given string conforms to the filtering logic of the regular expression (called "matching"); (2) the desired specific part can be obtained from the string through regular expressions. For example, the style of the resulting text can be as follows:

[0118] ------Finish Successfully------

[0119] Cutsize: 903

[0120] Runtime: 0.047ms

[0121] Memory peak: 4.898MB

[0122] When extracting cutsize, runtime, and memory peaks, the following regular expression can be written:

[0123] Cutsize = re.match("[\s\S]+Cutsize:(\d+)[\s\S]+",context) # Regular expression to parse the cutsize value

[0124] Runtime = re.match("[\s\S]+Runtime:([\d\.]+ms)[\s\S]+",context) # Regular expression to parse runtime value

[0125] Memory_peak = re.match("[\s\S]+Memory peak:([\d\.]+MB)[\s\S]+",context) # Regular expression to parse the memory peak value

[0126] Where Cutsize represents the partitioning situation; Runtime represents the running time; and Memory peak represents the peak memory usage during the partitioning algorithm's execution. The regular expression is the matching regular expression.

[0127] The desired content can be extracted using regular expressions. Generally, once the testing system is stable and users are clear about the content they care about for different algorithms or software tools, the result detection tool can be developed. After development, the result detection tool is basically stable. If there is any subsequent expansion, it can be added to the result detection tool according to the regular expression rules, which is quite convenient.

[0128] For developers unfamiliar with regular expression rules, line-by-line parsing extraction can be used. This method reads the file line by line, extracts keywords, and sets a flag; when a condition is met, the current line is read. For example, to extract cutsize, runtime, and memory peak from the text above using line-by-line parsing extraction, the steps are as follows:

[0129] 1. Read the contents of the result file;

[0130] 2. For the current line being read, if the keyword A is matched in the line, mark the line as 1 and continue reading the next line;

[0131] 3. If the keyword B is matched in the next line, the value of B is parsed and saved into a Python dictionary;

[0132] 4. Continue reading the next line, searching for other keywords C. If found, parse and save; otherwise, continue reading the next line until the entire text is read. After finishing reading, reset the flag to 0 and return the obtained keyword content.

[0133] A, B, and C are only examples of keywords.

[0134] Furthermore, the obtained keywords can be transformed into visual output results.

[0135] The corresponding pseudocode is as follows:

[0136]

[0137]

[0138] The final visualization analysis and statistical format obtained for the partitioning algorithm can be shown in the table below:

[0139]

[0140] Wherein, Case_name represents the test case name; Result represents the test result; Pass indicates that the test case result meets the expected result and the test verification is passed; Fail indicates that the test case failed, and the reason for the failure is described after Fail; Cutsize represents the partitioning result obtained by the partitioning algorithm in executing the test case, and the smaller the value, the better; Vio represents the number of violations that occurred during the partitioning process, and the smaller the value, the better; Tdm represents the maximum TDM ratio of the partitioning, and the smaller the value, the better; Fpga_num represents the number of FPGAs used for partitioning; Run_time represents the time consumed by the partitioning algorithm in executing the test case; Score represents the score after weighted comprehensive analysis of cutsize, vio, and tdm through the objective optimization function. This indicator can be used as a reference to measure the quality of similar algorithms, and the smaller the value, the better.

[0141] The above indicators reflect the characteristics of the partitioning algorithm. Users can use these results to comprehensively analyze and evaluate the advantages and disadvantages of the partitioning algorithm.

[0142] The third method is to use third-party tools for verification. This means that the execution result needs to be verified by a third-party tool to confirm its correctness. Whether the test case passes or fails depends entirely on the result of the third-party tool. Depending on the verification needs, this testing system can integrate more verification methods for expansion; no specific limitations are set here.

[0143] 2) Execution layer

[0144] The execution layer comprises the specific actions / commands executed by this testing system, mainly including the execution flow, execution target, and execution command. Based on the parsing results, it calls the corresponding test case set and matching testing tools in the case layer to complete the test on the execution target and obtain the result data.

[0145] Specifically, the test execution configuration file in the parsed results contains the test type, matching test tools, and a description of the test tool execution flow. This content serves as the test configuration, specifying which test tools will be used for the execution target, which specific test functions will be performed, and the detailed test method flow. The execution layer then uses this information to complete automated testing.

[0146] 3) Case layer

[0147] In this embodiment of the invention, the test case layer is a collection of all test case sets, each describing different characteristics. The test case set is designed only for specific test functions, without depending on the algorithm tool under test. Each test case set describes a type of test function that can be executed and verified. For example, various test functions for a partitioning algorithm could include algorithm parsing and algorithm segmentation; similarly, for a software tool like WeChat, its test functions could include text and voice functions. One test function corresponds to at least one test case. That is, a test case set belonging to the same type of test function contains at least one test case. If it contains multiple test cases, the overall test function of these multiple test cases in that test case set is the same, but the test direction or test purpose of each test case differs. For example, for a partitioning algorithm, if you want to test the segmentation method function, you can design test cases for the minimum cut method, interconnection constraint method, interconnection equilibrium method, forced minimum cut method, and abnormal segmentation method, etc., but these test cases as a whole belong to a test set that tests the segmentation method. Each test case set consists of a test execution configuration file and at least one test case. The test execution configuration file expresses the association between the test case set, the algorithm tool under test, and the testing tool. It also describes the execution process of the test cases in the test case set and is the only standard file for the script layer to obtain test cases and the execution process. The structure of a test case set can be found in [link to documentation]. Figure 1 We need to understand parts of each test case set.

[0148] To achieve thorough testing, different test case sets are needed to test different algorithms and implement corresponding testing functions. This invention automatically generates relevant test case sets in advance based on the different functions to be tested.

[0149] The following section uses a partitioning algorithm as an example to illustrate the process of generating a test case set.

[0150] Specifically, regarding the partitioning algorithm, the process of generating multiple test case sets includes:

[0151] Repeat the test case generation steps to generate multiple test cases; the test case generation steps include:

[0152] The generated test cases are categorized according to the test functions to form different test case sets.

[0153] This invention automatically generates test cases and categorizes them to form a test case set.

[0154] The specific steps can be detailed as follows:

[0155] Step 1: Automated generation of hypergraph design files; the hypergraph design files are automatically designed and generated according to test requirements; the automated generation of hypergraph design files includes the automatic generation of node list files and netlist files for test cases;

[0156] Step 2: Based on the hypergraph design file generated in Step 1, design the test constraint file in combination with the test characteristic requirements;

[0157] Step 3: Combine the hypergraph design file generated in Step 1 and the test constraint file generated in Step 2 to form a test case. Specifically, each test case of the partitioning algorithm mainly consists of a hypergraph design file and a design constraint file. Different combinations of hypergraph design files and test constraint files form different test case characteristics.

[0158] Step 4: Repeat steps 1 to 3 to generate multiple test cases;

[0159] Step 5: Classify the generated test cases according to their test functions to form different test case sets, which serve as the case input part of the automated testing system of this invention embodiment for system call and use.

[0160] Those skilled in the art will understand that the input to the partitioning algorithm is primarily an RTL (Register Transfer Level) circuit description file (i.e., a hypergraph design file) and a test constraint file. The hypergraph design file includes a node list file and a netlist file.

[0161] For the node list file, the number of nodes, the range of resources each node carries, and the percentage of resources each node carries can be flexibly configured according to the test scale requirements. For the netlist file, the connections between nodes and the number of peers each node connects to can be configured, such as 10% of nodes having no more than N1 connections to their peers, and 30% having no more than N2 connections, where N1 and N2 are natural numbers greater than 0.

[0162] For example, automatically generating a node list file with M nodes, K resource dimensions per node, weight values ​​ranging from [a, b], and a resource ratio of p per node. The automatic generation process of the node list file includes:

[0163] Step a1: Define a node generation function, where the parameters are the total number of nodes M and the resource ratio of each node P;

[0164] Step a2: Calculate the number of resource nodes N that need to be brought according to N = M * P;

[0165] Step a3: Set the value range of the total node, and randomly generate N resource nodes within that range;

[0166] Step a4: Set the range of node weight changes [a, b];

[0167] Step a5: Iterate through the nodes. If a node is a node with resources, randomly generate a weight value for the node within the range [a, b]. If a node is a node without resources, set the weight value of the node to 0.

[0168] Step a6: Put all the generated nodes and their weight values ​​into a list, and write the list back to a file to obtain a node list file.

[0169] The corresponding pseudocode is shown below:

[0170]

[0171]

[0172] Here, node_num and node_perce represent the total number of nodes and the proportion of resources a node carries, respectively.

[0173] The automatic generation process of the netlist file includes:

[0174] Step b1, define the netlist connection generation function, where the parameters are node file, node percentage q, and number of connections L;

[0175] Step b2: Read all nodes in the node file line by line and store them in the node list;

[0176] Step b3: Randomly select M*q nodes from the node list obtained in step b2 as driving nodes; where M is the total number of nodes.

[0177] Step b4: Remove the driver node from the node list obtained in step b2, randomly select L load nodes and form a connection relationship with the driver node in step b2, and save it to the list file;

[0178] Step b5: Iterate through the list of all driver nodes in step b3 until each driver node has a connection with a load node, then exit the loop.

[0179] Step b6: Write the generated connection list file back to the netlist file as the netlist input file for the test cases.

[0180] Among them, M, N, a, b, p, q, L, and K are all natural numbers greater than 0.

[0181] The automatic generation process of netlist files will not be illustrated with code examples. As can be seen from the above, the node and netlist description sections of the test cases can be generated automatically. However, a complete test case also includes a test constraint file. This test constraint file includes: FPGA resource constraints, topology constraints between FPGAs, and fixed node constraints; of course, the types of test constraint files are not limited to those described above and can be customized according to the user's needs.

[0182] In this invention, FPGA resource constraints refer to the maximum upper limit of resource values ​​on each FPGA. The partitioning algorithm must segment the input graph nodes according to the grouped resource information, and the segmentation output cannot exceed the resource weight value of each group. In this embodiment of the invention, the description format of FPGA resource constraints includes the FPGA number and the upper limit of K types of resource values ​​on the FPGA represented by K groups of numbers after the FPGA number. In this embodiment of the invention, K represents the resource dimension of the node and the resource dimension of the FPGA, which are consistent.

[0183] Topology constraints between FPGAs refer to whether there are physical connections between them. If there are physical connections, the two FPGAs can communicate with each other; if there are no physical connections, they cannot communicate with each other. Nodes with interconnected relationships in the partitioning result cannot be assigned to two FPGAs that are not interconnected, otherwise a violation will occur. The description format of topology constraints between FPGAs includes the FPGA numbers of the two connected FPGAs and the value of the physical connection resources between them.

[0184] Fixed node constraints refer to predefined node assignment values. Each row contains pre-assigned information for one or more nodes. The partitioning algorithm must segment the nodes in the graph according to this pre-assigned information, and the output must satisfy all pre-assigned node grouping information. The description format of fixed node constraints includes the FPGA number and the node numbers pre-assigned to that FPGA.

[0185] For ease of understanding, an example of a test constraint file format is given below.

[0186]

[0187] Taking K=10 as an example, in the FPGA resource constraints, P1 to P4 represent the FPGA numbers, namely FPGA1 to FPGA4. Taking row P1 as an example, 998, 200, 39286, 491, 6138, 9648, 4824, 259, 43, and 0 represent the upper limits of ten resource values ​​on FPGA1.

[0188] In the topology constraints between FPGAs, taking the first row as an example, T12 represents the topology constraint between FPGA1 and FPGA2, P1 and P2 represent FPGA1 and FPGA2 respectively, and 100 represents the physical connection resource value between the two.

[0189] In the fixed node constraint, taking the first row as an example, F1 represents the fixed node constraint for FPGA1, P1 represents FPGA1, and g1 and g2 represent the node numbers pre-assigned to FPGA1.

[0190] The above describes the components of a complete test case. Different test cases can be grouped into a test case set according to a certain classification method, such as testing functions. Test cases can be categorized by set.

[0191] Each algorithm tool under test integrates multiple functions, and each function may be divided into many test points with different dimensions during testing. Test cases can be classified based on function or test point. For example, the functions of an algorithm can be divided into algorithm parsing, algorithm segmentation, abnormal input, and result verification. Each function can be further subdivided into multiple test points. For example, algorithm parsing can be divided into smaller test points such as node parsing, netlist parsing, and constraint parsing. Test cases divided into different categories are called a test case set. The test execution of a test case set should have the same behavior. Having the same behavior means that the execution process and commands required by the test cases under the test case set are the same. In this way, a general test execution configuration file can be used to describe the test execution of the test case set, without strong correlation with specific test cases. The biggest advantage of separating the test case set from the test execution configuration file representing the specific test execution is that when developers or testers add test cases, they only need to add the input files required by the test cases to the test case set in a certain format. That is, test cases with the same test function can be placed in the same folder, without needing to add execution commands separately. The diagrammatic structure of separating test cases and test execution in the test case set is as follows. Figure 4 As shown, Figure 4 In the example, the test case set for the partitioning algorithm can be divided into several categories, such as the algorithm parsing test case set, the algorithm splitting test case set, the result verification test case set, and the algorithm performance test case set. Each category of test case set can contain multiple test case sets as its subsets. Figure 4Only one optional example is given to facilitate understanding.

[0192] In this embodiment of the invention, the execution of test cases supports multi-level searching, implemented using a depth-first search (DFS) algorithm. That is, when multiple test cases at the same level or multiple subdirectories at a certain level need to be executed, only the top-level directory of all test cases needs to be specified. The DFS algorithm will perform a multi-level search until the lowest-level directory. The specific description of the access algorithm is as follows: Starting from the initial access node (i.e., the top-level directory), the initial access node may have multiple adjacent nodes, i.e., multiple directories. The DFS strategy is to first visit the first adjacent node (directory), and then use this visited adjacent node as the initial node to visit its first adjacent node. Each time, after visiting the current node, the first adjacent node of the current node is visited first. As follows: Figure 5 (a)~ Figure 5 As shown in (f). The numbers inside the circles represent node numbers. Specifically, the traversal starts from the root node 1. Its adjacent nodes are 2, 3, and 4. First, node 2 is traversed, then its child node 5 is traversed, and then the child node 9 of node 5 is traversed, as shown in (f). Figure 5 As shown in (b). After reaching node 9, there are no more nodes to traverse. At this point, we backtrack from node 9 to the previous node 5, checking if node 5 contains any nodes other than node 9. If not, we continue backtracking to node 2. Since there are also no nodes other than node 5, we backtrack to node 1. We find that there is a node 3 other than node 2, so we begin a depth-first traversal from node 3, as follows... Figure 5 As shown in (c). Similarly, starting from node 10 and backtracking upwards to node 6, node 6 has no child nodes other than node 10. Further backtracking, we find that node 3 has a child node 7 other than node 6. Therefore, we will traverse node 7 at this point, as shown... Figure 5 As shown in (d), traversing back from node 7 to node 3 and then node 1, we find that node 1 still has a child node 4 that has not been traversed. Therefore, we proceed along nodes 4 and 8, as follows: Figure 5 As shown in (e), the traversal is now complete. For the complete node traversal order, please refer to [reference needed]. Figure 5 The order of the numbers outside the circle in (f) is [1 2 5 6 3 6 10 7 4 8].

[0193] The design of test cases is independent of the specific algorithm tool under test. When there is a dependency between the test cases and the algorithm tool under test, it is implemented by placing them in the test execution configuration file as configuration parameters. The relationship between the algorithm tool under test and the test case set can be many-to-many, which only needs to be specified through the test execution configuration file of the test case set. For example, for the algorithm under test V1, the test case set used is test case set 1 and test case set 2. When testing the algorithm under test V2, test case set 1 and test case set 2 can also be used. However, the execution commands and processes of the two algorithms under test can be different. Therefore, when testing the algorithm under test V1, it is only necessary to modify the test execution configuration files of test case set 1 and test case set 2 to adapt the content to the execution commands and processes of the algorithm under test V1. When testing Algorithm V2, the test execution configuration files for Test Case Set 1 and Test Case Set 2 only need to be modified to adapt to the execution commands and processes of Algorithm V2. The test case design files and script layer architecture do not require modification. Furthermore, when commands or options corresponding to the testing tool need updating, only the test execution configuration file under the corresponding test case set needs to be updated synchronously. When adding test cases, only the test cases need to be added to the corresponding test case set; there is no need to repeatedly add the corresponding execution scripts. Moreover, if a test function is no longer needed in the latest version, only the entire test case set corresponding to that function needs to be deleted, without affecting the test execution of any other test case sets. This achieves decoupling between the execution layer and the test case layer, and between the script layer and the test case layer, facilitating switching between different Algorithm testing tools. Different test case sets can also be used for testing by multiple Algorithm testing tools. The relationship between test case sets and Algorithm testing tools is many-to-many, and the correspondence is adjusted through the test execution configuration file. See also... Figure 6 It is understood that the relationship between the algorithm tool under test and the test case set can be many-to-many. Figure 6 Taking the algorithm under test as an example, the relationship between the partitioning algorithms and the test case sets of different versions V1, V2...Vx (where x is a natural number greater than 2) is distinguished by different line types. Figure 6 As can be seen, each partitioning algorithm can be associated with each test case set.

[0194] The test execution configuration file can be in .ini format.

[0195] Specifically, the test execution configuration file sets the test type that represents the test function, the matching test tool, the description of the test tool execution process, the target execution objective, and the corresponding execution command.

[0196] The test type determines which type of testing tool to use to execute the test. For functional testing, a functional testing tool is used; for performance testing, a performance testing tool is used. This embodiment of the invention pre-defines corresponding testing tools for each type of test function. Furthermore, each type of testing tool in this embodiment can be independently developed and extended; the tools are not interdependent and can be added as needed. The matched testing tools are typically represented by a list of execution tools [Resource], which specifies which testing tools(s) will be used to execute the test of the algorithm tool under test on the test case set. The test tool execution flow description can be represented by [TestFlow], which details the execution behavior of the testing tool or some execution parameter options. The file format of the test execution configuration file can be as follows:

[0197]

[0198] The script layer uses Python's configparser to parse the test execution configuration file line by line. First, it reads the test type line, using the `conf.get` method to retrieve the `[Functionality Test]` field. If it's a Functionality Test, the functional test tool is invoked; if it's a Performance Test, the performance test tool is invoked. This can be extended to other test types for easy expansion. The `[Resource]` field is parsed in the same way. If the test tool is Tool 1, Tool 1 from the functional test is selected. Next, the `[Test Flow]` field and flow description are parsed, along with the execution options, to extract the execution command. The execution command contains two variables, `{name}` and `{cuttunt_dir}`. These are automatically passed as variable parameters based on the algorithm tool under test and its directory, and then concatenated to form the execution command, initiating automated command invocation and execution. Specifically, the directory of the algorithm tool under test in the input information is passed to the `cuttunt_dir` parameter, which is concatenated with other parameters to form the command used by the algorithm tool under test to execute the test case. The `{name}` parameter specifies the name of the algorithm tool being tested. After execution, the results are saved and logs are collected.

[0199] The process from parsing the test execution configuration file to receiving the automated test execution results from the execution layer can include the following steps:

[0200] 1. Define a function to parse and execute the test configuration file;

[0201] 2. Read and parse the file;

[0202] 3. Call the parser to read the test execution configuration file line by line;

[0203] 4. Analyze the test types and classifications, test tools, descriptions of the test tool execution process, the target execution objectives, and the corresponding execution commands;

[0204] 5. Transmit the parsing results and related parameters to the execution layer so that the execution layer can call the corresponding test tools to execute the test cases and complete the test;

[0205] 6. Receive the result data of the execution layer's test completion;

[0206] The pseudocode for this part is as follows:

[0207]

[0208]

[0209] It should be noted that the code parameters and other parameters given as examples in the embodiments of the present invention are merely format examples, and users can set them reasonably according to their design ideas, without any restrictions.

[0210] In this embodiment of the invention, parts that need to be modified during test execution can be added or extended through test execution configuration files. This is beneficial for beginners and testers without relevant background knowledge to quickly conduct tests. They only need to configure the test execution configuration file according to their own requirements and the instructions and specifications, without needing in-depth knowledge of the code. At the same time, the weak coupling between code allows multiple different developers to develop in parallel without affecting each other, which can improve development efficiency. The development of functional testing tools and performance testing tools only requires the development of testing tools in their respective fields, which are then integrated into the system. Finally, this embodiment of the invention has strong scalability. As needed, the required testing tools can be developed and integrated into the system without modifying the system's framework structure.

[0211] In this embodiment of the invention, the association update between the test case set, test tools, and execution targets is achieved by modifying the test execution configuration file in the test case set. This avoids the hard-coding problem for a specific requirement, has high scalability, and can be extended to test different types of algorithms or software tools.

[0212] As can be seen, the automated testing system based on Python algorithms proposed in this embodiment of the invention has the following advantages compared with the prior art:

[0213] 1) It transforms a large number of repeatable test steps into automated execution, which can be executed with one click without human intervention, thereby improving testing efficiency and reducing labor costs;

[0214] 2) Different types of functions or features of the algorithm tool under test are classified and executed, and test cases can be categorized; multiple test case sets used for testing the algorithm tool under test are specified through the test case set configuration file. The association and update between the test case set and the test tool and execution target are achieved by modifying the test execution configuration file in the test case set. Test cases can be easily added, searched, and deleted, and changes to test cases do not affect changes to other content.

[0215] 3) To test different types or versions of the algorithm tool under test, only the directory of the algorithm tool under test and the relevant configuration file parameters in the input information need to be modified. This avoids the problem of hard coding for a specific requirement, has high scalability, and can be extended to test different types of algorithms or software tools.

[0216] 4) It can automatically perform visual statistical analysis of test results, helping testers evaluate the quality and performance of the algorithm tools under test;

[0217] 5) Implemented using Python, it has good platform portability and can be built on different types of operating system platforms, with wide coverage.

[0218] Secondly, embodiments of the present invention provide an automated testing method for algorithm tools based on Python, implemented using an automated testing system, which includes a script layer, an execution layer, and a test case layer; such as Figure 7 As shown in the embodiment of the present invention, an automated testing method for algorithm tools based on Python may include the following steps:

[0219] S1, the script layer interacts with user input information and manages and schedules the overall process of the automated testing system. This includes receiving input information, initiating the automated testing process, parsing the input information to obtain the results, and passing them to the execution layer. The script layer contains automated test scripts. Input information includes the directory of the algorithm tool under test, the name and directory of the test case set configuration file, and the algorithm tool under test (either the algorithm itself or the software tool), including partitioning algorithms in circuit design. Calling different algorithm tools is achieved by adjusting their location configuration. Multiple test cases used for testing the algorithm tool are specified through the test case set configuration file.

[0220] S2, the execution layer, based on the parsing results, calls the corresponding test case set and matching test tool in the case layer to complete the test of the algorithm tool to be executed. The case layer includes multiple test case sets; each test case set describes a type of test function; one test function corresponds to at least one test case; each test case set consists of a test execution configuration file and at least one test case; the test execution configuration file sets the test type representing the test function, the matching test tool, the test tool execution flow description, the target execution target, and the corresponding execution command; the association update between the test case set and the test tool / target execution target is achieved by modifying the test execution configuration file in the test case set; multiple test case sets are formed by pre-generating and categorizing several test cases in the case layer.

[0221] S3, the script layer calls the result detection tool to process the result data of the execution layer and outputs automated test results.

[0222] For a detailed implementation process of the automated testing method based on Python algorithm tools in this embodiment of the invention, please refer to [link / reference]. Figure 8 This is related to the content of the Python-based algorithm tool automated testing system described in the first part of the previous text, and will not be explained in detail here.

[0223] In the solution provided by this invention, the script layer, in conjunction with the execution layer and the case layer, implements a Python-based automated testing method for algorithm tools. This method transforms numerous repetitive test steps into automated execution, eliminating the need for manual intervention, thus improving testing efficiency and reducing labor costs. Furthermore, it automatically performs visualization, statistical analysis, and evaluation of test results, helping testers assess the quality and performance of the algorithm tools under test. Implemented in Python, this testing method boasts excellent platform portability, allowing it to be built on various operating system platforms with broad coverage. Moreover, it can categorize and execute different types of functions or features of the algorithm tools under test, enabling the classification of test cases. Using relevant configuration files, test cases can be easily added, searched, and deleted without affecting other content, avoiding hard-coding for specific needs. This high scalability allows for the testing of different types of algorithms or software tools.

[0224] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A python-based algorithmic tool automated testing system, characterized in that, Implemented using the Python language, the system includes a script layer, an execution layer, and a case layer. The script layer interacts with user input and manages and schedules the automated testing system's workflow. This includes receiving the input, initiating the automated testing process, parsing the input to obtain results, and passing these results to the execution layer; and calling a result detection tool to process the execution layer's data and output the automated testing results. The script layer contains automated testing scripts. The input information includes the directory of the algorithm tool under test, and the name and directory of the test case set configuration file. The algorithm tool under test includes either the algorithm itself or the software tool under test, including partitioning algorithms in circuit design. Calling different algorithm tools is achieved by adjusting their location configuration. Multiple test cases used for testing the algorithm tool are specified through the test case set configuration file. The execution layer is used to call the corresponding test case set and matching test tool in the case layer according to the parsing result, and to complete the test on the algorithm tool to be tested, which is the execution target. The test case layer includes multiple test case sets; each test case set describes a type of test function; one test function corresponds to at least one test case; each test case set consists of a test execution configuration file and at least one test case; the test execution configuration file sets the test type representing the test function, the matching test tool, the test tool execution flow description, the target execution objective, and the corresponding execution command; the association update between the test case set and the test tool and execution objective is achieved by modifying the test execution configuration file in the test case set; the multiple test case sets are formed by automatically generating and classifying several test cases in the test case layer in advance. The script layer parses the input information to obtain the parsing result, including: reading the input information; parsing the input information using a parser to obtain the directory of the algorithm tool under test, the name and directory of the test case set configuration file; locating the algorithm tool under test according to the directory of the algorithm tool under test, and obtaining the type and version of the algorithm tool under test as the parsing result of the algorithm tool under test; locating each test case set according to the name and directory of the test case set configuration file, and parsing the test execution configuration file of each test case set to obtain the test execution configuration file parsing result; Specifically, the generation process of the multiple test case sets for the partitioning algorithm includes: repeatedly executing the test case generation step to generate multiple test cases; wherein the test case generation step includes: automatically generating a hypergraph design file; designing a test constraint file based on the hypergraph design file and combined with test characteristic requirements; combining the hypergraph design file and the test constraint file to form a test case; classifying the generated multiple test cases according to test functions to form different test case sets; the automatic generation of the hypergraph design file includes automatically generating a node list file and a netlist file for the test cases; The automatic generation process of the node list file includes: Step a1: Define a node generation function, where the parameters are the total number of nodes M and the resource ratio of each node P; Step a2: Calculate the number of resource nodes N that need to be supported based on N=M*P; Step a3: Set the value range of the total node, and randomly generate N resource nodes within that range; Step a4: Set the range of node weight changes [a, b]; Step a5: Iterate through the nodes. If a node is a node with resources, randomly generate a weight value for the node within the range [a, b]. If a node is a node without resources, set the weight value of the node to 0. Step a6: Put all the generated nodes and their weight values ​​into a list, and write the list back to a file to obtain a node list file; The automatic generation process of the netlist file includes: Step b1, define the netlist connection generation function, where the parameters are node file, node percentage q, and number of connections L; Step b2: Read all nodes in the node file line by line and store them in a node list; Step b3: Randomly select M*q nodes from the node list obtained in step b2 as driving nodes; where M is the total number of nodes. Step b4: Remove the driver node from the node list obtained in step b2, randomly select L load nodes and form a connection relationship with the driver node in step b2, and save it to the list file; Step b5: Iterate through the list of all driver nodes in step b3 until each driver node has a connection with a load node, then exit the loop. Step b6: Write the generated connection list file back to the netlist file.

2. The python-based algorithmic tool automated testing system of claim 1, wherein, The test constraint file includes: FPGA resource constraints, topology constraints between FPGAs, and fixed node constraints; The description format of the FPGA resource constraints includes the FPGA number and the upper limit of K types of resource values ​​on the FPGA represented by K groups of numbers after the FPGA number. The description format of the topology constraints between the FPGAs includes the FPGA numbers of the two connected FPGAs and the physical connection resource values ​​between them. The description format of the fixed node constraint includes the FPGA number and the node number pre-assigned to the FPGA.

3. The python-based algorithmic tool automation testing system of claim 1, wherein, The call result detection tool processes the result data of the execution layer, including: The result detection tool is invoked to detect and verify the result data of the execution layer, and then visualization analysis and statistics are performed. The methods used to detect and verify the result data of the execution layer include: file comparison verification, keyword extraction verification, and verification by calling third-party tools; the methods used for keyword extraction verification include: regular expression extraction and line-by-line parsing extraction.

4. The Python-based algorithmic tool automated testing system of claim 1, wherein, The system is equipped with a running display interface, which is used to write input information and display prompts during and after the test.

5. The python-based algorithmic tool automation testing system of claim 1, wherein, The input information is set according to a predetermined parameter format and includes, in sequence: an entry script, an input identifier for the algorithm tool to be tested set with a first preset character, a directory of the algorithm tool to be tested, a test case set configuration file identifier set with a second preset character, a directory of the test case set configuration file, and the name of the test case set configuration file.

6. The python-based algorithmic tool automated testing system of claim 5, wherein, The test case configuration file is in config format, and the content of each test case includes the test case name, test case directory and test case description. The test case set supports multi-level search, implemented using a depth-first traversal algorithm.

7. A python-based algorithm tool automatic testing method, characterized in that, The method is implemented using the automated testing system according to any one of claims 1-6, wherein the automated testing system includes a script layer, an execution layer, and a test case layer; the method includes: The script layer interacts with user input information and manages and schedules the workflow of the automated testing system using scripts. This includes receiving the input information, initiating the automated testing process, parsing the input information to obtain the parsing results, and passing them to the execution layer. The script layer contains automated test scripts. The input information includes the directory of the algorithm tool under test, and the name and directory of the test case set configuration file. The algorithm tool under test includes the algorithm or software tool under test, and the algorithm under test includes partitioning algorithms in circuit design. Calling different algorithm tools under test is achieved by adjusting the position configuration of the algorithm tools under test. Multiple test case sets used for testing the algorithm tools under test are specified through the test case set configuration file. Based on the parsing results, the execution layer calls the corresponding test case set and matching test tool in the case layer to perform tests on the algorithm tool under test, which is the execution target. The case layer includes multiple test case sets; each test case set describes a type of test function; one test function corresponds to at least one test case; each test case set consists of a test execution configuration file and at least one test case; the test execution configuration file sets the test type representing the test function, the matching test tool, the test tool execution flow description, the target execution target, and the corresponding execution command; the association update between the test case set and the test tool / target execution target is achieved by modifying the test execution configuration file in the test case set; the multiple test case sets are formed by automatically generating and classifying several test cases in the case layer beforehand. The script layer calls the result detection tool to process the result data of the execution layer and outputs automated test results.

Images