Verification performance profiling with regression test

Abstract

This application discloses a computing system to simulate a circuit design describing an electronic device with a plurality of functional verification tests, separately collect samples of performance data during simulation of the circuit design for each of the functional verification tests, and generate test databases for each of the functional verification tests. Each of the test databases include samples for a corresponding functional verification test. The computing system can merge the samples of performance data in the test databases for the functional verification tests. The computing system can filter the merged samples of performance data into a subset of the merged samples of performance data based, at least in part, on profile filtering commands, and generate a profile presentation including the subset of the merged samples of performance data. The profile presentation, when displayed, can annunciate portions of the circuit design corresponding to at least one performance hotspot.

Description

VERIFICATION PERFORMANCE PROFILING WITH REGRESSION TEST DATABASE MERGING TECHNICAL FIELD

[0001] This application is generally related to electronic design automation and, morespecifically, to verification performance profiling with regression test database merging.BACKGROUND

[0002] Designing and fabricating electronic systems typically involves many steps, knownas a “design flow.” The particular steps of a design flow often are dependent upon the type of electronic system to be manufactured, its complexity, the design team, and the fabricator or foundry that will manufacture the electronic system from a design. Initially, a specification for a new electronic system can be transformed into a logical design, sometimes referred to as a register transfer level (RTL) description of the electronic system. With this logical design, the electronic system can be described in terms of both the exchange of signals between hardware registers and the logical operations that can be performed on those signals. The logical design typically employs a Hardware Design Language (HDL), such as System Verilog or Very high speed integrated circuit Hardware Design Language (VHDL).

[0003] The logic of the electronic system can be analyzed to confirm that it will accuratelyperform the functions desired for the electronic system, sometimes referred to as “functional verification.” Design verification tools can perform functional verificationoperations, such as simulating, emulating, and / or prototyping the logical design. For example, when a design verification tool simulates the logical design, the design verification tool can provide transactions or sets of test vectors, for example, generated by a simulated test bench, to the simulated logical design. The design verification tools can determine how the simulated logical design responded to the transactions or test vectors, and verify, from that response, that the logical design describes circuitry to accurately perform functions.

[0004] As the logical designs increase in size and verification runtime becomes longer,functional verification speed-up can be obtained by either making design verification tool faster or by writing logical design more efficiently. Many design verification tools have integrated performance profiling capability that can collect runtime performance data of the functional verification operations performed by the design verification tools and provide analysis on the performance data to locate performance hotspots in the functional verification of the logical design. The performance profiling tools typically gather performance data and determine performance hotspots on a test-by-test basis for design teams to manually interpret and perform root cause analysis. While this test-by-test approach can identify some of the performance hotspots in the functional verification of the logical design, some design elements, such as those having a large amount of simulated activity, but across a wide range of tests, tend to remain obfuscated from root cause analysis. SUMMARY

[0005] This application discloses a computing system implementing a design verificationtool to perform functional verification on a circuit design describing an electronic devicewith a plurality of functional verification tests, for example, performed by at least one of a digital simulator, an analog simulator, a hardware-based emulator, or a hardware-based prototyping system. The design verification tool can separately collect samples of performance data, on a per functional verification test basis, during the functional verification, for example, periodically during the functional verification of the circuit design or in response to one or more events occurring during the functional verification of the circuit design. The samples of the performance data can include one or more of a simulation time, a design instance of the circuit design being executed, a design unit of the circuit design being executed, a function of the circuit design being executed, a call stack of the functional verification, or a code type for the circuit design.

[0006] The computing system can generate test databases for each of the functionalverification tests, with each of the test databases including samples for a corresponding functional verification test. The computing system can merge the samples of performance data collected during each of the functional verification tests, which can be stored in a regression database. The computing system can filter the merged samples of performance data into a subset of the merged samples of performance data based, at least in part, on profile filtering commands, and generate a profile presentation including the subset of the merged samples of performance data. The profile presentation, when displayed, can annunciate portions of the circuit design corresponding to at least one performance hotspot. Embodiments will be described in greater detail below. DESCRIPTION OF THE DRAWINGS

[0007] Figures 1 and 2 illustrate an example of a computer system of the type that may beused to implement various embodiments.

[0008] Figure 3 illustrates an example design verification system having performanceprofiling with regression test database merging that may be implemented according to various embodiments.

[0009] Figure 4 illustrates an example regression suite profile analysis system forperformance profiling using a regression database with merged performance files, which may be implemented according to various embodiments.

[0010] Figure 5 illustrates an example flowchart implementing performance profiling of adesign verification tool with regression-level test database merging, which may be implemented according to various embodiments. DETAILED DESCRIPTION Illustrative Operating Environment

[0011] Various embodiments may be implemented through the execution of softwareinstructions by a computing device 101, such as a programmable computer. Accordingly, Figure 1 shows an illustrative example of a computing device 101. As seen in this figure, the computing device 101 includes a computing unit 103 with a processing unit 105 and a system memory 107. The processing unit 105 may be any type of programmable electronic device for executing software instructions, but will conventionally be a microprocessor. The system memory 107 may include both a read-only memory (ROM) 109 and a random accessmemory (RAM) 111. As will be appreciated by those of ordinary skill in the art, both the read-only memory (ROM) 109 and the random access memory (RAM) 111 may store software instructions for execution by the processing unit 105.

[0012] The processing unit 105 and the system memory 107 are connected, either directlyor indirectly, through a bus 113 or alternate communication structure, to one or more peripheral devices 117-123. For example, the processing unit 105 or the system memory 107 may be directly or indirectly connected to one or more additional memory storage devices, such as a hard disk drive 117, which can be magnetic and / or removable, a removable optical disk drive 119, and / or a flash memory card. The processing unit 105 and the system memory 107 also may be directly or indirectly connected to one or more input devices 121 and one or more output devices 123. The input devices 121 may include, for example, a keyboard, a pointing device (such as a mouse, touchpad, stylus, trackball, or joystick), a scanner, a camera, and a microphone. The output devices 123 may include, for example, a monitor display, a printer and speakers. With various examples of the computing device 101, one or more of the peripheral devices 117-123 may be internally housed with the computing unit 103. Alternately, one or more of the peripheral devices 117-123 may be external to the housing for the computing unit 103 and connected to the bus 113 through, for example, a Universal Serial Bus (USB) connection.

[0013] With some implementations, the computing unit 103 may be directly or indirectlyconnected to a network interface 115 for communicating with other devices making up a network. The network interface 115 can translate data and control signals from the computing unit 103 into network messages according to one or more communicationprotocols, such as the transmission control protocol (TCP) and the Internet protocol (IP). Also, the network interface 115 may employ any suitable connection agent (or combination of agents) for connecting to a network, including, for example, a wireless transceiver, a modem, or an Ethernet connection. Such network interfaces and protocols are well known in the art, and thus will not be discussed here in more detail.

[0014] It should be appreciated that the computing device 101 is illustrated as an exampleonly, and it not intended to be limiting. Various embodiments may be implemented using one or more computing devices that include the components of the computing device 101 illustrated in Figure 1, which include only a subset of the components illustrated in Figure 1, or which include an alternate combination of components, including components that are not shown in Figure 1. For example, various embodiments may be implemented using a multi-processor computer, a plurality of single and / or multiprocessor computers arranged into a network, or some combination of both.

[0015] With some implementations, the processor unit 105 can have more than oneprocessor core. Accordingly, Figure 2 illustrates an example of a multi-core processor unit 105 that may be employed with various embodiments. As seen in this figure, the processor unit 105 includes a plurality of processor cores 201A and 201B. Each processor core 201A and 201B includes a computing engine 203A and 203B, respectively, and a memory cache 205A and 205B, respectively. As known to those of ordinary skill in the art, a computing engine 203A and 203B can include logic devices for performing various computing functions, such as fetching software instructions and then performing the actions specified in the fetched instructions. These actions may include, for example, adding, subtracting,multiplying, and comparing numbers, performing logical operations such as AND, OR, NOR and XOR, and retrieving data. Each computing engine 203A and 203B may then use its corresponding memory cache 205A and 205B, respectively, to quickly store and retrieve data and / or instructions for execution.

[0016] Each processor core 201A and 201B is connected to an interconnect 207. Theparticular construction of the interconnect 207 may vary depending upon the architecture of the processor unit 105. With some processor cores 201A and 201B, such as the Cell microprocessor created by Sony Corporation, Toshiba Corporation and IBM Corporation, the interconnect 207 may be implemented as an interconnect bus. With other processor units 201A and 201B, however, such as the Opteron™ and Athlon™ dual-core processors available from Advanced Micro Devices of Sunnyvale, California, the interconnect 207 may be implemented as a system request interface device. In any case, the processor cores 201A and 201B communicate through the interconnect 207 with an input / output interface 209 and a memory controller 210. The input / output interface 209 provides a communication interface to the bus 113. Similarly, the memory controller 210 controls the exchange of information to the system memory 107. With some implementations, the processor unit 105 may include additional components, such as a high-level cache memory accessible shared by the processor cores 201A and 201B. It also should be appreciated that the description of the computer network illustrated in Figure 1 and Figure 2 is provided as an example only, and it not intended to suggest any limitation as to the scope of use or functionality of alternate embodiments. Verification Performance Profiling with Regression Test Database Merging

[0017] Figure 3 illustrates an example design verification system 300 having performanceprofiling with regression test database merging that may be implemented according to various embodiments. Referring to Figure 3, the design verification system 300 can include a design verification tool 310, for example, implemented with a computer network 101 described above with reference to Figure 1, to functionally verify a circuit design 301 describing an electronic device. In some embodiments, the circuit design 301 can describe the electronic device both in terms of an exchange of data signals between components in the electronic device, such as hardware registers, flip-flops, combinational logic, or the like, and in terms of logical operations that can be performed on the data signals in the electronic device. The circuit design 301 can model the electronic device at a register transfer level (RTL), for example, with code in a hardware description language (HDL), such as System Verilog, Very high speed integrated circuit Hardware Design Language (VHDL), System C, a combination thereof, or the like.

[0018] The design verification tool 310 also can utilize a test bench 302 to generate teststimulus during functional verification operations, such as clock signals, activation signals, power signals, control signals, data signals or the like. The test stimulus, when grouped, may form test bench transactions capable of prompting operation of the circuit design 301 being functionally verified by the design verification tool 300. In some embodiments, the test bench 302 can be written in an object-oriented programming language, for example, System Verilog or the like, which, when executed during elaboration, can dynamically generate test bench components for verification of the circuit design. A methodology library, for example, a Universal Verification Methodology (UVM) library, an Open Verification Methodology (OVM) library, an Advanced Verification Methodology (AVM)library, a Verification Methodology Manual (VMM) library, or the like, can be utilized as a base for creating the test bench. The test bench 302 can include a suite of test benches that can be written in a variety of programming languages, for example, corresponding to the hardware description languages of the portions of the circuit design 301.

[0019] The functional verification system 311 can include at least one of a simulation tool, ahardware-based emulation tool, a hardware-based prototyping tool, or the like, to perform the functional verification operations on the circuit design 301. The simulation tool can perform the functional verification operations with one or more processors configured to simulate the circuit design 301. The emulation tool and the prototyping tool can perform functional verification operations with configurable hardware, such as Field Programmable Gate Arrays (FPGAs), configured to implement electronic device described by the circuit design 301. The functional verification system 311 can utilize the test bench 302 to perform multiple functional verification tests on the circuit design 301, for example, to implement a regression suite. The regression suite can include multiple separate tests run on the circuit design 301 during functional verification by the test bench 302. The functional verification system 311, for each test run in the regression suite, can generate output corresponding to the operations of the circuit design 301 in response to the test stimulus during the functional verification operations, which can be compared to expected output of the circuit design 301.

[0020] The design verification tool 310 can include a performance profiler 312 to gatherinformation on the functional verification operations performed on the circuit design 301 in each of the tests within the regression suite and create individual test-level performancedatabases that includes the gathered information. The performance profiler 312 can then merge the test-level performance databases together to create merged performance files 316 to be stored in a regression database 320.

[0021] The performance profiler 312 can include a sampling system 313 to collect samplesof performance data for multiple tests within the regression suite performed by the functional verification system 311. In some embodiments, the sampling system 313 can collect the samples of performance data for the functional verification system 311 periodically or synchronously, for example, utilizing an interrupt-based procedure to capture samples of the functional verification system 311 at a predetermined frequency. The performance profiler 312 also can collect the samples of performance data for the functional verification system 311 asynchronously, for example, utilizing an event-based procedure to capture samples of the functional verification system 311.

[0022] The performance profiler 312 can include a test database generation system 314 toseparately generate performance data files for each test in the regression suite by organizing the performance data by different design elements associated with the functional verification of the circuit design 301 and arranging the organized data relative to the samples in the performance data files 313. In some embodiments, the design elements can include one or more of a verification time, a design instance, a design unit, a hardware description language process, a hardware description language task and function, a source file, a call stack, or the like. The verification time can correspond to a duration of simulation, emulation, or prototyping activity performed by the functional verification system 311. The design instance time can represent verification activity for each hardwaredescription language design instance in the circuit design 301 and within its hierarchy. The design unit can represent verification activity for each hardware description language design unit in the circuit design 301. The hardware description language process can represent verification activity for each hardware description language process associated with the circuit design 301. The hardware description language task and function can represent verification activity for each hardware description language task and function associated with the circuit design 301. The source file can represents verification activity for each hardware description language file associated with the circuit design 301. The call stack can represent an internal list of executable code portions during functional verification of the circuit design 301.

[0023] The design verification tool 310 can include a sample merging system 315 toaggregate the performance data files for each test in the regression suite run by the functional verification system 311. The sample merging system 315 can merge the performance data files into an omnibus list of all samples collected by the sampling system 313, called the merged performance files 316, which can include performance data associated with the samples, such as the design elements, verification time, design instance, design unit, hardware description language process, hardware description language task and function, source file, call stack, or the like. The merged performance files 316 can also include a listing of which test in the regression suite was associated with the various samples. By merging the performance data files for each test in the regression suite into the merged performance files 316, the sampled performance data can be analyzed downstream across multiple tests, rather than test-by-test as performed previously.

[0024] The design verification tool 310 can store the merged performance files 316 in theregression database 320, for example, after the functional verification system 311 has completed all of the verification runs of the circuit design 301 and the performance profiler 312 has merged the performance data files for each verification run into the merged performance files 316. In some embodiments, the merged performance files 316 can be uncollated sample data, for example, not already arranged and organized by design element, but instead left relative to the sample of the collected performance data. Although the regression database 320 is shown in Figure 3 to be external to the design verification tool 310, in some embodiments, the design verification tool 310 can include the regression database 320.

[0025] The design verification system 300 can include a visualization tool 330 to receive themerged performance files 316 from the regression database 320 and to generate one or more profile presentations 332 based on the samples of performance data in the merged performance files 316. The profile presentation 332, when displayed, can be configured to provide a view of the merged performance data 316, which can annunciate portions of the circuit design 301 corresponding to at least one performance hotspot for the functional verification system 311. For example, when the functional verification system 311 includes a simulation tool, the profile presentation 332 can present the performance data to show a performance hotspot as a design unit or design instance in the circuit design 301 being functionally verified that consumed the most simulation time or had the most simulation activity across a plurality of verification runs. By analyzing sampled performance data across multiple verification runs, the profile presentation 332 can locate hotspots that impact the simulation runs in the entire regression suite rather than within any specifictest. For example, when a design unit has a relatively low number of samples in any given test, a test-by-test analysis may not identify the design unit as a hotspot, but when that same design unit has samples in a high number of the overall tests, the volume of activity across the regression suite can indicate the design unit as a hotspot. Without a regression suite-level view of the performance data, many broadly implemented design units, instances, functions, or the like, may be obfuscated, leading to unresolved simulation bottlenecks.

[0026] The visualization tool 330 can include a regression suite profile analysis system 400to determine a subset of the merged performance files 316 to present as a view based on one or more profile filter commands 333, identify design elements in each of the samples of the performance data in the filtered version of the merged performance files 316, and arrange and collate the performance data according to the identified design elements. The regression suite profile analysis system 400 can build call trees from the collated version of the performance data that can be a flat or hierarchical representation for the types of the identified design elements and identify a frequency associated with the presence of the design elements in the subset of the merged performance files 316. The regression suite profile analysis system 400, in some embodiments, can utilize the profile filter commands 333 by applying filters the merged performance files 316 in the regression database 320, for example, to identify the subset of the merged performance files 316 for the regression suite profile analysis system 400 to utilize to build the call trees. In some embodiments, the regression suite profile analysis system 400 can include an application programming interface (API) to interface between the regression database 320 and the visualization tool330, which can selectively access the merged performance files 316 from the regression database 320.

[0027] The visualization tool 330 can include a presentation system 331 to generate a viewof the subset of the merged performance files 316 in profile presentations 332, for example, by inserting the call trees built by the regression suite profile analysis system 400, and prompt a display device to present the profile presentations 332. In some embodiments, the call trees can each include counter values for each node of the call trees. The counter values can correspond to design elements in the samples of the performance data, for example, a number of the samples collected when a corresponding node in one of the call trees was active or had code executing. One or more performance hotspots can correspond to those counter values in the nodes in the hierarchical trees with the profile presentation 332 having higher counts for verification activity during the function verification of the circuit design 301. In some embodiments, the presentation system 331 also can annunciate performance hotspots in the profile presentation 332 utilizing the flat representation, for example, by ranking nodes and combined nodes in the flat representation by their corresponding counter values and then utilizing a group or subset of the lines or functions having the highest counter values in the ranking of the nodes and combined nodes to correspond to the performance hotspots.

[0028] By having the performance profiler 312 store performance data relative to thesample into the regression database 320 across all of the tests in the regression suite, the visualization tool 330 can generate call trees of the performance data with varying degrees of refinement or granularity and allow visibility into a simulation activity of designelements across multiple tests to reveal performance hotspots previously obfuscated by the test-by-test analysis approach. Embodiments of the regression suite profile analysis system 400 will be described below with reference to Figure 4 in greater detail.

[0029] Figure 4 illustrates an example regression suite profile analysis system 400 forperformance profiling using a regression database with merged performance files, which may be implemented according to various embodiments. Referring to Figure 4, the regression suite profile analysis system 400 can include a data filtering system 410 to receive merged performance files 401, for example, from a regression database, which includes samples of performance data collected during functional verification of a circuit design over multiple tests in a regression suite. The merged performance files 401 can be a list of the samples of performance data organized by different design elements associated with the functional verification of the circuit design and be arranged relative to the samples. In some embodiments, the merged performance files 401 can correspond to samples of performance data that were collected periodically or synchronously, for example, utilizing an interrupt-based procedure to capture samples of the design verification tool at a predetermined frequency across the multiple tests in the regression suite. The merged performance files 401 can correspond to samples of performance data that were collected asynchronously, for example, utilizing an event-based procedure to capture samples of the design verification tool.

[0030] The data filtering system 410 also can receive profile filter commands 402 whichidentify a subset of performance data in the merged performance files 401 to analyze. For example, the profile filter commands 402 can include at least one attribute of the circuitdesign, such as design unit, design instance, function name, line of code, design file, type of hardware description language, a test or group of tests in the regression suite, or the like.

[0031] The data filtering system 410 can selectively filter the merged performance files 401based, at least in part, on the profile filter commands 402. In some embodiments, the data filtering system 410 can identify the attribute(s) of the circuit design from the profile filter commands 402 and remove selected portions of the merged performance files 401 that correspond to one or more of the identified attributes based on the profile filter commands 402. The profile filter commands 402 also can direct the data filtering system 410 to retain selected portions of the merged performance files 401 that correspond to one or more of the identified attributes. The ability to identify portions of the merged performance files 401 to remove and / or retain during data filtering operations can allow the regression suite profile analysis system 400 to customize the views of the merged performance files 316 for analysis.

[0032] The regression suite profile analysis system 400 can include a data collating system420 to identify a presence of the design elements in the samples of performance data in the merged performance files 401. In some embodiments, the design elements can include one or more of a verification time, a design instance, a design unit, a hardware description language process, a hardware description language task and function, a source file, a call stack, test in the regression suite, or the like. The data collating system 420 can analyze each sample individually to identify the design elements corresponding to the performance data. The design collating unit 420 can organize the performance data according to the design elements identified by the sample traversal unit 411. The design collating system420, in some embodiments, can utilize the identified design elements to identify which call trees to build and a hierarchical structure of the call trees. The design collating system 420 also can collate the identified design elements to the different call trees. For example, when a sample identifies a specific design instance performing a specific function, the design collating system 420 can collate the specific design instance to a call tree for that specific function and collate the specific function to a call tree for that specific design instance and increment counter values which represent the number of samples for that instance.

[0033] The regression suite profile analysis system 400 can include a filtered tree buildingsystem 430 to generate call trees 403 with varying levels of granularity utilizing merged performance files 401. The filtered tree building system 430 can build the call trees 403 as hierarchical trees for the different types of the identified design elements. Each hierarchical tree can include nodes corresponding to the design elements in the performance data, which can be organized according to the hierarchical structure of the circuit design under verification. Each of the nodes in the hierarchical tree can include a bin or counter to accumulate a value corresponding to the presence of the design element in the performance data files. The filtered tree building system 430 can utilize the design elements to add nodes to hierarchical trees when one was not previously added and increment the value in the bin or counter. The resulting call trees 403 built by the filtered tree building system 430 can include a hierarchical set of node representing design elements in sample data along with counters corresponding to a number of times the specific design element was present in the samples of the performance data.

[0034] Although Figure 4 shows the filtered tree building system 430 generating call trees403 using the merged performance files 401 and the profile filter commands 402, in some embodiments, the filtered tree building system 430 can generate different views of the merged performance files 401 based on the profile filter commends 402. For example, the filtered tree building system 430 can create a view of the merged performance files 401 by arranging the merged performance files 401 or selecting a subset of the merged performance files 401 based, at least in part, on a verification time, a design instance, a design unit, a hardware description language process, a hardware description language task and function, a source file, a call stack, or the like.

[0035] Figure 5 illustrates an example flowchart implementing performance profiling of adesign verification tool using data reduction, which may be implemented according to various embodiments. Referring to Figure 5, in a block 501, a computing system implementing a design verification tool can simulate a circuit design describing an electronic device with a plurality of functional verification tests. In some embodiments, the circuit design can describe the electronic device both in terms of an exchange of data signals between components in the electronic device, such as hardware registers, flip-flops, combinational logic, or the like, and in terms of logical operations that can be performed on the data signals in the electronic device. The circuit design can model the electronic device at a register transfer level (RTL), for example, with code in a hardware description language (HDL), such as System Verilog, Very high speed integrated circuit Hardware Design Language (VHDL), System C, or the like.

[0036] The test stimulus generated using the test bench during simulation, such as clocksignals, activation signals, power signals, control signals, data signals or the like, may be grouped to form test bench transactions capable of prompting operation of the circuit design being functionally verified by the design verification tool. In some embodiments, the test bench can be written in an object-oriented programming language, for example, System Verilog or the like, which, when executed during elaboration, can dynamically generate test bench components for verification of the circuit design. A methodology library, for example, a Universal Verification Methodology (UVM) library, an Open Verification Methodology (OVM) library, an Advanced Verification Methodology (AVM) library, a Verification Methodology Manual (VMM) library, or the like, can be utilized as a base for creating the test bench. The test bench can include a suite of test benches that can be written in a variety of programming languages, for example, corresponding to the hardware description languages of the portions of the circuit design. The computing system implementing the design verification tool can generate output corresponding to the operations of the circuit design in response to the test stimulus during the simulation, which can be compared to expected output of the circuit design.

[0037] In a block 502, a computing system can separately collect samples of performancedata during simulation of the circuit design for each of the functional verification tests. In some embodiments, the computing system implementing the performance profiling tool can collect the samples of performance data for the design verification tool periodically, for example, utilizing an interrupt-based procedure to capture samples of the design verification tool. The computing system implementing the performance profiling tool also can collect the samples of performance data for the design verification tool asynchronously,for example, utilizing an event-based procedure to capture samples of the design verification tool.

[0038] In a block 503, a computing system can store the separately collected samples ofperformance data into corresponding test databases for each of the functional verification tests. The computing system can separately store, in different test databases, the samples collected during different tests performed on the circuit design during the simulation.

[0039] In a block 504, a computing system can merge the samples of performance data inthe test databases into a regression database storing samples for all of the functional verification tests. The computing system can merge the samples of performance data into an omnibus list of all samples collected, which can include performance data associated with the samples, such as the design elements, verification time, design instance, design unit, hardware description language process, hardware description language task and function, source file, call stack, or the like. The merged samples of performance data can also include a listing of which test in the regression suite was associated with the various samples. By merging the performance data files for each test in the regression suite into the merged samples of performance data, the sampled performance data can be analyzed downstream across multiple tests, rather than test-by-test as performed previously.

[0040] In a block 505, a computing system implementing a regression suite profile analysistool can filter the merged samples of performance data into a subset of the merged samples of performance data based, at least in part, on profile filtering commands. The profile filter commands can identify a subset of performance data in the merged samples of performance data to analyze. For example, the profile filter commands can include at least one attributeof the circuit design, such as design unit, design instance, function name, line of code, design file, type of hardware description language, a test or group of tests in the regression suite, or the like. The computing system implementing the regression suite profile analysis tool can selectively filter the merged samples of performance data based, at least in part, on the profile filter commands. In some embodiments, the computing system implementing the regression suite profile analysis tool can identify the attribute(s) of the circuit design from the profile filter commands and remove selected portions of the merged samples of performance data that correspond to one or more of the identified attributes based on the profile filter commands 402. The profile filter commands also can direct the computing system implementing the regression suite profile analysis tool to retain selected portions of the merged samples of performance data that correspond to one or more of the identified attributes. The ability to identify portions of the merged samples of performance data to remove and / or retain during data filtering operations can allow the computing system implementing the regression suite profile analysis tool to customize the views of the merged samples of performance data for analysis.

[0041] In a block 506, a computing system implementing the regression suite profileanalysis tool can generate a profile presentation including the subset of the merged samples of performance data and annunciate portions of the circuit design corresponding to at least one performance hotspot. The computing system implementing the regression suite profile analysis tool can identify design elements in each of the samples of the performance data in the performance data files and arrange and collate the performance data according to the identified design elements. In some embodiments, the computing system implementing the profile analysis tool can build call trees for the types of the identified design elements andidentify a frequency associated with the presence of the design elements in the performance data files. The computing system implementing the profile analysis tool can insert the call trees into the profile presentations. The computing system implementing the regression suite profile analysis system can prompt a display device to present the profile presentations. The profile presentation, when displayed, can annunciate portions of the circuit design corresponding to at least one performance hotspot for the design verification tool with the call trees.

[0042] The system and apparatus described above may use dedicated processor systems,micro controllers, programmable logic devices, microprocessors, or any combination thereof, to perform some or all of the operations described herein. Some of the operations described above may be implemented in software and other operations may be implemented in hardware. Any of the operations, processes, and / or methods described herein may be performed by an apparatus, a device, and / or a system substantially similar to those as described herein and with reference to the illustrated figures.

[0043] The processing device may execute instructions or "code" stored in memory. Thememory may store data as well. The processing device may include, but may not be limited to, an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, or the like. The processing device may be part of an integrated control system or system manager, or may be provided as a portable electronic device configured to interface with a networked system either locally or remotely via wireless transmission.

[0044] The processor memory may be integrated together with the processing device, forexample RAM or FLASH memory disposed within an integrated circuit microprocessor or the like. In other examples, the memory may comprise an independent device, such as an external disk drive, a storage array, a portable FLASH key fob, or the like. The memory and processing device may be operatively coupled together, or in communication with each other, for example by an I / O port, a network connection, or the like, and the processing device may read a file stored on the memory. Associated memory may be "read only" by design (ROM) by virtue of permission settings, or not. Other examples of memory may include, but may not be limited to, WORM, EPROM, EEPROM, FLASH, or the like, which may be implemented in solid state semiconductor devices. Other memories may comprise moving parts, such as a known rotating disk drive. All such memories may be "machine- readable" and may be readable by a processing device.

[0045] Operating instructions or commands may be implemented or embodied in tangibleforms of stored computer software (also known as "computer program" or "code"). Programs, or code, may be stored in a digital memory and may be read by the processing device. “Computer-readable storage medium" (or alternatively, "machine-readable storage medium") may include all of the foregoing types of memory, as well as new technologies of the future, as long as the memory may be capable of storing digital information in the nature of a computer program or other data, at least temporarily, and as long at the stored information may be "read" by an appropriate processing device. The term "computer- readable" may not be limited to the historical usage of "computer" to imply a complete mainframe, mini-computer, desktop or even laptop computer. Rather, "computer-readable" may comprise storage medium that may be readable by a processor, a processing device, orany computing system. Such media may be any available media that may be locally and / or remotely accessible by a computer or a processor, and may include volatile and non-volatile media, and removable and non-removable media, or any combination thereof.

[0046] A program stored in a computer-readable storage medium may comprise a computerprogram product. For example, a storage medium may be used as a convenient means to store or transport a computer program. For the sake of convenience, the operations may be described as various interconnected or coupled functional blocks or diagrams. However, there may be cases where these functional blocks or diagrams may be equivalently aggregated into a single logic device, program or operation with unclear boundaries. Conclusion

[0047] While the application describes specific examples of carrying out embodiments,those skilled in the art will appreciate that there are numerous variations and permutations of the above described systems and techniques that fall within the spirit and scope of the invention as set forth in the appended claims. For example, while some of the specific terminology has been employed above to refer to electronic design automation processes, it should be appreciated that various examples may be implemented using any electronic system.

[0048] One of skill in the art will also recognize that the concepts taught herein can betailored to a particular application in many other ways. In particular, those skilled in the art will recognize that the illustrated examples are but one of many alternative implementations that will become apparent upon reading this disclosure.

[0049] Although the specification may refer to “an”, “one”, “another”, or “some” example(s)in several locations, this does not necessarily mean that each such reference is to the same example(s), or that the feature only applies to a single example.

Claims

CLAIMS 1. A method comprising: simulating, by a computing system, a circuit design describing an electronic device with a plurality of functional verification tests; separately collecting, by the computing system, samples of performance data during simulation of the circuit design for each of the functional verification tests; merging, by the computing system, the samples of performance data from each of the functional verification tests; filtering, by the computing system, the merged samples of performance data into a subset of the merged samples of performance data based, at least in part, on profile filtering commands; and generating, by the computing system, a profile presentation including the subset of the merged samples of performance data, wherein the profile presentation, when displayed, is configured to annunciate portions of the circuit design corresponding to at least one performance hotspot.

2. The method of claim 1, wherein separately collecting the samples of performance data during simulation of the circuit design for each of the functional verification tests further comprises generating test databases for each of the functional verification tests, and wherein the merging of the samples of performance data from each of the functional verification tests further comprises merging the test databases into a regression database including samples of performance data for all of the functional verification tests.

3. The method of claim 2, wherein merging the samples of performance data from each of the functional verification tests includes aggregating the separately collected samples of performance data for each of the functional verification tests and including an indicator in the merged samples of the functional verification test associated with the samples of performance data.

4. The method of claim 1, wherein filtering the merged samples of performance data into the subset of the merged samples of performance data is performed based, at least in part, on which functional verification test is associated with the samples of performance data identified in the profile filtering commands.

5. The method of claim 1, wherein generating the profile presentation further comprises: identifying design elements in the subset of the merged samples of performance data; and building call trees having a hierarchical structure for each of the types of the identified design elements, wherein each of the call trees includes counter values associated with a presence of the design elements in the subset of the merged samples of performance data.

6. The method of claim 1, wherein collecting the samples of the performance data is performed periodically during the functional verification of the circuit design or in response to one or more events occurring during the functional verification of the circuit design.

7. The method of claim 1, wherein the samples of the performance data include one or more of a simulation time, a design instance of the circuit design being executed, a design unit of the circuit design being executed, a function of the circuit design being executed, a call stack of the functional verification, or a code type for the circuit design.

8. An apparatus comprising at least one computer-readable memory device storing instructions configured to cause one or more processing devices to perform operations comprising: simulating a circuit design describing an electronic device with a plurality of functional verification tests; separately collecting samples of performance data during simulation of the circuit design for each of the functional verification tests; merging the samples of performance data from each of the functional verification tests; filtering the merged samples of performance data into a subset of the merged samples of performance data based, at least in part, on profile filtering commands; and generating a profile presentation including the subset of the merged samples of performance data, wherein the profile presentation, when displayed, is configured to annunciate portions of the circuit design corresponding to at least one performance hotspot.

9. The apparatus of claim 8, wherein separately collecting the samples of performance data during simulation of the circuit design for each of the functional verification tests further comprises generating test databases for each of the functional verification tests, and wherein the merging of the samples of performance data from each of the functional verification tests further comprises merging the test databases into a regression database including samples of performance data for all of the functional verification tests.

10. The apparatus of claim 9, wherein merging the samples of performance data from each of the functional verification tests includes aggregating the separately collected samples of performance data for each of the functional verification tests and including an indicator in the merged samples of the functional verification test associated with the samples of performance data.

11. The apparatus of claim 8, wherein filtering the merged samples of performance data into the subset of the merged samples of performance data is performed based, at least in part, on which functional verification test is associated with the samples of performance data identified in the profile filtering commands.

12. The apparatus of claim 8, wherein generating the profile presentation further comprises:identifying design elements in the subset of the merged samples of performance data; and building call trees having a hierarchical structure for each of the types of the identified design elements, wherein each of the call trees includes counter values associated with a presence of the design elements in the subset of the merged samples of performance data.

13. The apparatus of claim 8, wherein collecting the samples of the performance data is performed periodically during the functional verification of the circuit design or in response to one or more events occurring during the functional verification of the circuit design.

14. The apparatus of claim 8, wherein the samples of the performance data include one or more of a simulation time, a design instance of the circuit design being executed, a design unit of the circuit design being executed, a function of the circuit design being executed, a call stack of the functional verification, or a code type for the circuit design.

15. A system comprising: a memory system configured to store computer-executable instructions; and a computing system, in response to execution of the computer-executable instructions, is configured to: simulate a circuit design describing an electronic device with a plurality of functional verification tests;separately collect samples of performance data during simulation of the circuit design for each of the functional verification tests; merge the samples of performance data from each of the functional verification tests; filter the merged samples of performance data into a subset of the merged samples of performance data based, at least in part, on profile filtering commands; and generate a profile presentation including the subset of the merged samples of performance data, wherein the profile presentation, when displayed, is configured to annunciate portions of the circuit design corresponding to at least one performance hotspot.

16. The system of claim 15, wherein the computing system, in response to execution of the computer-executable instructions, is further configured to: separately collect the samples of performance data during simulation of the circuit design for each of the functional verification tests by generating test databases for each of the functional verification tests, and merge of the samples of performance data from each of the functional verification tests by merging the test databases into a regression database including samples of performance data for all of the functional verification tests.

17. The system of claim 16, wherein the computing system, in response to execution of the computer-executable instructions, is further configured to merge the samples ofperformance data from each of the functional verification tests by aggregating the separately collected samples of performance data for each of the functional verification tests and including an indicator in the merged samples of the functional verification test associated with the samples of performance data.

18. The system of claim 15, wherein the computing system, in response to execution of the computer-executable instructions, is further configured to filter the merged samples of performance data into the subset of the merged samples of performance data based, at least in part, on which functional verification test is associated with the samples of performance data identified in the profile filtering commands.

19. The system of claim 15, wherein the computing system, in response to execution of the computer-executable instructions, is further configured to generate the profile presentation by: identifying design elements in the subset of the merged samples of performance data; and building call trees having a hierarchical structure for each of the types of the identified design elements, wherein each of the call trees includes counter values associated with a presence of the design elements in the subset of the merged samples of performance data.

20. The system of claim 15, wherein the computing system, in response to execution of the computer-executable instructions, is further configured to collect the samples of theperformance data periodically during the functional verification of the circuit design or in response to one or more events occurring during the functional verification of the circuit design.

Images