Interactive visual log file comparison

Abstract

The invention relates to interactive visual log file comparison. A method includes receiving first structured data extracted from a first log file generated by a first run of an electronic design automation process and second structured data extracted from a second log file generated by a second run of the electronic design automation process; determining, by a processing device, outputs of a first section of the first log file and a second section of the second log file corresponding to the same stage of the electronic design automation process based on the first structured data and the second structured data; extracting a first metric from the first section of the first log file and a second metric from the second section of the second log file; and generating a user interface to display the first metric from the first section of the first log file adjacent to the second metric from the second section of the second log file.

Description

Technical Field

[0001] The present disclosure relates to an analysis and user interface for visual comparison of log files. Background Art

[0002] Many software programs generate output in the form of log files. These log files are produced as a supplement to some primary output and are used to record events that occur during the operation of the software, for example, and other metadata associated with the process that generates the primary output. Users and software developers can review the log files to analyze the primary output of the software program, for example, in ways that identify ways to improve the quality of the primary output or identify the root cause of errors in the underlying software program.

[0003] The above information disclosed in this background section is only for enhancing the understanding of the present disclosure, and thus it may contain information that does not form the prior art already known to those of ordinary skill in the art. Summary of the Invention

[0004] Aspects of embodiments of the present disclosure relate to systems and methods for interactive visual log file comparison.

[0005] According to one embodiment of the present disclosure, a method includes: receiving first structured data extracted from a first log file generated by a first run of an electronic design automation process and second structured data extracted from a second log file generated by a second run of the electronic design automation process; determining, by a processing device, that a first section of the first log file and a second section of the second log file correspond to outputs of the same stage of the electronic design automation process based on the first structured data and the second structured data; extracting a first metric from the first section of the first log file and a second metric from the second section of the second log file; and generating a user interface to display the first metric from the first section of the first log file adjacent to the second metric from the second section of the second log file.

[0006] The first section may have a different position in the first log file than the second section has in the second log file, and the first structured data may be generated by parsing the first log file, including: identifying a first workflow step start marker at a first position in the first log file; and determining the position of the first section in the first log file based on the first position of the first workflow step start marker, and the second structured data may be generated by parsing the second log file, including: identifying a second workflow step start marker at a second position in the second log file; and determining the position of the second section in the second log file based on the second position of the second workflow step start marker.

[0007] The parsing of the first log file may be performed concurrently with the first run of the electronic design automation process.

[0008] The user interface may include a control configured to select one display mode from a plurality of display modes for displaying the second metric, the plurality of display modes including: the raw value of a second metric of the second metric; the percentage change between the raw value of the second metric and the corresponding metric of a baseline run; and the absolute change between the raw value of the second metric and the corresponding metric of the baseline run.

[0009] The user interface may be implemented using a hypertext document and a style sheet, nodes of the hypertext document corresponding to the second metric may include a plurality of child nodes storing the raw value, the percentage change, and the absolute change, and interaction with the control of the user interface may cause the user interface to modify the style sheet to make one of the child nodes visible and hide the other child nodes.

[0010] The user interface may include a control configured to select one run of a plurality of runs of the electronic design automation process as the baseline run.

[0011] The method may further include: receiving a plurality of first messages extracted from the first log file and a plurality of second messages extracted from the second log file; grouping the plurality of first messages and the plurality of second messages by message type to generate a first plurality of message groups from the first log file and a second plurality of message groups from the second log file; generating a summary of the first plurality of message groups and a summary of the second plurality of message groups, the summary of a message group including the message count in the group and a representative example message from the message group; and highlighting in the user interface the differences between a first summary of a first message group from the first log file and a second summary of a second message group from the second log file.

[0012] The first metric may include a first plurality of metric sub-tables, and the second metric includes a second plurality of metric sub-tables. The first plurality of metric sub-tables and the second plurality of metric sub-tables correspond to sub-stages of the same stage of the electronic design automation process, and the user interface may display metrics from the first plurality of sub-tables adjacent to corresponding metrics from the second plurality of sub-tables section by section in a hierarchy according to the order of the electronic design automation process.

[0013] According to one embodiment of the present disclosure, a system includes: a memory that stores instructions; and a processor coupled to the memory and executing the instructions, the instructions when executed causing the processor to: receive structured data extracted from a log file generated by a run of an electronic design automation process during an iteration of an integrated circuit design, the structured data including a plurality of sections corresponding to stages of the electronic design automation process; extract a plurality of metric sub-tables from the plurality of sections; and generate an interactive user interface report to display the plurality of metric sub-tables section by section in a hierarchy according to the order of the electronic design automation process.

[0014] A first sub-table of the plurality of sub-tables may include metrics of a first plurality of types of metric data, and a second sub-table of the plurality of sub-tables includes metrics of a second plurality of types of metric data different from the first plurality of types of metric data, and the interactive user interface report may include: a first portion that includes a first header identifying the first plurality of types of metric data and metrics from the first sub-table; and a second portion that includes a second header identifying the second plurality of types of metric data and metrics from the second sub-table.

[0015] The user interface may be configured to: maintain the display of the first header when any portion of the metrics from the first sub-table is visible in the user interface; and maintain the display of the second header when any portion of the metrics from the second sub-table is visible in the user interface.

[0016] The first sub-table may include metrics written to the log file during a first stage of the electronic design automation process, the second sub-table may include metrics written to the log file after a first plurality of metrics of the first sub-table and before a second plurality of metrics of the first sub-table during the first stage, and the second sub-table may split the first sub-table in the interactive user interface report.

[0017] The interactive user interface report may further include: a third portion that includes the first header identifying the first plurality of types of metric data and additional metrics from the first sub-table, and the second portion of the metrics from the second sub-table may be displayed in the user interface between: the first portion that includes the metrics from the first sub-table; and the third portion that includes the additional metrics from the first sub-table.

[0018] The user interface may be configured to: maintain the display of the first header in the first portion when any of the metrics from the first sub-table are visible in the user interface; maintain the display of the first header in the third portion when any of the additional metrics from the first sub-table and the second sub-table are visible in the user interface; and maintain the display of the second header in the second portion when any of the metrics from the second sub-table are visible in the user interface.

[0019] The memory may further store instructions that, when executed, cause the processor to: receive second structured data extracted from a second log file generated by a second run of the electronic design automation process during a second iteration of the integrated circuit design, the second structured data including a second plurality of sections corresponding to the stages of the electronic design automation process; extract a second plurality of metric sub-tables from the second plurality of sections of the second log file; and determine a correspondence between the plurality of sections of the structured data and the second plurality of sections of the second structured data, and the interactive user interface report may further display the metrics from the second plurality of metric sub-tables adjacent to the corresponding metrics from the plurality of metric sub-tables.

[0020] According to an embodiment of the present disclosure, a non - transitory computer - readable medium contains stored instructions that, when executed by a processor, cause the processor to: receive first structured data extracted from a first log file generated by a first run of an electronic design automation process during a first iteration of an integrated circuit design, the first structured data including a first plurality of sections corresponding to stages of the electronic design automation process; receive second structured data extracted from a second log file generated by a second run of the electronic design automation process during a second iteration of the integrated circuit design, the second structured data including a second plurality of sections corresponding to the stages of the electronic design automation process; generate an interactive user interface report to hierarchically display, in the order of the electronic design automation process: a first metric of the first plurality of sections from the first structured data; and a second metric of the second plurality of sections from the second structured data, adjacent to the first metric of the corresponding section in the first plurality of sections from the first structured data.

[0021] The interactive user interface report may include user interface controls to switch between an expanded view and a collapsed view of a portion of the interactive user interface report that displays metrics of a first section from the first log file and a corresponding second section from the second log file, the first section and the corresponding second section corresponding to the same stage of the electronic design automation process, the expanded view may display a first raw value of the first section from the first log file and a second raw value of the corresponding second section from the second log file, and the collapsed view may display a plurality of first summary metrics calculated from the first raw value and a second summary metric calculated from the second raw value.

[0022] The interactive user interface report may highlight a second metric among the second metrics that differs in value from a corresponding first metric among the first metrics by at least a threshold.

[0023] The interactive user interface report may highlight a second non - numerical metric among the second metrics that is different in value from a corresponding first non - numerical metric among the first metrics.

[0024] The first metric may include a first plurality of metric sub - tables, and the second metric may include a second plurality of metric sub - tables, the first plurality of metric sub - tables and the second plurality of metric sub - tables corresponding to sub - stages of the same stage of the electronic design automation process, and the interactive user interface report may display metrics from the first plurality of sub - tables adjacent to corresponding metrics from the second plurality of sub - tables in a hierarchy in the order of the electronic design automation process, section by section. Brief Description of the Drawings

[0025] The present disclosure will be more fully understood from the detailed description given below and the accompanying drawings of the embodiments of the present disclosure. The drawings are used to provide knowledge and understanding of the embodiments of the present disclosure and do not limit the scope of the present disclosure to these specific embodiments. In addition, the drawings are not necessarily drawn to scale.

[0026] Figure 1 is a flowchart depicting a method for processing log files and generating a comparison between log files according to an embodiment of the present disclosure.

[0027] Figure 2 is a flowchart depicting method 200 for processing log files according to an embodiment of the present disclosure.

[0028] Figure 3 is a schematic depiction of a layout workflow phase using incremental log file capture according to an embodiment of the present disclosure.

[0029] Figure 4A is a screenshot of a part of a user interface for selecting log file data according to an embodiment of the present disclosure.

[0030] Figure 4B is a screenshot of a part of a user interface according to an embodiment of the present disclosure, which shows data extracted from log files corresponding to two different runs.

[0031] Figure 4C is a screenshot of a part of a user interface according to an embodiment of the present disclosure, which shows a part of a report of data extracted from log files corresponding to numerical metrics representing the quality of output from two different runs.

[0032] Figure 4D is a screenshot of a part of a report highlighting differences in non - numerical data according to an embodiment of the present disclosure.

[0033] Figure 4E is a screenshot of a part of a user interface according to an embodiment of the present disclosure, which shows three different modes for displaying a comparison of numerical metrics representing the quality of output from two different runs.

[0034] Figure 4F is a screenshot of a part of a user interface of a report for presenting different phases and sub - phases of a workflow organized in a hierarchy according to an embodiment of the present disclosure.

[0035] Figure 4G contains a screenshot of a part of a user interface showing summary metrics and expanded detailed metrics according to an embodiment of the present disclosure.

[0036] Figure 4H An example of a part of a user interface that shows messages generated by different commands or software programs during different sub - stages of a workflow.

[0037] Figure 4I A part of a user interface that depicts a part of a report with multiple sub - tables according to an embodiment of the present disclosure.

[0038] Figure 4J An example showing sub - table headers of a sub - table displayed in a part of a user interface before and after scrolling according to an embodiment of the present disclosure.

[0039] Figure 4K Another example showing the display of a header row of a sub - table that is maintained while being split by another sub - table according to an embodiment of the present disclosure.

[0040] Figure 4L Depicts the display of a sub - table of a report when a first primary - measure sub - table and a second secondary - measure sub - table of a sub - table are displayed according to an embodiment of the present disclosure.

[0041] Figure 5 A flowchart of a method for depicting an interactive report for displaying metrics from a log file of a computational workflow process, where the metrics are hierarchically organized based on the stages of the computational workflow process according to an embodiment of the present disclosure.

[0042] Figure 6 A flowchart depicting various processes used during the design and manufacture of an integrated circuit according to some embodiments of the present disclosure.

[0043] Figure 7 A diagram depicting an example computer system in which embodiments of the present disclosure may operate. Detailed Description

[0044] Aspects of the present disclosure relate to interactive log - file comparison.

[0045] Software programs generate record outputs when executed and save these outputs to a log file (also referred to as a logfile). These log files can be in plain - text form, for example, represented in a character encoding (such as American Standard Code for Information Interchange (ASCII)) or Unicode (e.g., Unicode Transformation Format, such as UTF - 8).

[0046] In some contexts, an electronic design automation (EDA) software program for developing integrated circuit (IC) designs generates a log file that records, for example: the sequence of steps or commands run by the software program; the configuration of those steps or commands (e.g., user-specified parameters or parameters automatically set based on inputs); the result of each step (e.g., result quality or QOR metric or other numerical metric representing the quality of the output, performance metrics such as the run time and memory usage when executing the step, and the like); and errors or warnings generated during the run. (An example of an EDA process is described in more detail below with reference to Figure 6 ).

[0047] However, the log file generated by an EDA software program can be hundreds of thousands to millions of lines of text for a given execution (or run), corresponding to a single version of an integrated circuit design being generated. These log files contain information generated through many nested and overlapping relationships between the steps and command sequences run by a given program, but may also lack structure (e.g., a formal hierarchy). Additionally, the log files can contain thousands of repeated warnings or errors, which can obscure other warnings and errors that may be more relevant. The large size and complex relationships of the data in these log files are factors that make it impractical for a human engineer to manually review the log file for a given run, let alone compare the log files from two or more different runs of the program. The large size and complexity of the log files also make it difficult for a user to understand the output of text comparison software (e.g., diff), and thus manual review can involve opening multiple log files side by side in two different windows of a text editor (in a text-based terminal interface or a graphical user interface), then manually searching the text of the log files and scrolling the windows to find the values to be compared between different log files.

[0048] These challenges make it difficult for a user to compare the results from two or more different executions of an EDA process to generate different versions or iterations of an integrated circuit design, such as exploring the quality tradeoffs of results due to changing some configuration parameters or making some different choices in the design. Manually reading log files to compare results is a skill that some engineers develop over time, so inexperienced users may be completely unable to interpret the log files and may rely on the help of more experienced engineers. However, the size and complexity of these log files can even mislead experienced users (e.g., engineers). Additionally, the metrics used for comparison (e.g., result quality metrics) are distributed across hundreds of thousands to millions of lines of the log file and can be located at different positions in different log files from different runs (e.g., due to different numbers of previous lines of text), and determining whether one metric is better than another is generally a complex multi-factor analysis involving data from different parts of the log file.

[0049] Accordingly, aspects of embodiments of the present disclosure relate to automatically analyzing log files to extract data and providing a user interface to visually compare heterogeneous, time-ordered log files based on those analyses. The log files are heterogeneous because different parts or sections of the log file have different formats, corresponding to the outputs of different commands or tools. The log files are time-ordered because the data in the log file appears in the order of the steps or commands of the execution process (e.g., in the order of the sub-stages or steps executed during the run of an EDA process for generating an integrated circuit design). Some aspects of embodiments of the present disclosure relate to automatically analyzing multiple log files to identify corresponding sections (e.g., sections from two different log files containing the output of the same step in an EDA process), and then presenting the corresponding sections of the log files together such that corresponding data values (e.g., some corresponding result quality metrics of aspects of an integrated circuit design) from different log files can be easily compared. Different sections of the log file may have different formats, and thus some aspects of embodiments relate to using specialized parsers to extract data from the corresponding sections. Aspects of embodiments of the present disclosure further relate to visually differentiating parts of the log file that are identified as more significant or important to the user through automatic analysis (e.g., identifying differences in warning or error messages generated in two different log files, suppressing or collapsing or summarizing repeated error messages into representative instances, and the like).

[0050] Technical advantages of embodiments of the present disclosure include, but are not limited to, automatically generating comparisons of log files from different runs of an electronic design automation (EDA) process for generating an integrated circuit design. Some aspects of embodiments of the present disclosure relate to detecting corresponding sections and corresponding data values (e.g., result quality metrics) between two or more different log files and presenting those corresponding data values adjacent to each other in a user interface, thereby allowing a user to easily and visually compare the results of different runs.

[0051] This improves the process for designing integrated circuits and other similar processes involving large amounts of heterogeneous, time-ordered log files by providing an easily understandable representation of the differences between different log files to the user. This improves the EDA process (or other processes) by revealing potential trade-offs between the results of different runs, without requiring an engineer to search through thousands or millions of lines of log files. This time savings and cost savings reduce the turnaround time and expenses associated with designing integrated circuits (or other engineering processes), thereby allowing for faster and higher-quality development of such products.

[0052] Aspects of embodiments of the present disclosure relate to methods for analyzing multiple log files and generating representations of data contained in the multiple log files, where the depiction of these representations in a user interface allows for easy comparison of the log files. These representations include simplified access to log file data, visual markers for identifying shifts in result quality metrics between runs, hierarchical representations of processes recorded in the log files (e.g., hierarchical representations of stages and sub-stages of an EDA process), simplified capture of automatically generated messages (e.g., informational, warning, and error messages), organization of data into sub-tables to clarify connections between different tool commands and engines, dynamic display of relevant table or sub-table headers, summary views of data in the log files, reduction of clutter by collapsing or separating frequently used metrics from infrequently used metrics, incremental capture and analysis of log files during a run, separate customizable log file parsers for different sections of heterogeneous log files, prescriptive guidance based on log file traces, and high-performance re-rendering of the user interface when using a web browser or web browser engine as a framework to provide a front-end user interface.

[0053] Figure 1 is a flow chart depicting a method 100 for processing log files and generating comparisons between log files according to one embodiment of the present disclosure. Method 100 may be executed by a computer system, such as the computer system 700 described below with respect to Figure 7 For example, the method may be implemented using program instructions stored in non-volatile or non-transitory memory of the computer system 700 and executed by a processing device (e.g., a microprocessor, a central processing unit, or the like), where the instructions configure the computer system 700 as a dedicated device for performing the methods according to embodiments of the present disclosure. Although Figure 7 depicts a single computer system, embodiments of the present disclosure are not limited thereto and may be implemented in a manner distributed across different computers (e.g., where different parts of the method are executed by different computer systems and / or where multiple computer systems perform the same steps on different data simultaneously).

[0054] As Figure 1 shown, at 110, a processing device parses heterogeneous time-ordered log files generated by multiple runs of a computational process to generate structured data including different sections corresponding to different stages of the computational process. For ease of description, aspects of embodiments of the present disclosure are presented in the context of an electronic design automation (EDA) process for designing an integrated circuit based on an input specification (e.g., expressed in a hardware description language). Examples of EDA processes are described in more detail below with respect to Figure 6 However, embodiments of the present disclosure are not limited thereto and may be applied to the analysis of log files generated by runs of a computational workflow.

[0055] Different stages or subprocesses of an EDA process can use different software programs to perform operations, where the output of one stage is provided as the input to subsequent stages in the EDA process. As an example, Figure 6 Show the layout or physical implementation stage 624. This stage can include various sub-parts, including initial layout, initial design rule check (DRC), initial optimization, final layout, and final optimization. During and between the execution of these different sub-parts, the software program records information about the execution of the stage on the integrated circuit design. For example, the initial DRC sub-part can generate messages about the initial layout (placement) of the circuit structure in the layout of the integrated circuit design, such as warnings and errors. As another example, the initial optimization and final optimization sub-parts can each be followed by a quality of results (QOR) analysis, which measures the quality of the layout and routing of the connections (wires) between circuit structures as a numerical value. These quality metrics include, for example: congestion metrics (e.g., routing overflow, layout congestion, global congestion, local congestion, and the like); and timing metrics (e.g., total negative timing slack, worst hold timing slack, and the like).

[0056] The different stages and sub-stages of the EDA process can be appended to the same log file or write data to different log files. The different software programs executed in different stages can record text data in the log file in a data format specific to those software programs, such that the resulting log file may lack an overall structure, but the data is time-ordered as the data appears in the order of operations performed on the integrated circuit design. For example, the quality of results analysis software program can be executed after the initial layout, initial optimization, final layout, and final optimization sub-stages of the physical implementation stage. The quality of results metrics generated after each sub-stage can have the same format, which can make it difficult for the user to determine which sub-stage a given set of quality of results metrics is associated with, and thus difficult to compare the quality of results metrics between different runs of the EDA process on the relevant integrated circuit design, especially since the number of other lines in the log generated by the sub-stage can vary. In addition, even if the user determines that there is a difference in the quality of results metrics between two different runs, the lack of structure in the log file makes it difficult to identify the reason for the difference in the values of the quality of results metrics.

[0057] Accordingly, at 110, the processing device analyzes the heterogeneous time-ordered log file to generate structured data by identifying sections written by different software programs during different stages of the EDA process and extracting data metrics from the different sections based on the data format applied by the corresponding software program to generate structured data representing the input log file.

[0058] Figure 2It is a flowchart depicting a method 200 for processing a log file according to an embodiment of the present disclosure. In some embodiments of the present disclosure, a two-pass method is used to parse the log file to capture all relevant data. In the first pass, at 210, the processing device runs a quick parse to identify different categories of log file data and their locations within the log file (e.g., the offset from the start of the log file in terms of bytes or number of lines). In some embodiments, these categories include workflow step marker categories, information message categories, and sub-table identifier categories.

[0059] A workflow step marker is a message that marks the start and end of the most important steps in the overall process workflow (e.g., for example, for the stages of the EDA process described). Specifically, it is expected that these messages are written to the log file by the software program at the start and end of their execution, thereby marking different sections of the log file. Figure 6 An information message is a single-line information (info), warning, and error message that conveys information about the configuration of the command or step or stage or sub-stage being executed and whether any errors occurred during the operation of the corresponding part of the process. The lines of the log file containing these messages can start with prefixes such as INFO, WARNING, ERR, and the like. However, in some embodiments of the present disclosure, the parser can be configured to detect other keywords or identifiers used by the software program to mark the logged messages.

[0060] A sub-table identifier marks the start (and in some cases, the end) of data that is logged and specific to a sub-component of the data output by a stage, sub-stage, command, or software program. In some embodiments, these sub-tables correspond to the main stages of the overall computing process. In the EDA process, these different sub-tables can correspond to optimization, global placement, legalization, high-fanout synthesis, multi-bit banking, etc., all of which can output data in different formats. Additionally, a sub-stage can have multiple sub-tables. For example, the detailed routing step can include multiple sub-tables, such as a routing summary sub-table, a design rule violation category sub-table, a wire length sub-table, and the like, all of which output different data in different formats.

[0061]

[0062] ​In some embodiments, portions of the input log files corresponding to these different categories are determined based on parsing analysis, such as by using pattern matching (e.g., patterns specified using regular expressions) to classify individual lines of the input log files into the various categories described above. In some embodiments, this analysis is performed on a line-by-line basis, which reduces runtime and memory overhead (e.g., because the parser only needs to analyze the text up to the next new line or end-of-line (EOL) or line feed or carriage return and line feed (CRLF) character in the log file), enabling high-speed processing of the log file during the first pass at 210, which is notable in cases where the log file can have hundreds of thousands to millions of lines of text (e.g., log files generated by running an EDA process).

[0063] At 230, the processing device begins a second pass of the input log file by dividing the input log file into segments based on the positions of the workflow step markers determined at 210. As mentioned above, the workflow steps may generate entries or lines in the log file marking the start of where they log information to the log file (e.g., workflow step start markers), and may also include entries or lines marking the end of where they log information to the log file (e.g., workflow step end markers). Thus, at 230, the processing device divides the log file into segments based on the positions of these workflow start and end markers. Additional lines may appear between these segments corresponding to workflow steps (e.g., lines may appear between the end marker of the first workflow step and the start marker of the next workflow step in a workflow process), and these additional lines between workflow steps may also be grouped into their own segments. In various embodiments of the present disclosure, positions within the log file may be specified based on, for example, line numbers (e.g., the number of new line characters between the start of the file and the position) or the number of bytes from the start of the file.

[0064] At 250, the processing device applies a sub-table parser to each identified sub-table in the input log file. A given segment in the log file may contain one or more sub-tables, or may not contain sub-tables, depending on the behavior of the software program that generates the output for the segment. As mentioned above, different software programs may record data in the log file in different formats. For example, different software programs may use different identifiers for the same metric (e.g., abbreviations vs. full names) or may not use identifiers at all (e.g., a list of values separated by, for example, commas, slashes, colons, or the like between different values). Additionally, this data may span multiple lines, making it difficult to separate this data on a line-by-line basis alone. Some software programs may generate tables using plain text, in which case the columns or rows of the table may specify labels associated with the text.

[0065] Accordingly, at 250, the processing device uses multiple independent parsers to capture this sub-table data. It is not necessary to re-check the entire log file during this second pass. Instead, in some embodiments, the processing device looks for (e.g., moves to) the sub-table identifier points identified during the first pass at 210 and invokes the relevant parser dedicated to parsing the data sub-table (e.g., designed to parse the data expected to appear in this section of the log file), where the parser terminates at the end of the log file sub-table data.

[0066] At 270, the processing device generates structured data representing the input log file data based on the sections, information messages, and parsed sub-tables. The dedicated parser extracts data values (e.g., metrics output by a software program for different workflow stages) from the text representation in the log file and converts these into structured semantic data in a text data format (e.g., comma-separated values (CSV), tab-separated values (TSV), JavaScript Object Notation (JSON), Extensible Markup Language (XML), yet another markup language (YAML), or the like) or a binary data format (e.g., binary JSON or BSON, MessagePack, or the like), such that the data values are easily loaded for comparison and display in a user interface, as will be discussed in more detail below. Additionally, the information messages are collected for analysis as a group, as will be discussed in more detail below.

[0067] The structured data generated by the processing device at 270 may include separate structured data for each category of log file information. In some embodiments, the structured data is stored as separate structured data files (e.g.,.csv and.json files), and in some embodiments, the structured data extracted by different parsers is stored in separate files (e.g., a dedicated parser for parsing result quality metrics stores the data extracted from the input log file in the QoR_heartbeat.csv file, a dedicated parser for parsing global router metrics stores the data extracted from the input log file in the global_router.csv file, and a dedicated parser for parsing legalizer metrics stores the data extracted from the input log file in the legalizer.csv file). In some embodiments, the positions of the parsed sub-sections, information messages, and parsed sub-tables within the original input log file are stored in the structured data such that reports can display or directly link to the locations in the log file containing those metrics (see, e.g., Figure 4I column 486 in, which shows the line numbers corresponding to the locations in the log file containing the corresponding metrics).

[0068] Although Figure 2Describes a method for generating structured data from an input log file in batch mode or post - processing mode (e.g., after the completion of the run of a workflow process (e.g., an EDA process) and when the input log file is static), but embodiments of the present disclosure are not limited thereto, and also include embodiments of automatically incrementally capturing data from a log file when a log file is generated by a computational workflow process (e.g., when an EDA workflow process such as a compiler is running). For example, in some embodiments, log file data is incrementally extracted at the end of each important step in the workflow, and log file data is incrementally extracted again when the workflow is closed.

[0069] As a non - limiting example of incremental capture of a log file, Figure 3 is a schematic depiction of a layout workflow phase 300 using incremental log file capture according to some embodiments of the present disclosure. As Figure 3 shown, the layout workflow phase 300 starts at 310 and includes various sub - parts or steps, including an initial layout 320, an initial design rule check (DRC) 330, an initial optimization 340, a final layout 350, and a final optimization 360, along with a workflow phase end step 370 (e.g., which may perform additional cleanup or generate additional record information).

[0070] An incremental recorder 305 according to an embodiment of the present disclosure operates concurrently with the layout workflow phase 300, and after each phase, for example, at 325 between the initial layout 320 and the initial design rule check (DRC) 330, at 335 between the initial design rule check (DRC) 330 and the initial optimization 340, at 345 between the initial optimization 340 and the final layout 350, at 355 between the final layout 350 and the final optimization 360, at 365 after the final optimization 360, and at 375 after the workflow phase end step 370, data is incrementally captured from the log file. Once captured, the corresponding captured portion of the log file can be immediately processed (e.g., using the two - pass method described above with respect to Figure 2 ), without waiting for the remaining sub - phases or parts of the layout workflow phase 300 to complete. Although Figure 3 depicts an example where incremental capture is applied to the layout workflow phase 300 of an EDA workflow, embodiments of the present disclosure are not limited thereto, and incremental capture can be applied to other computational workflows (whether or not related to EDA).

[0071] One technical advantage of incremental capture of log files over post - processing or batch processing of log files is reduced run - time for report generation. Log files can be large (typically hundreds of megabytes and sometimes gigabytes), which can take a long time to process each of these logs (e.g., even with efficient, optimized parsing, it may take about a minute to process). Any given run can contain multiple log files (e.g., 6 or more log files), and thus processing all log files across multiple runs can take, for example, 15 minutes to 1 hour, which can quickly increase engineering time. By incrementally capturing log files during the run of a computational workflow process, this parsing run - time is masked from the user (e.g., because the computational workflow process itself has a much longer run - time than log analysis, e.g., a single run of an EDA process can take hours to days). Additionally, in some embodiments, the incremental logger 305 executes as a concurrent process alongside the computational workflow process. In some embodiments, the incremental logger 305 executes on the same computer system as the computational workflow process (e.g., layout workflow stage 300), such as in a separate thread or operating system process and by reading from the log files written by the computational workflow process. In some embodiments, the incremental logger 305 executes on a computer system separate from the computational workflow process, such as where a first computer system executing the computational workflow process writes log data to a location readable by a second computer system running the incremental logger 305 (e.g., writing log data to a shared drive of the first computer system or to a shared network drive shared by the second or third computer system, such as a network file server).

[0072] Additionally, according to some embodiments of the present disclosure, incremental recording occurs within a workflow stage (e.g., within the software program executing the stage), such as implemented in a separate thread running on the same computer system or a separate thread running on a different computer system, and thus can capture additional data that may not otherwise be available in the log files, where the additional data can be specified by the incremental logger that captures data from a particular tool. In some embodiments, additional information about the execution environment of the software program is also captured and supplemented by this method.

[0073] The result of the incremental capture process is the same set of structured data extracted from the log files, as described above with respect to Figure 2 description.

[0074] Return reference Figure 1, at 130, the processing device determines corresponding sections of structured data from multiple different runs of a computational workflow process. Since the multiple different runs are based on the same computational workflow process, the number, type, and order of workflow stages between different runs are the same. Accordingly, it is expected that each of the heterogeneous time-ordered log files has sections corresponding to the logs generated by the same workflow stages. Additionally, embodiments of the present disclosure are not limited to cases where, for a given log file, all runs have the same type of data in the same order. For example, there may be step runs that exist in one run but not in other runs, or steps whose order has been switched between runs, or steps that are repeated in one run but not in another run. Thus, at 130, the processing device determines corresponding sections of structured data from the log files based on, for example, matching process workflow markers between different log files and based on, for example, the relative positions of sections in the log files (when those corresponding sections exist), and handles cases where specific data (e.g., sections) are available in less than all of the log files (e.g., showing blanks in some parts of the report where there is no corresponding available data for the run).

[0075] Reference Figure 3 As an example, two different runs of the same layout workflow stage 300 are expected to generate subsections related to an initial layout 320, an initial design rule check (DRC) 330, an initial optimization 340, a final layout 350, a final optimization 360, and an end of workflow stage 370. Since the length of each of these sections can vary depending on the run, the absolute positions of these sections within different log files can vary, but in some embodiments, the relative positions of these sections within the log files relative to each other will be maintained such that the processing device uses relative ordering to determine which sections correspond between runs. In some embodiments, as mentioned above, not every log file needs to contain every section (e.g., a section or step may be omitted or repeated a different number of times in different log files or in different positions). When there is a mismatch in the relative positions of sections or the number of steps in different log files, then in some embodiments, the workflow markers associated with the sections are used to identify the closest match between different log files. For example, if a given step appears once in a first log file and three times (e.g., three iterations) in a single line in a second log file, then the first occurrence of the step in the first log file can be identified as corresponding to one instance of the step in the second log file (e.g., the first instance of the step in the second log file). (As a specific example, a routing optimization step in an EDA workflow may be repeated multiple times to iteratively improve the network routing in an integrated circuit design, where the number of repetitions can depend on the quality of the results produced after each iteration.)

[0076] At 150, the processing device extracts metrics from corresponding sections of a structured data file. The processing device may also extract information messages from portions of the structured data file. As mentioned above, these metrics may include result quality metrics calculated during a periodic analysis of the current quality of results (which may be referred to herein as result quality heartbeat analysis) performed on results between different stages or sub-stages of a computational process workflow (a workflow in which the results are improved by each stage or sub-stage), summary metrics generated by workflow stages of the workflow, and the like.

[0077] At 170, the processing device generates a user interface to display metrics from corresponding sections of different ones of a heterogeneous time-ordered data file, where corresponding metrics from different runs are laid out adjacent to each other, thereby making it easy for a user to compare results of different runs. In some embodiments of the present disclosure, the user interface report displays a single log file without comparing multiple log files to each other (in such embodiments, determining corresponding sections of log files captured from different runs at 130 may be omitted).

[0078] Some aspects of embodiments of the present disclosure relate to simplified access to log file data, where a user may specify only the directory (folder) of the log file data once to generate a complete report. In these embodiments, the user interface provides an interface to open and access any log file data with relatively few selections. Figure 4A Is a screenshot of a portion of a user interface 400 for selecting log file data according to an embodiment of the present disclosure. Figure 4A Shows a list of types of log files 401 (e.g., log files from different workflow stages) that can be browsed on a left menu, and a dialog box 403 can be opened for selecting run data for viewing and comparison. Figure 4ASix runs of the selectable run data 403 are shown, labeled "run_timing_flow1", "run_timing_flow4", "run_power_flow2", "run_power_flow4", "run_area_flow7", and "run_congestion_flow2". Each run corresponds to a separate (e.g., independent) execution of a computational workflow, such as an EDA process for generating a layout of an integrated circuit (e.g., a set of masks for controlling semiconductor manufacturing equipment such as a lithography machine) based on an input integrated circuit design (e.g., expressed as a netlist), where any given run may generate multiple different log files 401 corresponding to different workflow stages. Different runs may produce different results due to changing the settings of various parameters of the computational workflow. Since the user interface 400 shows all available runs on the same screen, the user does not need to browse the file system to find the relevant log files between different runs. Instead, all comparable run files are displayed in a single screen. From the run selection dialog, the user can select one of the runs as a baseline run for comparison and select one or more additional runs as selected visible runs for comparison with the baseline run.

[0079] Figure 4B is a screenshot of a portion of a user interface 410 according to an embodiment of the present disclosure, which shows a report of data extracted from log files corresponding to two different runs. The runs shown in this user interface may correspond to runs selected using the Figure 4A user interface shown in. As Figure 4B shown in, the data from the runs is organized into sections, such as a global routing section 412, a placement tool foundation section 414, a legalization checker summary section 416, and a QOR heartbeat section 418. The different sections shown in the report provide a hierarchical representation or overview of the logged data collected during the steps or stages of the run in the process flow, since each step has an identified start and end point in the log file. Metrics from different runs are shown in adjacent columns, and different metrics within the same section are shown in different parts of the report in the user interface 410.

[0080] For example, in the global routing section 412, a first set of metrics relates to the overflow counts 412A for both horizontal and vertical directions (routing congestion metrics where the routing demand for the network through the local region exceeds the supply of tracks available for the network to be laid out), where the first run had 152 overflows in both directions and the second run had 154 overflows in both directions. The global routing section 412 also presents another set of metrics labeled GRC% 412B, as well as values extracted from the log files of the two runs. The reported hierarchy allows selective hiding or showing of metrics from different sections or subsections of the report, for example by selecting Figure 4B the controls 419 shown in to collapse or expand metric groups.

[0081] Aspects of embodiments of the present disclosure enable easy comparison of log files between any two runs of data loaded into the report. The comparison helps the user draw conclusions about the run results because many result quality metrics in integrated circuit design (such as power and area) do not have a specific target. Instead, the engineer seeks to obtain the best possible results without degrading other critical metrics. Viewing the results of two runs against each other provides this information to the user. Some aspects of embodiments of the present disclosure relate to providing visual markers to highlight significant offsets in the metrics. For example, green can be used to highlight improvements relative to the baseline, while red can be used to indicate degradation. Additionally, different degrees of shading can indicate the degree of change, with dark red or dark green indicating large changes and lighter shading indicating smaller changes. This shading draws the user's attention to important offsets in the result quality metrics, where the degree of importance is indicated by the degree of shading. As discussed in more detail below, in some embodiments, insignificant differences are de-emphasized in the report. Thus, the user does not need to search through different log files for corresponding numbers to see if there might be important differences. In the Figure 4B example shown, the values from the baseline run are shown in the left column of the metric, and the values from the comparison run are shown in the right column. In the column for the worst negative slack (WNS) metric in the QOR heartbeat section 418, the comparison run has a significantly improved WNS (0.196 vs. 0.322 for the baseline), and thus the value in the comparison run is highlighted.

[0082] Figure 4CA screenshot of a portion of a user interface 420 according to an embodiment of the present disclosure, showing a portion of a report of data extracted from a log file corresponding to numerical metrics representing the quality of outputs from two different runs. Values from the baseline run are shown in the left column of the metrics, and values from the comparison run are shown in the right column. In the column for the worst negative slack (WNS) metric 421, the comparison run has a significantly improved WNS (0.329 vs. 0.879 for the baseline), and thus the values in the comparison run are highlighted (e.g., with a diagonal in the background). However, in the leakage metric 423, the comparison run has a mix, where some values have significantly better leakage than the baseline (showing 159.76 leakage vs. 160.87 baseline for the NPO_START and PRE_C_4 rows), while other values have significantly worse leakage (161.48 vs. 159.45 baseline), and thus different shading is used to highlight the differences (e.g., with a dot pattern in the background).

[0083] In some embodiments, metrics showing relatively insignificant differences are de-emphasized in the user interface for reporting. Still referring to Figure 4C , in the first few rows (NPO_START, PRE_C_4, PRE_C_6, PRE_C_7_

[28] , and PRE_C_8_

[14] ), the total negative slack (TNS) metric 425 differs between the baseline (18.57) and the comparison run (17.83 or 16.95 in the respective rows). Since this difference is considered insignificant, in Figure 4CIn the example shown, the metric columns corresponding to the comparison runs (right - hand columns) are grayed out to de - emphasize those values in the TNS metric 425 and the run - to - run TNS (R2RTNS) metric 427. On the other hand, the last few rows (PRE_C_9, PRE_C_10, PRE_C_11, and PRE_C_12_[4]) show a significant improvement in TNS (baseline 18.21 versus comparison run 7.67). This de - emphasis of the metrics is used to mask false positives from being reported. In methods that use shading to indicate improvement or degradation, false - positive results can occur. For example, looking at the WNS (worst negative timing slack) timing, one run may have a negative timing slack of 1 picosecond (a small amount), and another run may have a negative timing slack of 3 picoseconds (another small amount). Naive shading based on the percentage change of these values would show that the second run degrades the result by 300% and would be marked as a large violation. To address this, aspects of embodiments of the present disclosure mask this as a small change and do not highlight it. This further helps ensure that visual markers of different colors (e.g., red and green) are meaningful and only highlight important differences. Determining whether a difference is a small or significant amount, and determining the degree of any difference, is specific to the characteristics of the metric (e.g., the value distribution of those metrics). Thus, when presenting the user interface, different calculations and different thresholds can be applied to different metrics.

[0084] Some aspects of embodiments of the present disclosure further relate to highlighting differences in non - numerical data (or text data) that appear in log files. Some of this data can convey information about software program configurations or the results of run - time workflow steps or commands. To emphasize these important changes, shading can be used in the report to highlight these non - numerical (or text) differences. Figure 4D is a screenshot of a portion 430 of a report highlighting differences in non - numerical data according to one embodiment of the present disclosure. In Figure 4DIn the example shown, in the baseline run, the workflow phase corresponding to a layout tool (e.g., a software program for laying out circuit portions in an integrated circuit design) can be configured to perform timing-driven layout using the automatic timing control feature 431, but a comparison test run can instead cause the layout tool to be configured to use the worst negative slack (WNS) drive method 433. In some cases, the software program can automatically select a configuration setting (e.g., timing-driven versus WNS-driven layout) based on input conditions (e.g., detecting a condition indicating that one configuration setting will result in better results than another). Finding differences such as these in a plaintext log file comparison is extremely difficult because this configuration setting can appear in a single line out of tens of thousands of other lines in the log file buried with the run of the layout tool workflow phase. However, such configuration differences can have a significant impact on the result quality metric between runs, and as shown in the darker background of the WNS drive setting 433 in the comparison run, highlighting this difference can help an engineer understand the reason for the timing metric differences.

[0085] In addition to presenting data values extracted from log files as shown in the example of Figure 4B some aspects of embodiments of the present disclosure relate to automatically displaying comparisons of values from different runs based on percent change and based on absolute change. Figure 4E is a screenshot of a portion 440 of a user interface according to one embodiment of the present disclosure, which shows three different modes or different formats for displaying a comparison of numerical metrics representing the quality of output from two different runs.

[0086] In the raw value mode 441 (or raw value format), the report shows the raw values of the result quality heartbeat metrics, including the total negative slack (TNS), register-to-register TNS (R2RTNS), and leakage extracted from the log file. For example, for the PRE_C_8_

[14] step, the baseline leakage is 159.45, and the comparison leakage is 164.18.

[0087] In the percent difference from baseline mode 443 (or percent difference format), the report shows the percent change from the raw values of the baseline run (left column) to the comparison run (right column). For example, the leakage value 164.18 in the comparison run is 3.0% higher than the baseline value 159.45, and thus the user interface displays the value of 3.0% for the comparison run.

[0088] In the absolute difference from baseline mode 445 (or absolute difference format), the report shows the value difference between the baseline value and the comparison run. For example, the leakage value 164.18 in the comparison run is 4.73 higher than the baseline value 159.45, and thus the user interface displays the value of +4.73 for the comparison run.

[0089] Automatically calculating these saves the user the effort of performing mental arithmetic or copying and pasting the values into a separate calculator (e.g., a handheld calculator or a calculator application running on a computer system), which reduces errors and allows for comparisons across reports rather than making comparisons based on a single metric each time.

[0090] Some aspects of embodiments of the present disclosure relate to a technique for switching between different display modes or different display formats of metrics. When using a web browser or an application built on a web technology framework (e.g., using an integrated web browser rendering engine) to display an interactive report according to embodiments of the present disclosure, the report may be represented using Hypertext Markup Language (HTML), styled with Cascading Style Sheets (CSS), and some additional interactivity may be provided by a scripting language such as JavaScript. Changing between display modes (e.g., between values, percentage differences from a baseline, and absolute differences from a baseline) involves replacing the values displayed in the HTML. One way to change these values would be to use a JavaScript function that is triggered when the user scrolls and updates the content of the newly rendered cell with the correct data (e.g., rendering the cell with the original value, percentage change, or absolute change). However, some browser engines may have poor performance, and running such a JavaScript function on thousands of cells in a report can result in poor performance.

[0091] Accordingly, some aspects of embodiments of the present disclosure relate to storing all possible content within the cells of a table and using CSS to selectively display only the correct content for the currently enabled mode while using CSS to hide the other content.

[0092] For example, a given cell of a table (e.g., Figure 4E the cell for the leakage value of the comparison run in the PRE_C_8_

[14] step shown in

[0093]

[0094] <div class＝”value”> 164.18

[0095] <div class＝”pctdiff”> 3.0％

[0096] <div class＝”absdiff”> +4.73

[0097]

[0098] Thus, all three pieces of content (value, percentage difference, and absolute difference) are stored in a node associated with that location in the report.

[0099] To show and hide different values, the CSS style sheet of the web page displaying the report is updated. For example, when the user enables the mode to display values (as in Figure 4E 441 of

[0100] .value {

[0101] display: visible;

[0102] }

[0103] .pctDiff{

[0104] display: none;

[0105] }

[0106] .absDiff{

[0107] display: none;

[0108] }

[0109] When the user enables the mode to display the percentage difference from the baseline (as in Figure 4E of 443), the computer system sets a part of the CSS style sheet as follows:

[0110] .value{

[0111] display: none;

[0112] }

[0113] .pctDiff{

[0114] display: visible;

[0115] }

[0116] .absDiff{

[0117] display: none;

[0118] }

[0119] When the user enables the mode to display the absolute difference from the baseline (as in Figure 4E of 445), the computer system sets a part of the CSS style sheet as follows:

[0120] .value{

[0121] display: none;

[0122] }

[0123] .pctDiff{

[0124] display: none;

[0125] }

[0126] .absDiff{

[0127] display: visible;

[0128] }

[0129] This updates the styles of all table cells and avoids calling JavaScript functions to operate on thousands of cells of the table individually, thereby improving the user interface rendering performance of large tables (e.g., reducing or avoiding jerky scrolling or slow changes in the display mode of the content presented in the user interface).

[0130] Some aspects of embodiments of the present disclosure relate to switching between different runs that are run as a baseline for comparison. For example, when initially comparing how a second run compares to a first run and then switching to reviewing the results of a third run, it may be more useful to use the second run as the baseline than the first run as the baseline. As described above Figure 4A shown in the example user interface of, a list of different run data 403 is presented, where radio buttons allow selection of exactly one of these runs as the baseline run. By accessing this part of the user interface, the user can select different runs as the baseline run while keeping the visible runs the same or different, and the user interface for displaying the report will automatically update to use the newly selected baseline run as the baseline for comparison (e.g., appearing in the leftmost value column).

[0131] As discussed above, the start and end positions of phases (workflow steps) are determined during the parsing process. Additionally, each phase or step of the workflow can have sub - phases or sub - steps enclosed or nested within it, such that the phases, sub - phases, sub - sub - phases, etc. of the workflow process have a hierarchical relationship. Representing the recorded data from runs of the workflow in a hierarchical report (e.g Figure 4B shown) provides a better understanding of the structure of the workflow and the relationships between the steps. Additionally, as mentioned above regarding Figure 4B the control 419 shown, sections of the hierarchy can be expanded or collapsed to allow the user to focus on the parts of the report that are most relevant to performing some specific analysis of the runs.

[0132] Figure 4F is a screenshot of a part 450 of a user interface for displaying a report of different phases and sub - phases of a workflow organized in a hierarchy. In the Figure 4F example shown, the report contains data from a compilation phase 452 (compile.log), a clock optimization and clock tree synthesis phase 454 (clock_opt_cts.log), a clock optimization phase 456, a routing phase 457, and a routing optimization phase 458, where these sections can be expanded (and made visible) or collapsed (and hidden) respectively by using controls 452A, 454A, 456A, 457A, and 458A. In the complete user interface for the report, the data for the corresponding phases can be similar toFigure 4B is displayed in the manner of the portion of the user interface shown in Figure 4F to the right of portion 450 shown in. The portions for reports for clock optimization and clock tree synthesis phase 454 and routing optimization phase 458 are expanded and visible, while the portions for reports for compilation phase 452, clock optimization phase 456, and routing phase 457 are hidden. A phase may include sub - phases (and sub - sub - phases, etc. in a tree structure or hierarchy), and the sub - phases may also be expanded or collapsed using corresponding controls (such as controls 454B, 454C, 454D, and 454E) provided that they have sub - phases.

[0133] Figure 4G A screenshot of a portion of a user interface that includes a summary metric and an expanded detailed metric according to an embodiment of the present disclosure. In the example described and shown above with respect to Figure 4B Some of the metrics shown in the report (such as the global routing overflow metric) are calculated as summaries of the detailed log outputs. For example, the routing metric may relate to all metal layers in an integrated circuit design. Figure 4G The summary table 460 shown in shows summary global routing metrics in two directions and individual metrics in the horizontal and vertical directions. This summary information reduces the amount of visual space consumed by the global routing sub - table in the user interface. However, in some cases, the user may want more detailed or verbose information at the detailed level at which the metrics appear in the log file. For example, the user may be interested in which metal layers have the largest number of congestion events. Thus, some aspects of embodiments of the present disclosure relate to providing user interface controls (such as a toggle button, a dropdown menu (dropdown menu / pulldown menu), or a multi - select interface such as a checkbox) such that the user can expand the summary data into verbose data, such as in Figure 4G the verbose table 461 shown in. The verbose table 461 contains a separate row for each metal layer (e.g., M2 to M13) in the integrated circuit design, and in some embodiments, the user interface provides a separate control for each row such that the user can toggle the visibility of any given row. In some embodiments, the verbose table 461 shows the raw values extracted from the log file. Thus, the report provides both a high - level view of the log file and functionality for the user to perform a more in - depth analysis of the execution details without having to open and search the original plain - text log file.

[0134] Log files can contain information (info), warnings, and error messages printed by commands or software programs during different stages of a workflow. In some examples, these messages constitute 50% or more of the total log file and often overwhelm users who attempt to manually read the log file. For example, when the workflow is for generating a design of an integrated circuit, the log file can contain 1,000 lines of warnings one after another, where the warnings are all the same but relate to different nets or cells in the IC design. In some cases, it is more useful for an engineer to understand the summary than to read each message in the log file. Thus, in some embodiments of the present disclosure, a processing device folds multiple duplicate messages into a summary of those duplicate messages. In some embodiments, determining whether two messages are variants of the same message or different messages is performed during parsing of messages generated by stages of a computational workflow, where different stages generate different types of output messages, and where a custom parser for the stage parses the messages into different message types (e.g., a warning about a potential timing violation versus a warning about a potential power violation). Thus, in some embodiments, the processing device summarizes the same type of messages generated during a given stage, where the summary presented in a report shows, for example: the existence of a given message, its location in the log file (e.g., during which stage or sub-stage command or engine call), a representative instance of the message, the number of times the message appears in the log file (e.g., the number of messages in a group of messages having the same message type), and how the message differs between two log files.

[0135] Figure 4H is an example of a portion 470 of a user interface that shows messages generated by different commands or software programs during different sub-stages of a workflow. In Figure 4H the example shown, the messages are colored (or color-coded) according to severity (e.g., in a color interface, red indicates an error, orange indicates a warning, and blue indicates information). For each section of the log file, a representative instance 471 of the message is captured and displayed, as well as a count 473 of the total number of times the message appears in that section of the log file for each run (e.g., a count 473A for a baseline run and a count 473B for a comparison run).

[0136] In some embodiments, in a manner similar to that described above with respect to Figure 4B and Figure 4COther ways of measuring described highlight significant changes in message counts. In some embodiments, a significant change includes a message occurring zero times in one run but a non-zero number of times in another run, or when the message occurs in both runs but there is a large change in the message count. For example, a warning that occurs five times in one run but a thousand times in another run could be a highlighted significant change. The threshold for determining the significance of the difference can be based on a function (e.g., determining that the change in message count differs by at least one order of magnitude). In some embodiments, the threshold for determining whether a change is significant is based on the type of warning message.

[0137] Some aspects of embodiments of the present disclosure relate to determining whether the message body has changed between runs. The specific part of the message indicating a significant difference is specific to the content of the log file presented in the report. For example, in the case of a log file from an EDA process for integrated circuit design, when considering a warning about a particular net, a change in the net name between runs may not be significant because the net name will almost certainly have changed between the two runs. On the other hand, another message indicating whether the optimization is performed in timing mode or total power mode could be significant because it will affect the entire flow trajectory and behavior of the optimization phase. Thus, some aspects of embodiments of the present disclosure relate to providing an interface for defining custom rules for specifying how to compare messages to determine whether the difference is significant (and thus should be highlighted) or not significant (and thus should be downplayed).

[0138] In some embodiments, when a significant change in a message is detected, the report highlights the message and may display both a representative message (e.g., from a baseline run) and another message (e.g., from a comparison run) determined to be significantly different by the custom rules.

[0139] Some aspects of embodiments of the present disclosure relate to a user interface for displaying these messages when comparing log files. The log file contains many types of messages, and even in a summary view of the report as shown in Figure 4B and Figure 4H there may be many messages to review. To address this issue, some aspects of embodiments of the present disclosure relate to suppressing the display of all messages that appear to be the same between two runs (e.g., between a baseline run and a comparison run). Thus, when operating in this mode, the report only shows messages whose count has changed significantly, messages that appear in one log file but not in the other, and informational messages whose body has changed significantly. For this view, in some cases, 95% or more of the messages will be hidden (and can be turned on by changing the settings), so that the user can focus on the messages that are different between runs.

[0140] Some aspects of embodiments of the present disclosure relate to the display of data sub-tables. As mentioned above regarding the parsing of data sub-tables, different stages and sub-stages or commands of a workflow process can record metrics into log files with different types of data. Displaying log file data is challenging because many different types of metric data corresponding to different sub-tables are to be displayed (e.g., outputs of different stages of a workflow process, e.g., in the case of an EDA process, rough placement tool, legalization checker, optimization, global / orbit / detail routing, high fan-out synthesis, clock tree synthesis, CCD (useful skew), multi-bit and the like). Generally, displaying this data in a table facilitates associating metrics with a given stage or sub-stage and facilitates comparison of metrics between runs. However, combining different tables with different types of data poses formatting challenges.

[0141] Accordingly, aspects of embodiments of the present disclosure relate to using different formatting for each sub-table presented in a report based on its content. Formatting customization for a given sub-table can include the number of columns, column headers / titles, column widths, and style selections (how to color cells based on differences in values, how to align values, whether values wrap in the case of a narrow column, and the like). These sub-tables are displayed chronologically based on their order of appearance in the log file. In some cases, sub-tables can be nested within other sub-tables (e.g., where a sub-command or sub-stage runs within a stage).

[0142] Figure 4I Depicted is a portion 480 of a user interface depicting a portion of a report having multiple sub-tables according to one embodiment of the present disclosure. Figure 4I Examples shown include five different sub-tables: QOR Heartbeat 481, Global Routing 482, Placement Tool Basics 483, Legalization Checker Summary 484, and Legalization Checker Large Displacements 485. Each of these sub-tables is presented chronologically based on the appearance of its data in the log file. The QOR Heartbeat sub-table 481 is split by other sub-tables and continues as the QOR Heartbeat table 481A after the Legalization Checker Large Displacements sub-table 485. Each sub-table has significantly different formatting requirements. For example, QOR Heartbeat 481 is all numerical data displayed in compact columns (e.g., worst negative slack (WNS), total negative slack (TNS), run-to-run total negative slack (R2RTNS), hold total negative slack (HTNS), and leakage, as Figure 4I(shown in). In contrast, both the layout tool base sub-table 483 and the legalizer checker large displacement sub-table 485 contain a large amount of text rather than numerical data that should be left-aligned, and coloring can be used to highlight differences, such as changing from "auto timing control" to "WNS-driven" highlighted by shading in the timing mode. The legalizer checker large displacement sub-table 485 has fields that contain very long cell names and can therefore be configured to be shown in wider columns that can wrap to multiple lines. Some sub-tables can share common columns with the same formatting (color, column width, alignment, etc.). Figure 4I The example shown in contains a "Line" column 486 that shows the line numbers of the log files from which the data is derived. This is information common to all sub-tables because they all contain data exported from log files. Thus, some aspects of the present disclosure further relate to data columns that additionally contain data that is commonly formatted across different sub-tables, although the formatting of each sub-table is different.

[0143] Therefore, Figure 4I Parts of the user interface shown in present different types of data (e.g., different sections of the log file from a baseline run) adjacent in time sequence to corresponding data from another log file (e.g., corresponding different sections of the log file from a comparison run). This enables the user to connect the behaviors of the workflow processes during these different runs. Additionally, in some cases, some runs may contain phases or steps that are omitted from other runs. In such cases, the rows may be filled only with data from the run that contains the phase or step, while the columns for the runs that omit the phase or step are blank. This provides the user with information about the workflow differences between the runs (differences in the phases or steps that are executed), where those differences can affect the final results of those runs.

[0144] For example, the results generated during one phase of a workflow have an impact on the next phase of the workflow. Embodiments of the present disclosure provide a concise and properly formatted summary of what was done during each workflow phase in chronological order in a single report, thereby making it easier for a user to understand how the workflow process generates the complete scenario of its results. For example, in the case of an EDA workflow, a timing QOR degradation in one phase can be traced back to an increase in routing congestion in an earlier phase, which can be narrowed down to a specific category of routing violations caused by an increase in layer boost performed in another earlier workflow phase in the process. Performing such root cause analysis from a plain text log file is a time-consuming and detailed task that only the most professional users can accomplish. Embodiments of the present disclosure relate to automatically extracting data from a log file and presenting the data in a way that highlights these significant differences in the data from the log file, thereby enabling even new users (e.g., new engineers) to analyze the running results of the workflow process and significantly improving the productivity of professional engineers, who will not need to manually search the text of the log file to investigate those differences between runs.

[0145] In some cases, there may be a large number of rows in a given child table such that not all rows can be simultaneously displayed within the viewport or portion of a user interface on a computer monitor. In such a case, if the header row remains above the first row of its corresponding child table, scrolling the rows down can hide the header row. This can make it difficult for a user to remember what type of data is presented in each column of the child table.

[0146] Accordingly, some aspects of embodiments of the present disclosure relate to a dynamic method for maintaining the display of the header row of a child table. In some embodiments, a processing device detects the current scroll position in a report and the visibility state of a child table of the report, and ensures that those relevant headers are also visible by maintaining the display of the relevant headers of all visible child tables.

[0147] Figure 4J Illustrates a child table header of a child table displayed in an example of a portion of a user interface before scrolling 487A and after scrolling 487B according to an embodiment of the present disclosure. As Figure 4J shown, before scrolling the table at 487A, the header 488A can be seen in the table, directly above the two first rows corresponding to NPO_START and PRE_C_4. After scrolling such that subsequent rows of the child tables PRE_C_9, PRE_C_10, and PRE_C_11 are displayed, the user interface maintains the display of the header 488B in the user interface (e.g., by pinning the header 488B to the top bar to appear at the top of the displayed child table).

[0148] Figure 4KShows another example of maintaining the display of the header row of a sub-table that is split by another sub-table according to an embodiment of the present disclosure. Initially, the user interface 490 has a header 491 corresponding to the displayed QOR heartbeat metrics (e.g., WNS column and TNS column) of the QOR heartbeat sub-table 492. In the user interface 495, the user has enabled the visibility of the legalizer summary table 496 (e.g., as discussed above, by toggling the visibility switch or expanding the step that includes the metrics according to the legalizer summary table), which splits the QOR heartbeat table 497 (the legalizer summary table 496 appears between the rows of PRE_C_16_[5] and PRE_C_18 of the QOR heartbeat sub-table). The header for the QOR heartbeat 491 sub-table still appears at the top, and a new header 498 is inserted into the user interface for the legalizer summary table 496. Another header 499 for the QOR heartbeat sub-table is dynamically inserted after the legalizer summary sub-table 496 because a portion of the QOR heartbeat sub-table appears after the legalizer summary sub-table 496 (starting with the row of PRE_C_18). Additionally, when a nested or sub-sub table is hidden by some mechanism (e.g., toggling the visibility of the sub-table or rows therein), the second header of the table that remains visible is now hidden because this second header has become redundant since it is no longer split by the second sub-table.

[0149] In some cases, a sub-table may have many different metrics and thus may have more columns than will fit in the user interface, even when the report is displayed on a high-resolution widescreen monitor. Additionally, the user may prefer to reduce the size of the window displaying the report in order to make other programs visible on the screen. Further, due to the amount of horizontal scrolling required to view different metrics, a large number of columns may make it difficult for the user to understand the relationships between different metrics.

[0150] Accordingly, some aspects of embodiments of the present disclosure relate to classifying the metrics of a sub-table into primary metrics and secondary metrics. Figure 4LDepicts the display of the reported sub - table 4100 showing the first primary measure sub - table portion 4102 and the second secondary measure sub - table portion 4104 of the sub - table according to an embodiment of the present disclosure. In some embodiments, the primary measure corresponds to a frequently used measure, and the secondary measure is a rarely used measure. Which measures are primary and which are secondary can be defined by, for example, the user, the developer of the software tool that generates the log data, or the administrator of the report - generating software according to embodiments of the present disclosure. In some embodiments, the division of measures into primary and secondary measures is performed based on criteria other than usage, such as category (e.g., in the context of EDA, the category can be timing, power, area, etc.). The sub - table is divided into a first primary measure sub - table portion and a second secondary measure sub - table portion, where the second sub - table can be hidden by default but can be toggled with the first sub - table. Additionally, the measures can be divided into more than two measure groups (e.g., primary, secondary, tertiary, and quaternary measures) presented in more than two sub - table portions (e.g., first, second, third, and fourth sub - table portions). This reduces the amount of horizontal scrolling performed by the user and allows the user to focus on the measures most likely to be relevant to the analysis.

[0151] In Figure 4L the example shown, the first sub - table portion 4102 of the QOR heartbeat sub - table shows the primary measures, which include the worst negative slack (WNS) 4120, total negative slack (TNS) 4121, register - to - register TNS (R2RTNS) 4122, number of violation endpoints (NVE) 4123, hold total negative slack (HTNS) 4124, leakage power 4125, total power (TotPower) 4126, area 4127, instance count (InstCnt) 4128, and transition cost (TranCost) 4129. User interface components such as a trigger or a switch or a button, a keyboard shortcut (e.g., a key combination or a function key), or a mouse click or a gesture (e.g., a middle - mouse click) cause the sub - table 4100 to switch from displaying the first sub - table portion 4102 with primary measures to displaying the second sub - table portion 4104 with secondary measures. In Figure 4LIn the example shown, the secondary metrics displayed in the second sub-table portion 4104 include the Hold Worst Negative Slack (HWNS) 4140, the number of Hold Violation Endpoints (HNVE) 4141, the Buffer Count (BufCnt) 4142, the Inverter Count (InvCnt) 4143, the Transition Design Rule Check Violation (TranDRC) 4144, the Capacitance Design Rule Check Violation (CapDRC) 4145, the Peak Memory (PeakMem) 4146, the total power consumption of the combinational circuit (TPwrComb) 4147, the total power consumption of the sequential circuit (TPwrSeq) 4148, and the total power consumption of the clock circuit (TPwrClk) 4149. In some embodiments of the present disclosure, switching between different sub-table portions that display the complete sub-table (e.g., switching between the first sub-table portion 4102 and the second sub-table portion 4104) maintains an approximate width of the columns. For example, the column of the first sub-table portion 4102 that displays the area metric 4127 and the column that displays the total power consumption of the combinational circuit (TPwrComb) 4147 metric of the second sub-table portion 4104 have an approximate same width and are in the same position.

[0152] Figure 5 is a flowchart of a method 500 for displaying an interactive report of metrics from a log file of a computational workflow process, the metrics being hierarchically organized based on the stages of the computational workflow process. Although Figure 4B 、 4I 、4J and 4K show examples of the display of sub-tables comparing multiple runs (e.g., two runs), embodiments of the present disclosure are not limited thereto, and a hierarchical display of data from different sections of a single log file is also applicable.

[0153] At 510, a processing device parses a log file generated by a run of a computational workflow process to generate structured data including different sections for different stages of the computational workflow process. As mentioned above, the structured data may be in the form of one or more structured data files in a data format such as CSV, TSV, JSON, XML, BSON, MessagePack, or the like. At 530, the processing device extracts metrics from the structured data file, where at least two of the sections have different types of data (e.g., different sets of metrics and / or text data), such that the sub-tables for these sections have different headers for their respective columns. At 550, the processing device generates an interactive user interface report to hierarchically display the metrics section by section in the order of the computational workflow process (e.g., display the sections in chronological order). The processing device generates a user interface such that sections of the report containing sub-tables with different types of metrics are displayed together with different headers corresponding to the metrics therein.

[0154] Figure 6 An example set of processes 600 are described for use during the design, verification, and fabrication of an article such as an integrated circuit to transform and verify design data and instructions representative of the integrated circuit. Each of these processes may be structured and enabled as multiple modules or operations. The term 'EDA' represents the term 'electronic design automation'. These processes begin with creating a product idea 610 with information supplied by a designer, which is transformed to create an article using a set of EDA processes 612. When the design is complete, the design is taped out 634, which is when the layout (e.g., geometric pattern) of the integrated circuit is sent to a manufacturing facility to fabricate a mask set, which is then used to fabricate the integrated circuit. After tape out, a semiconductor die is fabricated 636, and packaging and assembly processes 638 are performed to produce a finished integrated circuit 640.

[0155] The specification of a circuit or electronic structure can range from a low-level transistor material layout to a high-level description language. High-level representations can be used to design circuits and systems using a hardware description language ('HDL') such as VHDL, Verilog, SystemVerilog, SystemC, MyHDL, or OpenVera. The HDL description can be transformed into a logic-level register transfer level ('RTL') description, a gate-level description, a layout-level description, or a mask-level description. Each lower representation level as described in more detail adds more useful details to the design description, e.g., more details of the modules included in the description. The lower representation levels as described in more detail can be computer-generated, exported from a design library, or created by another design automation process. An example of a specification language for a lower level of the representation language for specifying a more detailed description is SPICE, which is used for the detailed description of circuits with many analog components. Enabling the description of each representation level to be used by the corresponding system (e.g., a formal verification system) of that layer. The design process can use Figure 6 the sequence depicted in Figure 6 The processes described by

[0156] During system design 614, the functionality of the integrated circuit to be fabricated is specified. The design can be optimized for desired characteristics such as power consumption, performance, area (physical and / or lines of code), and cost reduction, etc. The design can be partitioned into different types of modules or components at this stage.

[0157] During logic design and functional verification 616, modules or components in a circuit are specified using one or more description languages, and the functional accuracy of the specification is checked. For example, components of a circuit can be examined to produce outputs that match the specification requirements of the designed circuit or system. Functional verification can use simulators and other programs such as test bench generators, static HDL checkers, and formal verifiers. In some embodiments, a special system of components referred to as an 'emulator' or 'prototyping system' is used to accelerate functional verification.

[0158] During synthesis and test design 618, HDL code is converted into a netlist. In some embodiments, the netlist can be a graphical structure where the edges of the graphical structure represent components of the circuit and where the nodes of the graphical structure represent how the components are interconnected. Both the HDL code and the netlist are hierarchical artifacts that can be used by EDA products to verify that an integrated circuit will perform according to the specified design when manufactured. The netlist can be optimized for a target semiconductor manufacturing technology. Additionally, the fabricated integrated circuit can be tested to verify that the integrated circuit meets the specification requirements.

[0159] During netlist verification 620, the netlist is checked to see if it meets timing constraints and corresponds to the HDL code. During design planning 622, an overall floorplan of the integrated circuit is constructed and analyzed for timing and top-level routing.

[0160] During layout or physical implementation 624, physical layout (e.g., the placement of circuit components such as transistors or capacitors) and routing (connecting circuit components via multiple conductors) are performed, and cells can be selected from a library to implement a specific logic function. As used herein, the term 'cell' can specify a set of transistors, other components, and interconnects that provide a Boolean logic function (e.g., AND, OR, NOT, XOR) or a storage function (e.g., flip-flop or latch). As used herein, a circuit 'block' can refer to two or more cells. Both cells and circuit blocks can be referred to as modules or components and can be enabled as both physical structures and simulations. Parameters such as size are specified for selected cells (based on'standard cells') and made accessible in a database for use by EDA products.

[0161] During analysis and extraction 626, the circuit function is verified at the layout level, which allows for refinement of the layout design. During physical verification 628, the layout design is checked to ensure that manufacturing constraints such as DRC constraints, electrical constraints, and lithography constraints are correct and that the circuit function matches the HDL design specification. During resolution enhancement 630, the geometry of the layout is transformed to improve the way the circuit design is manufactured.

[0162] During tape-out, data is created for the production of a photolithography mask (after applying lithography enhancements if appropriate). During mask data preparation 632, the 'tape-out' data is used to generate a photolithography mask for the production of a finished integrated circuit.

[0163] A storage subsystem of a computer system (such as Figure 7 computer system 700) can be used to store programs and data structures used by some or all of the EDA products described herein, and products for developing cells for a library and for physical and logical designs using the library.

[0164] Figure 7 An example machine of computer system 700 in which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein. In an alternative embodiment, the machine can be connected (e.g., networked) to other machines in a LAN, intranet, extranet, and / or the Internet. The machine can operate in the capacity of a server or client in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or client in a cloud computing infrastructure or environment.

[0165] The machine can be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, network device, server, network router, switch, or bridge or any machine capable of executing a set of instructions (sequentially or otherwise) that specify actions to be taken by the machine. Further, although a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0166] Example computer system 700 includes a processing device 702, a main memory 704 (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM)), a static memory 706 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 718, which communicate with each other via a bus 730.

[0167] The processing device 702 represents one or more processors, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or multiple processors implementing a combination of instruction sets. The processing device 702 can also be one or more dedicated processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 702 can be configured to execute the instructions 726 for performing the operations and steps described herein.

[0168] The computer system 700 can further include a network interface device 708 for communicating over a network 720. The computer system 700 can also include a video display unit 710 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), a graphics processing unit 722, a signal generation device 716 (e.g., a speaker), a graphics processing unit 722, a video processing unit 728, and an audio processing unit 732.

[0169] The data storage device 718 can include a machine-readable storage medium 724 (also referred to as a non-transitory computer-readable medium) having stored thereon one or more sets of instructions 726 or software embodying any one or more of the methods or functions described herein. During execution of the instructions 726 by the computer system 700, the instructions 726 can also reside completely or at least partially within the main memory 704 and / or within the processing device 702, which also constitutes a machine-readable storage medium.

[0170] In some embodiments, the instructions 726 include instructions implementing functionality corresponding to the present disclosure. Although the machine-readable storage medium 724 is shown as a single medium in the exemplary embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium that is capable of storing or encoding a set of instructions for execution by a machine and that causes the machine and the processing device 702 to perform any one or more of the methods of the present disclosure. Thus, the term "machine-readable storage medium" should be considered to include, but not be limited to, solid state memories, optical media, and magnetic media.

[0171] Some of the foregoing have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the data processing arts most effectively convey the substance of their work to others skilled in the art. An algorithm is a sequence of operations leading to a desired result. The operations are those requiring physical manipulation of physical quantities. These quantities may take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. Such signals may be referred to as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0172] However, it should be borne in mind that all such and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless explicitly stated otherwise as apparent from the present disclosure, it should be understood that throughout the description, certain terms refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the computer system memory or registers or other such information storage devices.

[0173] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the intended purpose, or it may comprise a computer selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to the computer system bus.

[0174] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various other systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the method. Furthermore, the present disclosure is not described with reference to any particular programming language. It will be understood that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein.

[0175] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon, the instructions being usable to program a computer system (or other electronic device) to perform a process according to the present disclosure. The machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, the machine-readable (e.g., computer-readable) medium includes machine (e.g., computer) readable storage media such as read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices, etc.

[0176] In the foregoing disclosure, embodiments of the present disclosure have been described with reference to specific example embodiments of the present disclosure. Obviously, various modifications can be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the following claims. In cases where the present disclosure refers to some elements in the singular, more than one element may be depicted in the figures and the same elements are labeled with the same numerals. Accordingly, the present disclosure and the figures should be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method comprising: receiving first structured data extracted from a first log file generated by a first run of an electronic design automation process and second structured data extracted from a second log file generated by a second run of the electronic design automation process; determining, by a processing device based on the first structured data and the second structured data, that a first section of the first log file and a second section of the second log file correspond to outputs of a same stage of the electronic design automation process; extracting a first metric from the first section of the first log file and extracting a second metric from the second section of the second log file; and A user interface is generated to display the first metric from the first section of the first log file adjacent to the second metric from the second section of the second log file.

2. The method of claim 1 , wherein the first segment has a position in the first log file that is different from a position of the second segment in the second log file, and The first structured data is generated by parsing the first log file, including: identifying a first workflow step start marker at a first location in the first log file; and determining the position of the first segment in the first log file based on the first position of the first workflow step start marker, and The second structured data is generated by parsing the second log file, including: identifying a second workflow step start marker at a second location in the second log file; and The position of the second segment in the second log file is determined based on the second position of the second workflow step start marker. 3 . The method of claim 2 , wherein said parsing said first log file is performed concurrently with said first run of said electronic design automation process.

4. The method of claim 1, wherein the user interface comprises a control configured to select a display mode from a plurality of display modes for displaying the second metric, the plurality of display modes comprising: a raw value of one of the second metrics; a percentage change between the raw value of the second metric and a corresponding metric of a baseline run; and The absolute change between the raw value of the second metric and the corresponding metric of the baseline run.

5. The method of claim 4, wherein the user interface is implemented using a hypertext document and a style sheet, wherein the node of the hypertext document corresponding to the second metric includes a plurality of child nodes storing the original value, the percentage change, and the absolute change, and Wherein interaction with the control of the user interface causes the user interface to modify the style sheet to make one of the child nodes visible and to hide the other child nodes. 6 . The method of claim 4 , wherein the user interface includes a control configured to select one of a plurality of runs of the electronic design automation process as the baseline run.

7. The method according to claim 1, further comprising: receiving a plurality of first messages extracted from the first log file and a plurality of second messages extracted from the second log file; grouping the plurality of first messages and the plurality of second messages by message type to generate a first plurality of message groups from the first log file and a second plurality of message groups from the second log file; generating a summary of the first plurality of message groups and a summary of the second plurality of message groups, the summary of a message group comprising counts of messages in the groups and representative example messages from the message groups; and Differences between a first summary of a first group of messages from the first log file and a second summary of a second group of messages from the second log file are highlighted in the user interface.

8. The method of claim 1, wherein the first metric comprises a first plurality of metric sub-tables and the second metric comprises a second plurality of metric sub-tables, the first plurality of metric sub-tables and the second plurality of metric sub-tables corresponding to sub-phases of a same phase of the electronic design automation process, and Wherein the user interface displays metrics from the first plurality of sub-tables adjacent to corresponding metrics from the second plurality of sub-tables section by section in a hierarchy in an order of the electronic design automation process.

9. A system comprising: a memory storing instructions; and a processor coupled to the memory and executing the instructions, which when executed cause the processor to: receiving structured data extracted from a log file generated by execution of an electronic design automation process at an iteration of an integrated circuit design, the structured data comprising a plurality of sections corresponding to stages of the electronic design automation process; extracting a plurality of metric sub-tables from the plurality of sections; and An interactive user interface report is generated to hierarchically display the plurality of metric sub-tables section by section in the sequence of the electronic design automation process.

10. The system of claim 9, wherein a first sub-table of the plurality of sub-tables includes metrics of a first plurality of types of metric data, and a second sub-table of the plurality of sub-tables includes metrics of a second plurality of types of metric data different from the first plurality of types of metric data, and The interactive user interface report includes: a first portion comprising a first header identifying the first plurality of types of metric data and metrics from the first subtable; and A second portion includes a second header identifying the second plurality of types of metric data and metrics from the second sub-table.

11. The system of claim 10, wherein the user interface is configured to: maintaining display of the first header while any portion of the metric from the first sub-table is visible in the user interface; and Display of the second header is maintained while any portion of the metrics from the second sub-table is visible in the user interface.

12. The system of claim 10, wherein the first sub-table includes metrics written to the log file during a first phase of the electronic design automation process, wherein the second sub-table includes metrics written to the log file after a first plurality of metrics of the first sub-table written to the log file during the first phase and before a second plurality of metrics of the first sub-table written to the log file during the first phase, and The second sub-table segments the first sub-table in the interactive user interface report.

13. The system of claim 12, wherein the interactive user interface report further comprises: a third portion comprising the first header identifying the first plurality of types of metric data and additional metrics from the first subtable, and The second portion including the metric from the second sub-table is displayed in the user interface between: comprising said first portion of said metrics from said first subtable; and The third portion includes the additional metrics from the first sub-table.

14. The system of claim 13, wherein the user interface is configured to: maintaining the display of the first header in the first portion when any of the metrics from the first sub-table is visible in the user interface; maintaining the display of the first header in the third portion when any of the additional metrics from the first sub-table and the second sub-table are visible in the user interface; and The display of the second header in the second portion is maintained while any of the metrics from the second sub-table is visible in the user interface.

15. The system of claim 9, wherein the memory further stores instructions that, when executed, cause the processor to: receiving second structured data extracted from a second log file generated by a second run of the electronic design automation process at a second iteration of the integrated circuit design, the second structured data comprising a second plurality of sections corresponding to the phase of the electronic design automation process; extracting a second plurality of metric sub-tables from the second plurality of sections of the second log file; and determining a correspondence between the plurality of sections of the structured data and the second plurality of sections of the second structured data, and Wherein the interactive user interface report further displays metrics from the second plurality of metric sub-tables adjacent to corresponding metrics from the plurality of metric sub-tables.

16. A non-transitory computer readable medium comprising stored instructions that, when executed by a processor, cause the processor to: receiving first structured data extracted from a first log file generated by a first run of an electronic design automation process at a first iteration of an integrated circuit design, the first structured data comprising a first plurality of sections corresponding to stages of the electronic design automation process; receiving second structured data extracted from a second log file generated by a second run of the electronic design automation process at a second iteration of the integrated circuit design, the second structured data comprising a second plurality of sections corresponding to the phase of the electronic design automation process; An interactive user interface report is generated to hierarchically display, in the order of the electronic design automation process: a first metric from the first plurality of segments of the first structured data; and A second metric from the second plurality of segments of the second structured data is adjacent to the first metric from a corresponding segment of the first plurality of segments of the first structured data.

17. The non-transitory computer-readable medium of claim 16, wherein the interactive user interface report includes a user interface control to switch between an expanded view and a collapsed view of a portion of the interactive user interface report displaying metrics from a first section of the first log file and a corresponding second section of the second log file, the first section and the corresponding second section corresponding to a same stage of the electronic design automation process, wherein the expanded view displays a first original value of the first segment from the first log file and a second original value of the corresponding second segment from the second log file, and The collapsed view displays a plurality of first summary metrics calculated from the first raw values ​​and second summary metrics calculated from the second raw values.

18. The non-transitory computer-readable medium of claim 16, wherein the interactive user interface report highlights one of the second metrics that differs in value from a corresponding one of the first metrics by at least a threshold value.

19. The non-transitory computer readable medium of claim 16, wherein The interactive user interface report highlights a second non-numeric metric in the second metrics that is different in value from a corresponding first non-numeric metric in the first metrics.

20. The non-transitory computer-readable medium of claim 16, wherein the first metric comprises a first plurality of metric sub-tables and the second metric comprises a second plurality of metric sub-tables, the first plurality of metric sub-tables and the second plurality of metric sub-tables corresponding to sub-phases of a same phase of the electronic design automation process, and wherein the interactive user interface report displays metrics from the first plurality of sub-tables adjacent to corresponding metrics from the second plurality of sub-tables segment by segment in a hierarchy in the order of the electronic design automation process.

Images