Synchronized Hardware Event Collection

Abstract

To provide a method and hardware event collection system for analyzing the execution of program code.SOLUTION: In a method for monitoring the execution of program code by processor components, a computing system detects that a trigger condition is satisfied by identifying an operand in a portion of the program code or determining that a current time of a clock of the system indicates a predefined time value. The operand and the predefined time value are used to initiate trace events. When the trigger condition is satisfied, the system initiates trace events that generate trace data identifying respective hardware events occurring across the entire system and uses a correlated set of trace data to analyze the performance of the executing program code. The correlated trace data indicates a time-ordered sequence of the respective hardware events.SELECTED DRAWING: Figure 7

Description

[Background technology]

[0001] background This specification relates to analyzing the execution of program code.

[0002] Effective performance analysis of distributed software running within distributed hardware components can be a complex task. Distributed hardware components are comprised of two or more Central Processing Units (CPUs) (or graphics processors) that cooperate and interact to execute a larger software program or portion of program code. In some cases, the GPU may be a processor core of a graphics processing unit (GPU).

[0003] From a hardware perspective (e.g., in a CPU or GPU), there are broadly two types of information or functionality available for performance analysis: 1) hardware performance counters, and 2) hardware event traces. Summary of the Invention [Means for solving the problem]

[0004] overview In general, one innovative aspect of the subject matter described herein may be embodied in a computer-implemented method executed by one or more processors, the method including monitoring execution of a program code by a first processor component, the first processor component configured to execute at least a first portion of the program code, the method further including monitoring execution of the program code by a second processor component, the second processor component configured to execute at least a second portion of the program code.

[0005] The method further includes the computing system detecting that the trigger condition is satisfied based on at least one of: i) identifying an occurrence of an operand in at least a first portion of the program code or a second portion of the program code, or ii) determining that a current time of at least one clock of the computing system indicates a predefined time value used to initiate one or more trace events, wherein the operand includes a first parameter value used to initiate one or more trace events.

[0006] The method further includes, in response to detecting that the trigger condition is satisfied, the computing system initiating at least a first trace event that generates trace data, the trace data identifying respective hardware events occurring across the distributed processor unit including at least a first processor component and a second processor component. For each respective hardware event, the trace data includes at least a hardware event timestamp. The method further includes the computing system using the trace data to generate a correlated set of trace data indicative at least of a chronological order of the respective hardware events that are generated when the trigger condition is satisfied.

[0007] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, the first trace event generates trace data that identifies hardware events occurring across the distributed processor units. Synchronized trace events, where the trace data also identifies a unique trace identifier for each hardware event, and the hardware events include multiple synchronized hardware events, and where two hardware events are synchronized if they share a global hardware event timestamp.

[0008] In some implementations, detecting that the trigger condition is satisfied includes the computing system detecting one of: i) a first parameter value of the operand exceeding a first limit value of a register; or ii) a predefined time value indicated by the current time exceeding a first limit value of a register; and in response to detecting that the trigger condition is satisfied, the computing system initiating a second trace event that generates trace data, the trace data identifying at least one attribute shared between respective hardware events occurring across the distributed processor units.

[0009] In some implementations, the method further includes the computing system detecting one of: a second parameter value of the operand exceeding a second limit value of the register; or a second predefined time value indicated by the current time exceeding a second limit value of the register; and in response to the detecting step, the computing system stopping the second trace event when the second parameter value of the operand exceeds the second limit value or when the second predefined time value exceeds the second limit value.

[0010] In some implementations, the operands further include at least one of a parameter indicating a particular sequence step of the program code or a global control parameter indicating a particular performance state of the distributed processing unit, and the predefined time value includes at least one of a particular time value indicated by a global time clock of the distributed processing unit or a particular time value of a predefined time window associated with the global time clock.

[0011] In some implementations, the operand has a first binary data structure, a first parameter value of the first operand corresponds to a trace mark, the predefined time value has a second binary data structure, and the current time is indicated by a global time clock, the global time clock being used by the distributed processor units to generate one or more hardware event timestamps.

[0012] In some implementations, the method further includes a step in which a compiler of the computing system inserts an operand of the trigger condition into at least a first portion of the program code executed by the first processor component, and a step in which a compiler of the computing system inserts at least one predefined time value of the trigger condition into at least a second portion of the program code executed by the second processor component.

[0013] In some implementations, the step of initiating at least one of the first trace event or the second trace event includes the computing system generating a first control signal that is received by a first count register of the first processor core, the first control signal causing data associated with the first hardware event to be stored in the first count register, and further includes the computing system generating a second control signal that is received by a second count register of the second processor core, the second control signal causing data associated with the second hardware event to be stored in the second count register.

[0014] In some implementations, the data associated with one of the first hardware event or the second hardware event includes at least one of a number of bytes written to a particular memory buffer of a particular processor core of the distributed processor unit, or a number of instructions executed by a particular processor core of the distributed processor unit.

[0015] In some implementations, the method further includes the computing system identifying an occurrence of a second operand in at least a portion of the program code executed by the first or second processor component, the second operand including a second parameter value, and further including the computing system determining that a filter condition is met based on one of the second parameter value of the second operand being above a particular limit value of a register or being below a particular limit value of a register, and in response to determining that the filter condition is met, the computing system filtering one or more trace events, wherein filtering the one or more trace events includes excluding storage of trace data associated with the one or more hardware events.

[0016] Other innovative aspects of the subject matter described herein may be embodied in a hardware event collection system including one or more processors including one or more processor cores and one or more machine-readable storage units for storing instructions, the instructions being executable by the one or more processors to perform operations, the operations including monitoring execution of program code by a first processor component, the first processor component configured to execute at least a first portion of the program code, and the operations further including monitoring execution of the program code by a second processor component, the second processor component configured to execute at least a second portion of the program code.

[0017] The system further includes a step of the computing system detecting that the trigger condition is satisfied based on at least one of: i) identifying an occurrence of an operand in at least a first portion of the program code or a second portion of the program code, or ii) determining that a current time of at least one clock of the computing system indicates a predefined time value used to initiate one or more trace events, wherein the operand includes a first parameter value used to initiate one or more trace events.

[0018] The system further includes, in response to detecting that the trigger condition is satisfied, the computing system initiating at least a first trace event that generates trace data, the trace data identifying respective hardware events occurring across the distributed processor unit including at least a first processor component and a second processor component. For each respective hardware event, the trace data includes at least a hardware event timestamp. The system further includes the computing system using the trace data to generate a correlated set of trace data indicative at least of a chronological order of the respective hardware events that are generated when the trigger condition is satisfied.

[0019] In general, one innovative aspect of the subject matter described herein may be embodied in a computer-implemented method executed by one or more processors. The method includes monitoring execution of program code by a processor component, the method comprising: The processor component is configured to execute at least a first portion of the program code.

[0020] The method further includes the computing system detecting that the trigger condition is satisfied based on at least one of: i) identifying an occurrence of an operand in at least a first portion of the program code; or ii) determining that a current time of at least one clock of the computing system indicates a predefined time value used to initiate one or more trace events, the operand including a first parameter value used to initiate one or more trace events.

[0021] The method further includes, in response to detecting that the trigger condition is satisfied, the computing system generating a control signal that is received by a count register of the processor component, the control signal causing count data associated with the hardware event to be stored in the count register, and the method further includes the computing system generating a data structure indicative of one or more performance attributes associated with the executed program code, the data structure being generated based on one or more count parameters of the stored count data.

[0022] These and other implementations may each optionally include one or more of the following features: For example, in some implementations, the count register is one of a plurality of performance counters configured to store count data regarding the performance of one or more processor components of the computing system, where at least one performance counter includes one of an activity counter, a stall counter, a statistical counter, or a sampling counter.

[0023] In some implementations, the one or more count parameters indicate one of: i) a number of instructions received by a particular processor component; ii) a number of instructions processed by a particular processor component; iii) a number of instructions executed by a particular processor component; or iv) a number of memory reads or memory writes performed by a particular processor component.

[0024] In some implementations, the performance attributes associated with the executed program code include one of: i) a stall frequency of a particular processor component executing the program code; ii) an indication that utilization of the particular processor component is below a utilization threshold; or iii) an indication that a data store queue used by the particular processor component is below a queue occupancy threshold.

[0025] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods encoded on a computer storage device. One or more computer systems may be so configured by software, firmware, hardware, or combinations thereof installed on the system that, when operated, causes the system to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions.

[0026] The subject matter described herein may be implemented in certain embodiments to achieve one or more of the following advantages: The described hardware tracing system increases computational efficiency by using dynamic triggers implemented through hardware knobs / functions. Triggers typically require system resources to monitor and perform event capture. It allows for synchronized capture of events based in part on global time counters, embedded logical operands, and hardware registers, rather than solely via a required encoding sequence.

[0027] Similarly, system memory usage is optimized when hardware triggers are coordinated to capture simultaneous short duration events and synchronized global capture instead of uncorrelated and inefficient event capture. Control over synchronized event capture of shorter duration alleviates information overload, thereby allowing some memory resources to remain unused and available for other system processes.

[0028] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a block diagram of an exemplary computing system for distributed hardware tracing. [Diagram 2] FIG. 1 is a block diagram of a trace chain and respective nodes of an exemplary computing system for distributed hardware tracing. [Diagram 3] FIG. 2 is a block diagram of an example trace multiplexing design architecture and example data structures. [Figure 4] FIG. 2 is a block diagram illustrating trace activity for direct memory access trace events performed by an exemplary computing system for distributed hardware tracing. [Diagram 5] FIG. 1 illustrates an example data structure for a global time counter (GTC) and a table illustrating a usage scenario of the GTC by an example computing system for distributed hardware tracing. [Figure 6] FIG. 1 is a block diagram of example time counters and hardware functions associated with components of an example computing system for distributed hardware tracing. [Figure 7] FIG. 1 is a process flow diagram of an example process for distributed hardware tracing.

[0030] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Detailed Description The subject matter described herein generally relates to distributed hardware tracing. In particular, a computing system monitors the execution of program code executed by a first processor core and the execution of program code executed by a second processor core. The computing system stores a timeline of hardware events in a memory buffer. The stored events occur across a distributed processing unit including at least the first and second processor cores.

[0032] The timeline includes, for each hardware event, metadata characterizing the hardware event and an event timestamp. The system generates a data structure from the timeline that includes the hardware events. The system stores the data structure in a memory bank of the host device and uses the data structure to evaluate performance of program code executed by the first or second processor core. Event Tracing In this context, this specification describes an approach for hardware event tracing in distributed computing systems, which is shown in FIGS. 1-4.

[0033] This specification further describes approaches for synchronized hardware event collection based on one or more trigger mechanisms, as shown in Figures 5-7. As described in more detail below, aspects of the computing system described herein relate at least to coordinated / synchronized collection of tracing data and event counting. In particular, at least one aspect includes a system and method for synchronized collection of hardware performance counter data and trace event data in standalone and distributed systems. Synchronized event collection improves analysis of both performance and debug data of distributed program code. The improved analysis is achieved in part through correlation of events occurring in response to execution of connected software components / modules that are analyzed by system 100.

[0034] FIG. 1 illustrates a block diagram of an exemplary computing system 100 for distributed hardware tracing. As used herein, distributed hardware system tracing corresponds to the storage of events occurring within components and subcomponents of an exemplary processor microchip. Additionally, as used herein, a distributed hardware system (or tracing system) corresponds to a collection of processor microchips that cooperate to execute respective portions of software / program code configured for distributed execution. In some implementations, different processor chips of the system 100 can form respective nodes of the distributed hardware system. In alternative implementations, a single processor chip may include one or more processor cores and hardware functions that can each form a respective node of the processor chip.

[0035] For example, in the context of a central processing unit (CPU), a processor chip may include at least two nodes, where each node may be a respective core of the CPU. Alternatively, in the context of a graphical processing unit (GPU), a processor chip may include at least two nodes, where each node may be a respective streaming multiprocessor of the GPU. Computing system 100 may include multiple processor components. In some implementations, a processor component may be at least one of a processor chip, a processor core, a memory access engine, or at least one hardware component of the overall computing system 100.

[0036] In some examples, a processor component, such as a processor core, may be a fixed function component configured to perform at least one particular operation based on at least one issued instruction of an executing program code. In other examples, a processor component, such as a memory access engine (MAE), may be configured to execute program code at a lower level of detail or granularity than program code executed by other processor components of system 100.

[0037] For example, program code executed by a processor core may cause an MAE descriptor to be generated and sent to the MAE. After receiving the descriptor, the MAE may perform a data transfer operation based on the MAE descriptor. In some implementations, the data transfer performed by the MAE may include, for example, moving data between certain components of system 100 via certain data paths or interface components of the system, or issuing a data request to an example configuration bus of system 100.

[0038] In some implementations, each tensor node of an exemplary processor chip of system 100 may have at least two "front ends," which may be hardware blocks / functions that process program instructions. As described in more detail below, a first front end may correspond to a first processor core 104, while a second front end may correspond to a second processor core 106. Thus, the first and second processor cores may also be referred to herein as a first front end 104 and a second front end 106.

[0039] As used herein, a trace chain may be a particular physical data communication bus on which trace entries may be placed for transmission to an exemplary chip manager in system 100. A received trace entry may be a data word / structure that includes multiple bytes and multiple binary values ​​or numbers. Thus, the "word" descriptor refers to a fixed-sized piece of binary data that may be treated as a unit by the hardware devices of the exemplary processor core.

[0040] In some implementations, the processor chips of the distributed hardware tracing system are multi-core (i.e., having multiple cores) processors that each execute a portion of the program code on a respective core of the chip. In some implementations, the portion of the program code may correspond to vectorized calculations for an exemplary multi-layer neural network inference workload, while in alternative implementations, the portion of the program code may generally correspond to software modules associated with traditional programming languages.

[0041] The computing system 100 generally comprises a node manager 102, a first processor core (FPC) 104, and a second processor core (FPC). The system 100 includes a FPC 104, a node fabric (NF) 110, a data router 112, and a host interface block (HIB) 114. In some implementations, the system 100 may include a memory mux 108 configured to perform signal switching, multiplexing, and demultiplexing functions. The system 100 further includes a tensor core 116 within which the FPC 104 is disposed. The tensor core 116 may be an exemplary computing device configured to perform vectorized calculations on multi-dimensional data arrays. The tensor core 116 may be a vector processing unit (VEC). The VPU 118 may include a matrix processing unit (MPU) 118. MXU 120, transpose unit (XU) 122, reduction and substitution unit In some implementations, the computing system 100 may include other processors, such as a load / store unit, an arithmetic logic unit (ALU), and a vector unit. It may include one or more execution units, such as a conventional CPU or a GPU.

[0042] The components of system 100 collectively include a large set of hardware performance counters and supporting hardware that facilitates the completion of trace activity within the components. As described in more detail below, program code executed by each processor core of system 100 may include embedded triggers that are used to simultaneously enable multiple performance counters during code execution. In general, a detected trigger causes trace data to be generated for one or more trace events. The trace data may correspond to incremental parameter counts that may be stored in counters and analyzed to identify performance characteristics of the program code. Data for each trace event may be stored in an exemplary storage medium (e.g., a hardware buffer) and may include a timestamp that is generated in response to detection of a trigger.

[0043] Additionally, trace data may be generated for various events occurring within the hardware components of system 100. Exemplary events may include intra-node and cross-node communication operations, such as direct memory access (DMA) operations and synchronization flag updates (each of which is described in more detail below). In some implementations, system 100 includes a global time counter, commonly referred to as a Global Time Counter (GTC). In other implementations, the system 100 may include other types of global clocks, such as a Lamport clock. It can be seen.

[0044] GTC can be used for precise correlation of program code execution and performance of software / program code executing in a distributed processing environment. In addition, and in part related to GTC, in some implementations, system 100 can include one or more trigger mechanisms that are used by distributed software programs to start and stop data tracing in a highly coordinated manner in the distributed system.

[0045] In some implementations, the host system 126 compiles program code that may include embedded operands that, when detected, trigger the capture and storage of trace data associated with a hardware event. In some implementations, the host system 126 provides the compiled program code to one or more processor chips of the system 100. In an alternative implementation, the program code may be compiled (with embedded triggers) by an exemplary external compiler and loaded into one or more processor chips of the system 100. In some examples, the compiler may set one or more trace bits (described below) associated with certain triggers embedded in some of the software instructions. The compiled program code may be a distributed software program executed by one or more components of the system 100.

[0046] The host system 126 may include a monitoring engine 128 configured to monitor the execution of program code by one or more components of the system 100. In some implementations, the monitoring engine 128 enables the host system 126 to monitor the execution of program code executed by at least the FPC 104 and the SPC 106. For example, during code execution, the host system 126 may monitor the performance of the executing code via the monitoring engine 128 by at least receiving a periodic timeline of hardware events based on the generated trace data. Although a single block is shown for the host system 126, in some implementations, the host system 126 may include multiple hosts (or host subsystems) associated with multiple processor chips or chip cores of the system 100.

[0047] In another implementation, cross-node communication involving at least three processor cores may cause the host system 126 to monitor data traffic at one or more intermediate "hops" as the data traffic traverses a communication path between the FPC 104 and an exemplary third processor core / node. For example, the FPC 104 and the third processor core may be the only cores executing program code at a given time. Thus, data transfer from the FPC 104 to the third processor core may generate trace data for the intermediate hops in the SPC 106 as the data is transferred from the FPC 104 to the third processor core. In other words, during data routing in the system 100, data going from a first processor chip to a third processor chip may need to traverse a second processor chip, and thus, execution of a data routing operation may cause a trace entry to be generated for the routing activity at the second chip. do not have.

[0048] When the compiled program code is executed, the components of the system 100 can interact to generate a timeline of hardware events occurring in the distributed computer system. The hardware events can include intra-node and cross-node communication events. Exemplary nodes of a distributed hardware system and their associated communications are described in more detail below with reference to FIG. 2. In some implementations, a data structure is generated that identifies a set of hardware events for at least one hardware event timeline. The timeline enables reconstruction of events occurring in the distributed system. In some implementations, the event reconstruction can include correct event ordering based on analysis of timestamps generated during the occurrence of a particular event.

[0049] In general, an exemplary distributed hardware tracing system may include the above-mentioned components of system 100 and at least one host controller associated with host system 126. Performance or debugging of data obtained from a distributed tracing system may be useful if the event data is correlated, for example, in a time series or sequence. In some implementations, data correlation may occur when multiple stored hardware events corresponding to connected software modules are stored and then sequenced for structured analysis by host system 126. For implementations including multiple host systems, correlation of data obtained via different hosts may be performed, for example, by a host controller.

[0050] In some implementations, FPC 104 and SP 106 are each separate cores of a single multi-core processor chip, while in other implementations, FPC 104 and SP 106 are respective cores of separate multi-core processor chips. As mentioned above, system 100 may include a distributed processing unit having at least FPC 104 and SPC 106. In some implementations, the distributed processing unit of system 100 may include one or more hardware or software components configured to execute at least a portion of a larger distributed software program or program code.

[0051] Data router 112 is an inter-chip interconnect (ICI) that provides data communication paths between components of system 100. In particular, router 112 may provide communication couplings or connections between the FPC 104 and the SPC 106, and between the respective components associated with the cores 104, 106. The node fabric 110 interacts with data routers 112 to move data packets within the distributed hardware components and subcomponents of the system 100.

[0052] The node manager 102 is a high-level device that manages the lower level node functions in a multi-node processor chip. As described in more detail below, one or more nodes of the processor chip may include a chip manager that is controlled by the node manager 102 to manage and store hardware event data in a local entry log. The memory mux 108 responds to data signals provided to or received from an exemplary external high bandwidth memory (HBM). The present invention relates to a multiplexing device that can perform switching, multiplexing, and demultiplexing operations using a single 10-bit / slot 128 and a single 10-bit / slot 168.

[0053] In some implementations, as the mux 108 switches between the FPC 104 and the SPC 106, an exemplary trace entry (described below) may be generated by the mux 108. The memory mux 108 may impact the performance of a particular processor core 104, 106 that does not have access to the mux 108. Thus, the trace entry data generated by the mux 108 may aid in understanding resulting spikes in the latency of certain system activities associated with each core 104, 106. In some implementations, hardware event data (e.g., trace points, described below) occurring within the mux 108 may be grouped with event data for the node fabric 110 in an exemplary hardware event timeline. Event grouping may occur when a trace activity causes event data for multiple hardware components to be stored in an exemplary hardware buffer (e.g., trace entry log 218, described below).

[0054] In system 100, performance analysis hardware includes FPC 104, SPC 106, mux 108, node fabric 110, data router 112, and HIB 114. Each of these hardware components or units includes hardware performance counters and hardware event tracing mechanisms and functions. In some implementations, VPU 118, MXU 120, XU 122, and RPU 124 do not include their own dedicated performance hardware. Rather, in such implementations, FPC 104 may be configured to provide the necessary counters for VPU 118, MXU 120, XU 122, and RPU 124.

[0055] The VPU 118 may include an internal design architecture that supports localized high bandwidth data processing and arithmetic operations associated with the vector elements of the exemplary matrix-vector processor. The MXU 120 is a matrix multiplication unit configured to perform, for example, up to 128×128 matrix multiplications on vector data sets of multiplicands.

[0056] The XU 122 is a transposition unit configured to perform, for example, up to 128×128 matrix transposition operations on vector data associated with the matrix multiplication operations. The RPU 124 may include a sigma unit and a permutation unit. The sigma unit performs sequential reductions on the vector data associated with the matrix multiplication operations. The reductions may include sums and various types of comparison operations. The permutation unit may completely permute or duplicate all elements of the vector data associated with the matrix multiplication operations.

[0057] In some implementations, the program code executed by the components of system 100 may represent machine learning, neural network inference calculations, and / or one or more direct memory access functions. The components of system 100 may be configured to execute one or more software programs that include instructions that cause a processing unit or device of the system to perform one or more functions. The term "component" is intended to include any data processing device or storage device such as a control status register, or any other device capable of processing and storing data.

[0058] System 100 generally includes one or more processors (e.g., a microprocessor or central processing unit (CPU)), graphics processing units (GPUs), application specific integrated circuits (ASICs), or a combination of different processors. In alternative embodiments, system 100 may include other computing resources / devices (e.g., cloud-based servers) that each provide additional processing options for performing calculations related to the hardware tracing functionality described herein.

[0059] The processing unit or device may further include one or more memory units or memory banks (e.g., registers / counters). In some implementations, the processing unit , execute programmed instructions stored in memory to the devices of system 100 to perform one or more functions described herein. The memory units / banks may include one or more non-transitory machine-readable storage media. The non-transitory machine-readable storage media may include solid-state memory, magnetic and optical disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information.

[0060] 2 illustrates a block diagram of trace chains and respective exemplary nodes 200, 201 used in distributed hardware tracing performed by system 100. In some implementations, nodes 200, 201 of system 100 may be different nodes within a single multicore processor. In other implementations, node 200 may be a first node in a first multicore processor chip and node 201 may be a second node in a second multicore processor chip.

[0061] Although two nodes are illustrated in the implementation of FIG. 2, in alternative implementations, system 100 may include multiple nodes. For implementations involving multiple nodes, cross-node data transfers may generate trace data at intermediate hops along an example data path that traverses multiple nodes. For example, intermediate hops may correspond to data transfers that pass through separate nodes in a particular data transfer path. In some examples, trace data associated with ICI trace / hardware events may be generated for one or more intermediate hops that occur during a cross-node data transfer that passes through one or more nodes.

[0062] In some implementations, node 0 and node 1 are tensor nodes used for vectorized calculations associated with a portion of the program code for an inference workload. As used in this specification, a tensor is a multi-dimensional geometric object, and example multi-dimensional geometric objects include matrices and data arrays.

[0063] 2, node 200 includes a trace chain 203 that interacts with at least a subset of the components of system 100. Similarly, node 201 includes a trace chain 205 that interacts with at least a subset of the components of system 100. In some implementations, nodes 200, 201 are exemplary nodes of the same subset of components, while in other implementations, nodes 200, 201 are respective nodes of separate subsets of components. Data router / ICI 112 includes a trace chain 207 that generally converges with trace chains 203 and 205 to provide trace data to chip manager 216.

[0064] 2, nodes 200, 201 may each include a respective component subset having at least FPC 104, SPC 106, node fabric 110, and HIB 114. Each component of node 200, 201 includes one or more trace muxes configured to group trace points (described below) generated by a particular component of the node. FPC 104 includes trace mux 204, node fabric 110 includes trace muxes 210a / b, SPC 106 includes trace muxes 206a / b / c / d, HIB 214 includes trace mux 214, and ICI 212 includes trace mux 212. In some implementations, a trace control register for each trace mux allows individual trace points to be enabled and disabled. In some examples, for one or more trace muxes, their corresponding trace control registers include individual enable bits. and more extensive trace mux control.

[0065] In general, the trace control register may be a conventional control status register (CSR) that receives and stores trace instruction data. With regard to trace mux control, in some implementations tracing may be enabled and disabled based on CSR writes performed by system 100. In some implementations tracing may be dynamically started and stopped by system 100 based on the value of a global time counter (GTC), based on the value of an example trace mark register in FPC 104 (or core 116), or based on the value of an example trace mark register in SPC 106.

[0066] Additional details and explanations relating to systems and methods for dynamically starting and stopping trace activity and for synchronized hardware event collection are described in more detail with reference to the example implementations of Figures 5-7.

[0067] In some implementations, for a core 116, the FPC 104 can use trace control parameters to define a trace window associated with event activity occurring within the core 116. The trace control parameters allow the trace window to be defined by lower and upper bounds for the GTC and lower and upper bounds for the trace mark register.

[0068] In some implementations, system 100 may include functionality that allows for a reduction in the number of generated trace entries, such as trace event filtering functionality. For example, FPC 104 and SPC 106 may each include filtering functionality that limits the rate at which each core sets trace bits in exemplary generated trace descriptors (described below). HIB 114 may include similar filtering functionality, such as an exemplary DMA rate limiter that limits the trace bits associated with the capture of certain DMA trace events. In addition, HIB 114 may include control (e.g., via an enable bit) for limiting the queuing of source DMA trace entries.

[0069] In some implementations, the descriptor for the DMA operation may have a trace bit set by an example compiler of the host system 126. Once the trace bit is set, a hardware function / knob that determines and generates trace data is used to complete the example trace event. In some examples, the final trace bit in the DMA may be a logical OR of the trace bit statically inserted by the compiler and the trace bit dynamically determined by a particular hardware component. Thus, in some examples, the compiler generated trace bit may provide a mechanism for reducing the overall amount of trace data generated apart from filtering.

[0070] For example, a compiler for host system 126 may decide to set only trace bits for one or more remote DMA operations (e.g., DMA across at least two nodes) and clear trace bits for one or more local DMA operations (e.g., DMA within a particular tensor node, such as node 200). In this way, the amount of trace data generated may be reduced based on trace activity limited to cross-node DMA operations, rather than trace activity that includes both cross-node (i.e., remote) and local DMA operations.

[0071] In some implementations, at least one trace event initiated by the system 100 may include a memory access event that includes multiple intermediate operations that occur throughout the system 100. A descriptor for a memory access operation (e.g., an MAE descriptor) may include a trace bit that causes data associated with the intermediate operations to be stored in one or more memory buffers. Thus, the trace bit may be used to "tag" intermediate memory operations at intermediate hops of a DMA operation and generate multiple trace events as the data packet traverses the system 100.

[0072] In some implementations, the ICI 112 may include a set of enable bits and a set of packet filters that provide control functionality for each ingress and egress port of a particular component of the node 200, 201. These enable bits and packet filters allow the ICI 112 to enable and disable trace points associated with a particular component of the node 200, 201. In addition to enabling and disabling trace points, the ICI 112 may be configured to filter trace data based on an event source, an event destination, and a trace event packet type.

[0073] In some implementations, in addition to using step markers, GTCs, or trace markers, each trace control register for processor cores 104, 106, and HIB 114 may also include an "everyone" trace mode. This "everyone" trace mode may allow tracing across the entire processor chip to be controlled by either trace mux 204 or trace mux 206a. When in the everyone trace mode, trace muxes 204 and 206a may send an "in window" trace control signal that specifies whether that particular trace mux, i.e., either mux 204 or mux 206a, is within the trace window.

[0074] The intra-window trace control signals may be broadcast or sent globally to all other trace muxes, for example, within a processor chip or across multiple processor chips. If either mux 204 or mux 206a is performing trace activity, all tracing may be enabled by broadcasting to the other trace muxes. In some implementations, the trace muxes associated with processor cores 104, 106, and HIB 114 each include a trace window control register that specifies when and / or how the "trace each" control signal is generated.

[0075] In some implementations, trace activity in trace mux 210a / b and trace mux 212 is generally enabled based on whether a trace bit is set in a data word for a DMA operation or control message traversing ICI / data router 112. A DMA operation or control message may be a fixed-size binary data structure that may have a trace bit within the binary data packet set based on certain status or software conditions.

[0076] For example, if a DMA operation is initiated in FPC 104 (or SPC 106) by a trace type DMA instruction and the initiator (processor core 104 or 106) is within the trace window, then the trace bit will be set in that particular DMA. In another example, for FPC 104, if FPC 104 is within the trace window and a trace point that causes trace data to be stored is enabled, then a control message for writing data to another component in system 100 will cause the trace bit to be set.

[0077] In some implementations, zero-length DMA operations provide an example of a broader DMA implementation within system 100. For example, some DMA operations may be In addition, non-DMA activity may be generated within system 100. Execution of non-DMA activity may also be traced (e.g., generate trace data) as if the non-DMA activity were a DMA operation (e.g., a DMA activity that includes a non-zero length operation). For example, a DMA operation that is initiated at a source location but has no data to be sent or transferred (e.g., is zero length) may instead send a control message to the destination location. The control message would indicate that the destination has no data to be received or acted upon. And the control message itself would be traced by system 100 just as non-zero length DMA operations are traced.

[0078] In some instances, for SPC 106, a zero-length DMA operation may generate a control message, and the trace bit associated with that message will be set only if the DMA caused the trace bit to be set, i.e., if the control message did not have a zero length. In general, a DMA operation initiated from host system 126 will cause the trace bit to be set if HIB 114 is within the trace window.

[0079] 2, trace chain 203 receives trace entry data for the component subset aligned with node 0, while trace chain 205 receives trace entry data for the component subset aligned with node 1. Each trace chain 203, 205, 207 is a separate data communication path used by respective nodes 200, 201, and ICI 112 to provide trace entry data to an exemplary trace entry data log 218 of chip manager 216. Thus, the endpoint of trace chains 203, 205, 207 is chip manager 216 where trace events may be stored in an exemplary memory unit.

[0080] In some implementations, at least one memory unit of chip manager 216 may be 128 bits wide and have a memory depth of at least 20,000 trace entries. In alternative implementations, at least one memory unit may have a larger or smaller bit width and a memory depth that can store more or fewer entries.

[0081] In some implementations, the chip manager 216 may include at least one processing device that executes instructions to manage the received trace entry data. For example, the chip manager 216 may execute instructions to scan / analyze timestamp data for each hardware event in the trace data received via the trace chains 203, 205, 207. Based on the analysis, the chip manager 216 may populate the trace entry log 218 to include data that may be used to identify (or generate) a chronological order of the hardware trace events. The hardware trace events may correspond to the movement of data packets that occurs at the component and subcomponent level as the processing units of the system 100 execute the exemplary distributed software program.

[0082] In some implementations, the hardware units of the system 100 may generate trace entries (and corresponding timestamps) that populate the exemplary hardware trace buffer in a non-chronological (i.e., out of order) manner. For example, the chip manager 216 may cause multiple trace entries having the generated timestamps to be inserted into the entry log 218. The inserted trace entries may not be time-sequenced with respect to each other. In this implementation, the non-chronological trace entries may be populated in the exemplary host buffer of the host system 126. Once received by the host buffer, the host system 126 can execute instructions associated with the performance analysis / monitoring software to scan / analyze the timestamp data for each trace entry. The executed instructions can be used to sort the trace entries and to build / generate a timeline of hardware trace events.

[0083] In some implementations, trace entries may be removed from the entry log 218 via host DMA operations during a tracing session. In some examples, the host system 126 may not remove DMA entries from the trace entry log 218 as fast as they are added to the log. In other implementations, the entry log 218 may include a predefined memory depth. When the memory depth limit of the entry log 218 is reached, additional trace entries may be lost. To control which trace entries are lost, the entry log 218 may operate in a first-in-first-out (FIFO) mode, or alternatively, in an overwrite recording mode.

[0084] In some implementations, the overwrite recording mode may be used by system 100 to support performance analysis associated with post-mortem debugging. For example, program code may be executed for a period of time with trace activity enabled and overwrite recording mode enabled. In response to a post-mortem software event (e.g., program corruption) in system 100, monitoring software executed by host system 126 may analyze the data contents of an exemplary hardware trace buffer to understand hardware events that occurred before the program corruption. As used herein, post-mortem debugging relates to the analysis or debugging of program code after the code has been corrupted or has become largely unable to execute / operate as intended.

[0085] In the FIFO mode, if the entry log 218 is full and if the host system 126 removes a stored log entry within a time frame, the new trace entry may not be stored in the memory unit of the chip manager 216 to conserve memory resources. On the other hand, in the overwrite recording mode, if the entry log 218 is full and if the host system 126 removes a stored log entry within a time frame, the new trace entry may overwrite the oldest trace entry stored in the entry log 218 to conserve memory resources. In some implementations, the trace entries are moved to the memory of the host system 126 in response to a DMA operation using the processing capabilities of the HIB 114.

[0086] As used herein, a trace point is the origin of a trace entry and the data associated with that trace entry that is received by chip manager 216 and stored in trace entry log 218. In some implementations, a multi-core, multi-node processor microchip may include three trace chains within the chip, where a first trace chain receives trace entries from chip node 0, a second trace chain receives trace entries from chip node 1, and a third trace chain receives trace entries from the chip's ICI router.

[0087] Each trace point has a unique trace identification number within its trace chain that it inserts into the header of the trace entry. In some implementations, each trace entry identifies the trace chain in which it occurs in its header, indicated by one or more byte / bit data words. For example, each trace entry may identify a particular It may include a data structure with a defined field format that conveys information about a trace event (e.g., a header, a payload, etc.) Each field in a trace entry corresponds to useful data applicable to the trace point that generated the trace entry.

[0088] As mentioned above, each trace entry may be written or stored in a memory unit of the chip manager 216 associated with the trace entry log 218. In some implementations, trace points may be individually enabled or disabled, and multiple trace points may generate trace entries of the same type but with different trace point identifiers.

[0089] In some implementations, each trace entry type may include a trace name, a trace description, and a header that identifies the encoding for a particular field and / or set of fields in the trace entry. The name, description, and header collectively provide a description of what the trace entry represents. From the chip manager 216's perspective, this description may also identify the particular trace chain 203, 205, 207 that a particular trace entry participated in within a particular processor chip. Thus, a field in the trace entry represents a descriptive piece of data (e.g., in bytes / bits) and may be a trace entry identifier that is used to determine which trace point generated a particular trace entry.

[0090] In some implementations, trace entry data associated with one or more of the stored hardware events may correspond, in part, to data communications occurring a) at least between node 0 and node 1, b) at least between components within node 0, and c) at least between components within node 1. For example, the stored hardware events may correspond, in part, to data communications occurring at least one of: 1) between FPC 104 of node 0 and FPC 104 of node 1, between FPC 104 of node 0 and SPC 106 of node 0, and 2) between SPC 106 of node 1 and SPC 106 of node 1.

[0091] 3 illustrates a block diagram of an exemplary trace multiplexing design architecture 300 and an exemplary data structure 320. The trace multiplexing design 300 generally includes a trace bus in 302, a bus arbiter 304, and a local trace point arbiter 306, a bus FIFO 308, at least one local trace event queue 310, a shared trace event FIFO 312, and a trace bus out 314.

[0092] The multiplexing design 300 corresponds to an exemplary trace mux located within a component of the system 100. The multiplexing design 300 may include the following functionality: A bus in 302 may be associated with local trace point data that is temporarily stored in a bus FIFO 308 until time arbitration logic (e.g., arbiter 304) causes the trace data to be placed into an exemplary trace chain. One or more trace points for the component may insert trace event data into at least one local trace event queue 310. An arbiter 306 provides a first level of arbitration, allowing the selection of an event from the local trace events stored in the queue 310. The selected event is placed into a shared trace event FIFO 312, which also functions as a storage queue.

[0093] The arbiter 304 receives local trace events from the FIFO queue 312 and provides a second level of arbitration by merging the local trace events onto a particular trace chain 203, 205, 207 via the trace bus out 314. In some implementations, trace entries may be pushed into the local queue 310 faster than they can be merged into the shared FIFO 312. Or, alternatively, trace entries may be pushed into the shared FIFO 312 faster than they can be merged onto the trace bus 314. If these scenarios occur, then each queue 310 and 312 will fill up with trace data.

[0094] In some implementations, when either queue 310 or 312 fills with trace data, the system 100 may be configured to drop the most recent trace entries so that they are not stored or merged into the particular queue. In other implementations, rather than dropping trace entries when a queue (e.g., queues 310, 312) becomes full, the system 100 may be configured to stall the example processing pipeline until the once-full queue has available queue space to receive an entry.

[0095] For example, a processing pipeline that uses the queues 310, 312 may be stalled until a sufficient or threshold number of trace entries are merged onto the trace bus 314. The sufficient or threshold number may correspond to a particular number of merged trace entries that results in available queue space for one or more trace entries to be received by the queues 310, 312. Implementations in which the processing pipeline is stalled until downstream queue space is available may provide higher fidelity trace data based on trace entries being kept rather than dropped.

[0096] In some implementations, the local trace queue is as wide as required by the trace entries, such that each trace entry occupies only one place in the local queue 310. However, the shared trace FIFO queue 312 may use a unique trace entry line encoding, such that some trace entries may occupy two places in the shared queue 312. In some implementations, if any data in a trace packet is dropped, the entire packet is dropped, so that partial packets do not appear in the trace entry log 218.

[0097] In general, a trace is a timeline of activity or hardware events associated with a particular component of system 100. Unlike performance counters (described below), which are aggregate data, a trace contains detailed event data that provides insight into the hardware activity occurring during a specified trace window. The described hardware system enables large-scale support for distributed hardware tracing, including generation of trace entries, temporary storage of trace entries in hardware-managed buffers, static and dynamic enabling of one or more trace types, and streaming of trace entry data to host system 126.

[0098] In some implementations, traces may be generated for hardware events performed by components of system 100, such as generating a DMA operation, executing a DMA operation, issuing / executing an instruction, or updating a synchronization flag. In some examples, trace activity may be used to track DMA through the system or to track instructions executing on a particular processor core.

[0099] The system 100 can be configured to generate at least one data structure 320 that identifies one or more hardware events 322, 324 from a timeline of hardware events. In some implementations, the data structure 320 arranges the one or more hardware events 322, 324 in a chronological order of events associated with at least the FPC 104 and the SPC 106. In some examples, the system 100 generates at least one data structure 320 that identifies one or more hardware events 322, 324 from a timeline of hardware events. In some implementations, the data structure 320 arranges the one or more hardware events 322, 324 in a chronological order of events associated with at least the FPC 104 and the SPC 106. The data structure 320 may be stored in a memory bank of a host control device of the host system 126. The data structure 320 may be used to evaluate the performance of program code executed by at least the processor cores 104 and 106.

[0100] In some implementations, a particular trace identification (ID) number (e.g., trace ID '003) may be associated with multiple hardware events occurring across distributed processing units, as illustrated by hardware event 324. The multiple hardware events may correspond to a particular memory access operation (e.g., DMA), and the particular trace ID number is used to correlate one or more hardware events.

[0101] For example, a single trace ID for a DMA operation may include multiple timestamps corresponding to multiple different points in the DMA, as shown by event 324. In some examples, trace ID '003 may have a "issued" event, an "executed" event, and a "completed" event identified as being some time apart relative to each other. Thus, in this regard, the trace ID may be further used to determine latency attributes of the memory access operation based on the correlation and with reference to the timestamps.

[0102] In some implementations, generating the data structure 320 may include, for example, the system 100 comparing an event timestamp of each event in the first subset of hardware events to an event timestamp of each event in the second subset of hardware events. Generating the data structure 320 may further include the system 100 providing a set of correlated hardware events for presentation in the data structure based in part on the comparison of the first subset of events to the second subset of events.

[0103] 3, the data structure 320 may identify at least one parameter indicative of a latency attribute of a particular hardware event 322, 324. The latency attribute may indicate at least a duration of the particular hardware event. In some implementations, the data structure 320 is generated by software instructions executed by a control device of the host system 126. In some examples, the structure 320 may be generated in response to the control device storing the trace entry data to a memory disk / unit of the host system 126.

[0104] 4 is a block diagram 400 illustrating an example trace activity for a direct memory access (DMA) trace event performed by system 100. For DMA tracing, data about an example DMA operation occurring from a first processor node to a second processor node may travel through ICI 112 and generate intermediate ICI / router hops along the data path. As the DMA operation traverses ICI 112, it will generate a trace entry at each node within the processor chip and along each hop. Information is captured by each of these generated trace entries to reconstruct the time progression of the DMA operation along the nodes and hops.

[0105] An exemplary DMA operation may be associated with the process steps shown in the implementation of Figure 4. For this operation, a local DMA transfers data from a virtual memory 402 (vmem 402) associated with at least one of the processor cores 104, 106 to the HBM 108. The numbering shown in diagram 400 corresponds to the steps in table 404 and generally corresponds to activity in or initiated by the node fabric 110. Represents the activities that can be carried out.

[0106] The steps in table 404 generally describe the associated trace points. An exemplary operation would generate six trace entries for this DMA. Step 1 involves the initial DMA request from the processor core to node fabric 110, which generates a trace point in the node fabric. Step 2 involves a read command that asks node fabric 110 to transfer data to the processor core, which generates another trace point in node fabric 110. Once vmem 402 has completed reading node fabric 110, the exemplary operation has no trace entries for step 3.

[0107] Step 4 involves node fabric 110 making a read resource update to cause a synchronization flag update in the processor core, which generates a tracepoint in the processor core. Step 5 involves node fabric 110 notifying memory mux 108 that the next data is to be written to HBM. The notification via the write command generates a tracepoint in node fabric 110, while in step 6, the completion of the write to HBM also generates a tracepoint in node fabric 110. In step 7, node fabric 110 makes a write resource update to cause a synchronization flag update in the processor core, which generates a tracepoint in the processor core (e.g., in FPC 104). In addition to the write resource update, node fabric 110 can make an acknowledgement update (ack update) in which data completion for the DMA operation is signaled back to the processor core. The ack update can generate a trace entry similar to the trace entry generated by the write resource update.

[0108] In another exemplary DMA operation, when a DMA instruction is issued in the node fabric 110 of the originating node, a first trace entry is generated. Additional trace entries may be generated in the node fabric 110 to capture the time used to read data for DMA and write the data to a queue for transmission. In some implementations, the node fabric 110 may packetize the DMA data into smaller chunks of data. For data packetized into smaller chunks, read and write trace entries may be generated for the first and last data chunks. Optionally, all data chunks in addition to the first and last data chunks may be set to generate trace entries.

[0109] For remote / non-local DMA operations that may require ICI hops, the first and last data chunks may generate additional trace entries at the ingress and egress points at each intermediate hop along the ICI / router 112. When the DMA data arrives at the destination node, a trace entry similar to the previous node fabric 110 entry is generated at the destination node (e.g., read / write of the first and last data chunks). In some implementations, the last step of the DMA operation may include an executed instruction associated with the DMA causing a synchronization flag to be updated at the destination node. Once the synchronization flag is updated, a trace entry may be generated indicating the completion of the DMA operation.

[0110] In some implementations, DMA tracing is initiated by the FPC 104, SPC 106, or HIB 114 when each component is in trace mode so that trace points can be executed. Components of the system 100 can enter trace mode based on global control in the FPC 104 or SPC 106 via a trigger mechanism. Trace mode can be initiated in response to the occurrence of a particular action or condition associated with the execution of program code by a component of the system 100. For example, a portion of the program code may include an embedded trigger function that is detectable by at least one hardware component of the system 100.

[0111] Components of system 100 may be configured to detect a trigger function associated with a portion of program code executed by at least one of FPC 104 or SPC 106. In some examples, the trigger function may correspond to at least one of 1) a particular sequence step in the executed portion or module of program code, or 2) a particular time parameter indicated by a GTC used by a distributed processor unit of system 100.

[0112] In response to detecting a trigger function, a particular component of the system 100 may initiate, trigger, or execute at least one trace point (e.g., a trace event) that causes trace entry data associated with one or more hardware events to be stored in at least one memory buffer of the hardware component. As described above, the stored trace data may then be provided to the chip manager 216 via at least one trace chain 203, 205, 207.

[0113] As discussed above, Figures 1-4 have illustrated approaches for hardware event tracing in a distributed computing system. The remaining Figures 5-7 describe approaches for synchronized collection of hardware events in a distributed computing system. The synchronized collection of hardware events may be based at least in part on one or more trigger mechanisms.

[0114] 5 illustrates an example data structure for a global time counter (GTC) 502 and tables 504 and 506 illustrating usage scenarios of the GTC 502 by an example computing system (e.g., system 100) for distributed hardware tracing. In the implementation of FIG. 5, the GTC 502 is a 64-bit value that includes a 60-bit counter and a 4-bit offset. In alternative implementations, the GTC 502 may be a fixed-size binary piece of data with a size in bits ranging from less than 64 bits to more than 64 bits.

[0115] The data structure associated with the GTC 502 applies not only to the global master counter, but also to the local time counter (LTC) described below with reference to the implementation of FIG. 6. The upper 60 bits of the GTC 502 correspond to a real-time counter that increments by one every cycle, except when "catching up" or "catching down" to accommodate a GTC phase change (described below). For example, during an update of the local GTC 502 (i.e., the LTC), the local GTC 502 cannot "tick" for several cycles in order to "catch down" to the master global GTC 502. In these scenarios, the lower four "offset" bits are used to compensate for GTC phase changes that may occur during program code execution.

[0116] When dealing with GTC phase changes, the last 4 bits may be incremented in these situations. This is in part to maintain distinction between timestamps for local hardware event activity where the GTCs would otherwise be identical. In most other examples, the 4 offset bits of GTC 502 are clear and therefore unused. In some implementations, the offset bits may count up to 15 and stop there, while alternative implementations may count up to any integer value as required by the particular counter design. In some examples, the offset bits are cleared when the upper 60 bits of GTC 502 start ticking again. can be.

[0117] With regard to configuration and management of the GTCs 502, 602 (described below), in some implementations, the executed program code may configure or set up the GTCs by configuring one or more processor chips of the system 100. Configuring the processor chips may include designating a global master GTC for the system and designating a local master GTC (i.e., LTC) for each node or component of the system 100.

[0118] The components may include hardware knobs / functions that the executing program code / software uses to calculate and compensate for clock latency of one or more local GTCs (i.e., the LTCs of FIG. 6) and to make manual adjustments to the LTCs. Compensation and manual adjustments of clock latency of the LTCs are made to the master GTC to minimize or mitigate phase changes. In some implementations, the system 100 may include an autopilot mode for latency compensation in which the component hardware functions automatically determine link latency via a "ping and echo" mechanism.

[0119] In some implementations, the system 100 may include a link / software stack that includes a portion or subset of program code used to adjust parameter values ​​of exemplary control / status registers (CSRs). In some examples, control signals generated in response to execution of the link stack program code cause one or more parameter values ​​to be written to the CSRs. These CSR parameter values ​​may be used to configure various aspects of the GTC 502, 602 or LTC.

[0120] For example, the one or more parameters may include: 1) a GTC_config parameter that causes the selection of a master global GTC, a local master GTC, or a local slave GTC and the selection of an update sample size; 2) a GTC_sync parameter used to reset the master GTC or clear the "ping" measurement; 3) a GTC_latency_compensation_control parameter that configures manual latency compensation; 4) a GTC_link_ping parameter that indicates the minimum ping latency; 5) a GTC_count parameter that indicates the actual GTC value; and / or 6) a GTC_max_diff parameter that indicates the maximum observed difference between the local GTC and the master GTC.

[0121] As described above, the global master GTC 502 and one or more local GTCs 502 may be used by the system 100 to obtain event timestamps and to determine an ordered ordering between hardware events occurring within the system. In addition to ordering between activities, the GTCs 502 may be used to determine the latency of activities. In some implementations, a particular GTC 502 may exhibit phase changes, and thus the determined latency may be inaccurate by one or more cycles.

[0122] As mentioned above, the rate at which a GTC 502 increments can vary depending on whether the local GTC is different from the global master, i.e., if there is a phase change. In steady state, in the absence of a phase change, the master global GTC 502 increments once per cycle, just like the LTC of a particular component, and can be used to count the latency and order of local activity (e.g., of a component).

[0123] In some implementations, the master GTC 502 may count slower than an LTC. In these examples, the offset bits of the GTC 502 continue to increment once per local clock cycle. Given the unique functionality of the GTC offset bits, as shown by tables 504, 506, an extended GTC (GTC-ext) may be defined for use by the system 100. In some implementations, the GTC-ext may be constructed by adding the most significant 60 bits (GTC:top) and the least significant 4 offset bits (GTC:offset). Meanwhile, in other implementations, program code running on a particular processor may calculate the GTC-ext such that the fractional value of the GTC:offset is equal to the summed GTC:top.

[0124] In some implementations, in response to incrementing the offset bits, the GTC-ext parameters may be used to determine event ordering and local operation latency. Tables 504, 506 show usage scenarios in which the GTC-ext parameters may prove useful in determining hardware event ordering and event operation latency.

[0125] Table 504 illustrates an example of using the GTC to determine ordering between activities or hardware events. In some implementations, when using the GTC 502 to determine a global order, e.g., an order of one or more activities between at least two separate compute nodes (e.g., separate nodes each having a master GTC), the rate at which the GTC 502 increments during activity analysis may not be appropriate. However, if the local GTC 502 at a particular node is slower than the master GTC for that node, the GTC-ext may be used to distinguish between the two local activities or the time at which the two local activities initiated a write to a remote resource, as shown by function 508 in table 504.

[0126] Table 506 illustrates an example of using the GTC to calculate the latency of local activities / events or activities that span two or more nodes. In some implementations, system parameters may not indicate whether the GTC 502 on a particular chip is ticking at a normal rate (i.e., in sync with a designated master global clock) or at a rate that is out of phase with a designated master global clock. Thus, in some examples, the LTC may be used for event latency determination of local activities, as indicated by function 510 of table 506, while in other implementations the GTC or GTC-ext may be used to measure the latency of one or more operations.

[0127] In general, hardware trace events occurring at various nodes or components of the system 100 may include GTC parameters. In some implementations, when traced hardware events / activities span multiple nodes of the system 100, the latency associated with the local activity should be calculated via the GTC rather than the LTC. In these implementations, the use of GTC or GTC-ext may be the preferred counter / clock solution because LTCs across two or more separate nodes may not be synchronized or clock / tick at the same rate.

[0128] 6 illustrates a block diagram of an example time counter and one or more counters associated with respective components of an example computing system (e.g., system 100) for distributed hardware tracing. The time counters may include a global time counter (GTC) 602 and multiple local time counters (LTCs) each associated with a respective component of system 100.

[0129] For clarity, Fig. 6 illustrates a single LTC block coupled to a particular component of system 100, but in alternative implementations, each component may include one or more local time counters / LTCs. Also, while Fig. 6 illustrates a single master GTC 602, this functionality may also correspond to or represent a particular LTC selected to function as a global master. For example, in some implementations, GTC 602 corresponds to counter selection logic implemented by host system 126 to select a particular LTC to function as a global master clock / counter.

[0130] In some examples, the system 100 may select an exemplary node of the ICI 112 to function as a global master clock, with the exemplary node's LTC ticking once every clock cycle, while in an alternative implementation, the GTC 602 is a global time counter that acts as a global master clock and broadcasts certain time parameters to components of the system 100. In either implementation, the GTC 602 may be a globally synchronized clock used to correlate hardware events occurring throughout the system 100.

[0131] In some implementations, as described above, the GTC 602 may be configured to avoid long-term time drift, although it may have small phase changes across the system 100. For example, the GTC 602 may tick or count based on an exemplary oscillator of a particular processor chip (e.g., an oscillator selected to be the global master). Generally, there is no long-term drift or phase change specific to the selected oscillator of a processor chip in the system 100, but over time drift or changes may occur for oscillators between separate processor chips. Thus, the GTC 602 may increment (i.e., tick) faster or slower than a local clock (LTC) on a processor chip that is not the global master.

[0132] As a global master, the current master value of the GTC 602 may be broadcast to all nodes of the ICI 112. An exemplary ICI node may correspond to one or more components of the system 100. The receiving node may calculate the difference in its local value of the exemplary GTC used for local operations specific to the particular component / node. The minimum difference between the received master value and the local GTC value over an exemplary sampling period may be used to update the local GTC. Over time, the phase change, if any, between the master value and the local GTC value may be "caught up" based on the counter adjustment achieved using one or more offset bits described above.

[0133] As shown in the example implementation of Figure 6, each of the components of system 100 may include one or more performance counters. A collective set or subset of performance counters in system 100 is referred to herein as a number of counters 600. In particular, specifically for each component, FPC 104 may include a number of counters 604, SPC 106 may include a number of counters 606, mux memory (HBM) 108 may include a number of counters 608, node fabric 110 may include a number of counters 610, ICI 112 may include a number of counters 612, and HIB 114 may include a number of counters 614.

[0134] In system 100, performance analysis of program code executed by one or more processors may be enabled / disabled based on the use of one or more hardware performance counters associated with particular components of the system. These performance counters may correspond to the respective counters of FIG. 6, including: 1) activity counters; ) a stall counter, 3) a statistical counter, and 4) a sampling counter.

[0135] In some implementations, system 100 includes various other types of performance counters that can be programmed to increment and store specified count data associated with technical aspects of the executed program code. Through the use of one or more counters, hardware support for tracing activity can also include functionality related to, for example, capturing instruction tracing event data, capturing memory mux tracing event data, capturing DMA tracing event data, and capturing event data in a trace buffer.

[0136] In some implementations, at least a subset of the plurality of counters 600 may be CSRs or count registers accessible by the host system 126 via the HIB 114. These count registers / CSRs may be storage devices configured to store various types of count data / information identifying instructions received, processed, or executed by a particular hardware component, as well as various other types of data / information associated with technical aspects of the program code being executed. In some implementations, the data is stored in the form of counts or increments (e.g., count parameters) associated with a particular instruction or processor operation. In some implementations, the plurality of counters 600 of the system 100 may correspond to hundreds or thousands of performance counters / count registers.

[0137] With respect to the at least four types of performance counters described above, in some implementations, activity counters may be used to store and analyze data associated with utilization parameters for different hardware units. Activity counters may also be used to analyze portions of program code being executed, or to analyze the amount or type of data being transferred during code execution. For example, some processor cores may include activity counters that are used to derive instruction mix in various streams of instructions. For example, activity counters may be configured to store counts associated with the execution of particular instruction types in an exemplary instruction stream.

[0138] In some implementations, the activity counters may include a subset of counters, such as issue counters that are incremented upon issuance of an instruction type, memory activity counters that are available on the memory path and are incremented in response to reads / writes associated with memory transfers (e.g., between mux108 and VMEM402), and one or more other general activity counters that are incremented in response to the occurrence of an instruction, such as interrupts, synchronization flags, and warning counters.

[0139] Regarding the stall counters, in some implementations, these counters can provide a stall count that indicates the number of cycles that a hardware unit / component has stalled due to a particular stall reason instead of doing useful work associated with instruction processing or data computation. For example, if the utilization of a particular hardware unit is below a utilization threshold or if the program code is not functioning at a desired threshold performance level, the stall counters can be used by the system 100 to obtain appropriate stall information for performing a causal analysis of the utilization and performance issues (e.g., below a utilization / performance threshold).

[0140] In some implementations, the stall counter may be configured to count stalls based on detection of a normally available memory path being marked as unavailable, stalls based on delays in execution of one or more synchronization flag instructions, and / or stalls based on detection of one or more normally available memory paths being marked as unavailable. Appropriate information related to stalls caused by an execution unit (eg, XU 122 or RPU 124) may be provided.

[0141] In some implementations, the FPC 104 may include at least one scalar unit that provides scalar quantities for vectorized computations performed by the VPU 118. Stall counters associated with the scalar unit may provide appropriate information about stalls due to one or more hazards that may occur within the scalar unit. In some examples, various hazard types may occur, and the FPC 104 may include stall counters for each hazard type. Exemplary hazard types may include DMA delay hazards, scalar fence delay hazards, and / or scalar delayed instruction hazards.

[0142] With regard to statistical counters, in some implementations, components of the system 100 may include one or more data queues (e.g., storage media), and activity counters may be configured to provide information related to queue utilization. However, in some examples, activity counters for a queue may not be sufficient to provide all information regarding how the queue is being utilized by the executing program code. In these examples, statistical counters may be available in one or more components to obtain certain data counts so that calculations related to average queue occupancy and time spent in a queue may be determined.

[0143] In some implementations, the queue statistics counters may provide statistical counts related to queues at queue occupancy, queue insert status, queue full status, or threshold occupancy status. In some examples, when the queue statistics counters are active, an occupancy counter associated with a component queue is incremented based on the current occupancy of the queue at each processor / instruction cycle.

[0144] In another example, if an active queue statistics counter provides a count related to a queue full status, the counter is incremented (e.g., by one) on the cycle in which the queue is fully occupied. And, if an active queue statistics counter provides a count related to a queue insert status, the counter is incremented in response to a data chunk being placed in the queue (i.e., a queue insert). Furthermore, for a queue in a threshold occupancy status, the counter may be incremented (e.g., by one) on each cycle in which the current queue occupancy is greater than a particular queue threshold occupancy.

[0145] In some implementations, the components of system 100 may include a collection of sampling counters that may be periodically read from host system 126 to generate a data structure (e.g., structure 320 or 620). The data structure may include sampled statistics regarding hardware events, trace activity, or detailed data processing information about activity / events occurring at distributed processing units of system 100. In some implementations, the data structure generated based on the sampled statistics may include, for example, structure 320 (above) or structure 620 (described below). In some implementations, the generated data structure is used to build one or more monitoring tools that are used by monitoring engine 128 of host system 126 to analyze the performance of program code executed by the components of system 100.

[0146] As described above, components of the system 100 can initiate event tracing based on global controls associated with one or more trigger mechanisms. In some implementations, the plurality of counters 600 may include one or more The hardware event may be enabled, disabled, or otherwise controlled to initiate a trigger mechanism that executes and generates a trace entry containing specific data related to the hardware event. In some implementations, the trace entry related to the hardware event is populated with event data based at least in part on count data collected by one or more of the plurality of counters 600 described above.

[0147] Global control generally involves the use of CSR controls (i.e., specific CSR hardware functions / knobs) and trigger mechanisms that allow counters to be incremented and traces to be generated in a coordinated, synchronized manner. In some implementations, coordinated, synchronized control may occur across distributed processor units of system 100. As mentioned above, synchronized tracing may be enabled in part through the use of a globally synchronized counter that is used as a global master clock (e.g., GTC 602).

[0148] In some implementations, the system 100 triggers synchronized event tracing based on a particular time parameter (e.g., time parameter 09:01.13) indicated by the global time clock of the distributed processor units. For example, the system 100 can use the global master clock value of the GTC 602 as a trigger to precisely start and stop all counters and tracing for the duration of an operation.

[0149] In some examples, an example software compiler of system 100 may embed or insert instructions into the executed program code including operand or parameter values ​​for a trigger mechanism based on a particular predefined time window associated with GTC 602. The predefined time window may include an example trace duration having a trace event start time (e.g., a first GTC 602 time parameter) and a trace event stop time (e.g., a second GTC 602 time parameter that is later in time than the first GTC time parameter).

[0150] 6, components of system 100 may include multiple performance counters 600. As it relates to the trigger mechanism, performance counter control may include mechanisms for enabling and disabling the counters, as well as for clearing and pausing the counters. Also, in addition to the statistical counter functionality described above, some hardware units may include statistical counters with associated selectors (e.g., CSRs) that are used to select a subset of hardware blocks within the hardware unit for which events are counted.

[0151] As described above with reference to trace mux control, in some implementations tracing may be enabled and disabled based on CSR writes performed by system 100. In some examples, tracing may be dynamically started and stopped by system 100 based on the value of an example trace-mark (TM) register in FPC 104 (or core 116) or based on the value of an example TM register in SPC 106.

[0152] In some implementations, example trace control parameters may be used to define a particular trace window. For example, the trace control parameters may allow the trace window to be defined by lower and upper bounds (e.g., start time & stop time clock values) of the GTC 602 and lower and upper bounds of an example trace mark register in the FPC 104 or an example trace mark register in the SPC 106. It is possible.

[0153] In some implementations, program code executing within system 100 may include trigger processing logic for triggering a trace event at each processor cycle in one of FPC 104 or SPC 106. In some examples, the trigger logic is associated with a special "set trace mark" instruction that may be inserted into one or more instruction streams by the software / program code being executed.

[0154] For example, with respect to trigger logic for FPC 104, when an exemplary instruction to set a trace mark is issued within FPC 104, the trigger logic may include analyzing a trace mark operand of the instruction and comparing a parameter value of the operand to values ​​of an exemplary trace mark window. This analysis and comparison is used to determine whether the operand parameter value is equal to a lower limit value of a particular trace mark window or falls within lower and upper limits of the trace mark window.

[0155] If the operand parameter value is within the limits of the trace mark window, a trigger may be activated (e.g., a trigger condition may be satisfied) and FPC 104 may enter a tracing or counting mode to enable one or more counters of counters 604. In some implementations, if the next "set trace mark" instruction includes an operand parameter value that activates a counting stop trigger based on the operand value being equal to or outside the upper limit of an exemplary trace mark window, FPC 104 exits the tracing / counting mode.

[0156] In some implementations, the system 100 can detect that a trigger condition is met based on at least one of: i) identifying an occurrence of an exemplary trace mark operand in at least a first portion of the executed program code. The trace mark operand may include a parameter value (e.g., a code step sequence / value) used to initiate one or more trace events and / or performance counting. The system 100 can also detect that a trigger condition is met based on a determination that the current time of the GTC 602 indicates a predefined time value used to initiate one or more trace events and / or performance counting.

[0157] In response to detecting that a trigger condition is satisfied, the processor component of the system 100 may generate a control signal that is received by a count register of the processor component. The control signal may cause various count data associated with one or more hardware events to be stored in the count register. In some implementations, the system 100 may be configured or programmed, for example via the host system 126, to generate a data structure indicative of one or more performance attributes associated with the executed program code. For example, the data structure may be generated based on one or more count parameters of the stored count data. In some implementations, the count register is one of a plurality of performance counters 600 configured to store count data related to the performance of one or more processor components of the system 100.

[0158] To further illustrate the trace mark trigger, in the exemplary code sequence, the trace mark setting instruction may include a 32-bit trigger operand that identifies a distributed software construct (e.g., a computation step of an exemplary neural network inference workload). s". Although 32-bit operands are described in this example code sequence, in alternative implementations, the trigger operand or register limit operand / parameter value may be a binary data structure having fewer or more than 32 bits.

[0159] This special purpose CSR may encode at least two 32-bit values. A first encoded 32-bit value may correspond to a trace mark lower limit used to start tracing / counting, while a second encoded 32-bit value may correspond to a trace mark upper limit used to stop tracing / counting. In some implementations, the first and second encoded values ​​correspond to computational steps in a distributed software program / construct, such as an exemplary neural network inference workload, or any other distributed program code.

[0160] Referring back to the exemplary code sequence, when the "set trace mark" instruction is executed, the 32-bit operand value is compared to the first and second coded values ​​in the tracemark_limits register. Based on this comparison, if the trigger operand is equal / matches or has a value higher than the start value (i.e., the first coded value), a performance analysis activity is triggered. Similarly, if the trigger operand is equal / matches or has a value higher than the stop value (i.e., the second coded value), an ongoing performance analysis activity is stopped.

[0161] Much like the trace mark trigger, when the GTC 602 is used as a trigger (e.g., for the FPC 104 or SPC 106), the upper and lower limits of an exemplary GTC / LTC trigger register for a particular component may be evaluated against the current value of the GTC 602. Thus, to perform a trace activity using the GTC trigger, the current time parameter or value of the GTC 602 may be compared to the lower limit of a predefined exemplary time window. Similarly, after execution of the GTC trigger to begin a counting activity, the current time parameter or value of the GTC 602 may be compared to the upper limit of the GTC trigger register to determine whether the GTC 602 matches or deviates from the upper limit value. In response to the GTC 602 matching at least the upper limit value, a counting stop trigger will be actuated to cause the component to exit counting mode.

[0162] In some implementations, the SPC 106 includes an equivalent of the trace mark setting instruction of the FPC 104. Thus, the triggering logic for the SPC 106 can function in the same or substantially similar manner as in the FPC 104. For example, a trace mark parameter may be included as an operand to one or more instructions issued by the SPC 106. In some examples, trigger evaluation of the operand occurs at each instruction issue, and the evaluation logic may be substantially similar to the manner in which counting is enabled in the FPC 104 (e.g., using a trace mark operand). For example, in the SPC 106, tracing is enabled if the trace mark operand included in the trace mark setting instruction is within the lower and upper bounds of an exemplary trace window. Similarly, tracing is disabled if the trace mark operand included in the trace mark instruction is outside the lower and upper bounds of the trace window.

[0163] In addition to using GTC 602 or trace markers as described above, system 100 may include a "for each" trace mode. In general, this for each trace mode may allow tracing across a processor chip to be controlled by control signals generated by components of, for example, FPC 104 or SPC 106. In some implementations, the control signals may be generated by the components of the distributed processor units of system 100. It may be a global control parameter that indicates a particular performance state, which in some examples corresponds to an "each one" trace mode.

[0164] In some implementations, one or more hardware components (e.g., HIB 114 or ICI 112) may be programmed or configured to initiate event tracing in response to activation of the "each" trace mode, or to simply ignore the propagating control signal indicating the each trace mode. Thus, while system 100 may include a signaling mechanism for propagating the each trace control signal, system 100 may also include a mechanism for causing at least one hardware component to ignore the control signal.

[0165] For example, a user of system 100 (e.g., a performance analyst) may want to use HIB 114 to trace data packets representing all host communications. The user may insert, e.g., via an external compiler, exemplary program instructions to cause HIB 114 to enter a "always trace" mode while also using the respective trace modes to capture certain single step sequences of data from other hardware components of system 100. Thus, when programmed to run in the always trace mode, HIB 114 may ignore the respective trace control signals used to generate single step sequence trace data from other hardware components.

[0166] In some implementations, tracepoints can be triggered in response to the occurrence of a particular action or condition associated with the execution of program code by components of system 100. For example, a distributed software program can include multiple software modules that each include shared or overlapping code sequences or steps. For example, processor cores 104, 106 can each receive and execute a portion of a distributed software program, where each program portion can include exemplary code sequences 1-100.

[0167] As mentioned above, in some implementations, the system 100 may be configured to include one or more trigger conditions / mechanisms having or based on particular sequence steps of the distributed program code. Thus, the system 100 may be configured to trigger a tracepoint in response to each of the processor cores 104, 106 arriving at a certain coded sequence / step (e.g., code sequence step 33) during code execution. In some examples, portions of the executed distributed program code may include embedded trigger functions that are detectable by at least one hardware component of the system 100.

[0168] For example, as described above, system 100 may be configured to detect that one or more trigger conditions are met based on identifying an occurrence of an operand in at least a portion of program code executed by one of FPC 104 or SPC 106. In response to detecting that the at least one trigger condition is met, system 100 may initiate one or more trace events that generate trace entries / trace data or count data. In some implementations, the generated trace entries may include at least one attribute (e.g., a shared timestamp or code sequence step) that is shared between respective hardware events that occur across the distributed processing units.

[0169] For example, as shown by data structure 620, generated trace entries 622 may correspond to synchronized trace events, with the trace data being The trace entry 622 may include a unique trace identifier for the hardware event and a global hardware event timestamp (e.g., 09:01.13) that is shared between at least two respective hardware events. Thus, for trace entry 622, the at least one attribute shared between the respective hardware events may be the hardware event timestamp. Similarly, for trace entry 624, if system 100 triggered a tracepoint based on a particular code sequence, the at least one attribute shared between the respective hardware events may be the code sequence or the trace mark step.

[0170] In some implementations, the trace entry may include a timestamp header or field. The timestamp header may include a 48-bit timestamp that identifies the time the trace entry was generated. In some examples, the timestamp header may correspond to the lower 48 bits of the GTC 502.

[0171] As described above, in some implementations, a compiler of system 100 may insert one or more operands for multiple trigger conditions into a portion of program code executed by an example processor core of system 100. For example, a compiler associated with FPC 104 may insert or embed one or more operands for a trigger condition into a portion of program code executed by FPC 104. Similarly, a compiler associated with SPC 106 may insert or embed one or more operands for a trigger condition into a portion of program code executed by SPC 106.

[0172] In general, as described above, system 100 is configured to detect one or more trigger conditions and initiate at least one trace event in response to detecting the condition. In some implementations, initiating the at least one trace event may include system 100 providing, for example, a first control signal to a first performance counter / register (e.g., a CSR such as an activity counter) of FPC 104. The first control signal may cause data associated with the at least one hardware event to be stored in the first performance counter.

[0173] Similarly, in some implementations, initiating the at least one trace event may further include the system 100 providing, for example, a second control signal to a second performance counter / register (e.g., a CSR, such as a statistics counter) of the SPC 106. The second control signal may cause data associated with the at least one hardware event to be stored in the second performance counter.

[0174] In some implementations, the data associated with at least one hardware event may include one of: 1) the number of bytes written to a particular memory buffer of a processor core of a distributed processor unit of system 100, or 2) the number of instructions executed by a particular processor core of a distributed processor unit of system 100.

[0175] In general, the described tracing system includes a hardware-based trigger action mechanism. The triggers may be configured early to act based on software semantics. For example, operands for trigger conditions may be inserted or embedded in the prologues and epilogues of software components (e.g., at functions, loop headers, or other suitable locations in a distributed software system). As mentioned above, in some examples, the insertion of operands for trigger conditions is performed by a compiler in system 100, while in other examples, the triggers and trigger operands are embedded / embedded by a programmer prior to code execution by system 100. It may be inserted or embedded by the host system 126 during real-time code execution.

[0176] For example, in some implementations, a condition indicating when a trigger should be activated may be set up across the distributed hardware system (e.g., at each chip / node of system 100) by performance analysis software running on host system 126. When a trigger is activated, hardware tracing and performance counting is performed at a particular hardware unit, and a similar trigger condition is set up to stop trace data collection.

[0177] In another implementation, the trigger in system 100 may be a monotonically increasing value that is an operand to an instruction. The instruction may be inserted by an exemplary compiler of the processor core at the prologue and epilogue of a critical software construct. Generally, there is a negligible performance penalty in the executing program code for having the trigger operand always present in the instruction, since performance data is not collected until the trigger fires.

[0178] For example, a trigger condition may be set up by writing to an exemplary register in a hardware component. To evaluate or detect a trigger, each time a trigger instruction is encountered, the hardware component may compare the value in this register with the operand value of the trigger condition in the instruction. In general, the trigger condition should be set up well before a certain encoding sequence step so that no processor chip / node in the distributed system arrives at the trigger condition prematurely.

[0179] In some examples, a hardware trigger implementation requires propagating a trigger signal within different components of an exemplary processor chip / node. At least one advantage of the trigger action mechanism is that it allows a distributed software program to execute an activity (e.g., code sequence step 33) at a first node at a time that is very different from a related activity (e.g., code sequence step 33) at another (second) node. In this implementation, the system 100 can maintain an accurate collection of hardware performance data for identical or substantially related activities executed at different time periods by different hardware components or processor nodes.

[0180] In some implementations, an exemplary trigger use case may include system 100 collecting multiple sets of trace data without reprogramming / resetting / reinserting a particular trigger condition. For example, the program code being executed may include a software construct that generates an instruction sequence in which the same operand for setting a trace mark appears multiple times. In some examples, when a processor of system 100 executes a portion of the program code in a loop, the same trace mark setting operand appears multiple times. During execution of the program loop, event tracing may be started and ended according to the iterations of the program loop.

[0181] For example, based on an operand value appearing at the beginning of a loop iteration, a trigger condition may be satisfied, e.g., a trigger is actuated and event tracing is started. Similarly, based on an operand value appearing at the end of a loop iteration, a trigger condition may no longer be satisfied, e.g., event tracing is stopped. In general, the beginning of a loop iteration and the end of a loop iteration may correspond to program loop boundaries for a given software construct. Thus, each execution (e.g., each During each iteration of a loop, a tracing session may be started and ended.

[0182] In some implementations, the appropriate trigger operand value (e.g., GTC or trace mark) may be determined in the following manner: As described above, a distributed software program may include software modules that each include shared or overlapping code sequences or steps. During execution of the distributed program code within system 100, a performance analyst may read or analyze the CSR value of the trace mark register via a computing console of system 100 to determine, for example, the current code sequence or step number.

[0183] In some examples, an analyst repeatedly reads the CSR value to determine the average rate of increase of a sequence or step. The determined rate of increase can be used to predictively set one or more trigger conditions based, for example, on the occurrence of a trace mark operand value. In some implementations, when using a GTC, a performance analyst can read or analyze the system / machine (or hardware component) clock speed to predict or determine the rate at which the GTC increases. The analyst can then use the determined rate to set one or more trigger conditions based on the occurrence of a particular GTC value.

[0184] 7 is a process flow diagram of an exemplary process 700 for distributed hardware tracing using a computing system 100 and one or more nodes 200, 201 of the system 100. Thus, the process 700 may be implemented using one or more of the above-mentioned computing resources of the system 100, including the resources of the nodes 200, 201.

[0185] Process 700 begins at block 702 with computing system 100 monitoring execution of program code executed by a first processor core. In some implementations, the first processor core is configured to execute at least a first portion of the monitored program code. At block 704 of process 700, system 100 monitors execution of program code executed by a second processor core. In some implementations, the second processor core is configured to execute at least a second portion of the monitored program code.

[0186] At block 706, the system 100 detects that at least one trigger condition is satisfied. In some implementations, the system 100 detects that the trigger condition is satisfied based on identifying an occurrence of an operand in at least a portion of program code executed by one of the first processor core or the second processor core. In some implementations, the program code executing within the system 100 includes trigger handling logic for triggering a trace event.

[0187] In some examples, the trigger logic is associated with a “set trace mark” instruction that is inserted into one or more instruction streams. The trigger logic may include identifying an occurrence of a trace mark operand of the instruction, analyzing the trace mark operand, and comparing the operand to an example trace mark window value. In some examples, this analysis and comparison is used to detect that a first trigger condition is satisfied.

[0188] In other examples, the trigger logic is associated with a predefined time value that may be inserted into one or more instruction streams by an external user of the system 100 or by an exemplary compiler of the system. The method may include determining that a current time of one of the clocks indicates a predefined time value used to initiate one or more trace events. In some implementations, determining that the current time indicates a predefined time value may include receiving a predefined time value and comparing the predefined time value to time values ​​of an exemplary predefined time window.

[0189] At block 708, in response to detecting that the trigger condition is satisfied, the system 100 initiates at least a first trace event that generates trace data identifying respective hardware events occurring across the distributed processing unit including at least a first processor core and a second processor core.

[0190] For example, in response to detecting that a trigger condition is satisfied, the system 100 may initiate a synchronized trace event that generates trace data that identifies hardware events occurring across the FPC 104 and the SPC 106. In some implementations, the trace data identifies a unique trace identifier for each hardware event. In some examples, the hardware events include multiple synchronized hardware events, and the events are synchronized if at least two hardware events of the multiple synchronized events share a global hardware event timestamp.

[0191] In some implementations, multiple trigger conditions may be associated with the program code being executed. In some examples, the specific trigger condition may be met based on identifying an occurrence of a specific operand in at least a portion of the program code being executed by at least the first or second processor core. The specific operand may include one of multiple parameter values ​​that may each be used to initiate a second of the one or more trace events. In this implementation, detecting that the specific trigger condition is met may include the system 100 detecting that the specific parameter value exceeds a first limit value of a register.

[0192] In some implementations, in response to detecting that a particular trigger condition is met, the system 100 initiates a trace event that generates trace data that identifies at least one attribute shared between respective hardware events occurring across the distributed processing units. In some implementations, the attribute of each hardware event may include at least a source memory address, a destination memory address, or a sequence / program computation step. Meanwhile, the at least one shared attribute may include a particular sequence step or program step of the program code.

[0193] In some implementations, the one or more trigger conditions may be associated with at least one of: 1) a particular sequence step of program code; 2) a global control parameter indicating a particular performance state of the distributed processing unit; 3) a current time or a particular time parameter indicated by a global time clock / counter of the distributed processing unit; or 4) a predefined time window associated with the global time clock.

[0194] In some implementations, the system 100 may also detect that a particular parameter value of a particular operand or a particular predefined time value exceeds a second limit value of the register. In some examples, in response to this detection, the system 100 may detect that a particular parameter value of an operand or a particular predefined time value exceeds a second limit value of the register. If the threshold is exceeded, the second trace event may be stopped.

[0195] At block 710 of process 700, system 100 provides a correlated set of trace data including the respective hardware events to the host device. In some implementations, the correlated set of trace data indicates at least a chronological order of the respective hardware events that are generated if a first trigger condition is satisfied. For each respective hardware event, the trace data includes at least a hardware event timestamp for the hardware event and metadata that characterizes the hardware event. In some examples, the correlated set of trace data corresponds at least in part to data structure 620.

[0196] At block 712, computing system 100 uses the correlated set of trace data to analyze performance of program code executed by at least the first and second processor cores. In some implementations, data structure 620 (i.e., corresponding to the correlated trace data) is used by host system 126 to analyze performance of program code executed by at least processor cores 104 and 106. Similarly, data structure 620 may be used by host system 126 to analyze performance of at least one component of system 100.

[0197] For example, the host system 126 or an example user may analyze the data structure 620 to detect or determine whether there is a performance problem associated with the execution of a particular software module of program code. An example problem may include the inability of a software module to complete the execution of a certain instruction stream within an allotted execution time window or below a latency threshold.

[0198] Additionally, a user or host device / system 126 may detect or determine whether a particular component of system 100 is operating above or below a threshold performance level. An example problem related to component performance may include a particular hardware component generating result data that is outside of an acceptable threshold result parameter range. In some implementations, the generated result data may not be consistent with result data generated by other associated components of system 100 executing substantially similar instructions / operations.

[0199] For example, during execution of program code, a first component of system 100 may be required to complete an operation and produce a result. Similarly, a second component of system 100 may be required to complete a substantially similar operation and produce a substantially similar result. Analysis of a correlated set of trace data may indicate that the second component produced a result that is significantly different than the result produced by the first component. Similarly, an example data structure of correlated trace entries may indicate result parameter values ​​of the second component that are outside of acceptable parameter ranges. These results may indicate potential performance problems of the second component.

[0200] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of these. Embodiments of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., as tangible, non-transitory program carriers for execution by or to control the operation of a data processing apparatus. The computer storage medium may be realized as one or more modules of computer program instructions encoded on a body. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information to be transmitted to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0201] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data to generate output. The processes and logic flows may also be implemented using special purpose logic circuitry, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and the like. ), or a general purpose graphics processing unit (GPGPU), the apparatus may also be implemented as such special purpose logic circuitry.

[0202] A computer suitable for executing a computer program may include and be based on, for example, a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to receive data from or transfer data to the mass storage devices, or both. However, a computer need not have such devices.

[0203] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0204] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of the invention or the scope of the claims, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a certain combination and may initially be claimed as such, in some cases one or more features from a claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0205] Similarly, although acts are shown in the figures in a particular order, this should be understood as requiring such acts to be performed in the particular order or sequential order shown, or that all of the acts shown be performed, in order to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0206] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order depicted, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method comprising: a step of executing a sequence of program code to perform an operation using components of a processor, each of said components having a time counter indicating a time value, each time counter including a respective offset bit used to compensate for phase changes between said time counters while performing said operation, said sequence of program code including a first time parameter as a trigger condition for triggering a trace event across two or more components of the processor while performing said operation, one of said time counters being selected as a global time counter, said method further comprising: determining a global time value represented by the global time counter; using the respective offset bits to update at least the time values ​​represented by corresponding time counters not selected as the global time counter based on the global time value; determining whether the trigger condition is satisfied by comparing the time value with the first time parameter in the sequence of program code; and in response to determining that the trigger condition is satisfied, triggering the trace event to generate event data for the two or more of the components of the processor, the event data being synchronized based on the global time value, the method further comprising: providing said event data to a host where it is used to analyze performance of said sequence of program code.

2. The step of updating, using each offset bit, at least the time values ​​represented by corresponding time counters not selected as the global time counter based on the global time value, comprising: broadcasting the global time value represented by the global time counter to the corresponding time counters that are not selected as the global time counter; calculating a difference between the time value represented by the corresponding time counter and the global time value; and adjusting said corresponding time counter using said respective offset bits to reduce said difference.

3. A method as described in claim 1 or 2, wherein each time counter includes fixed-size binary data, the fixed-size binary data including a multi-bit counter and the respective offset bits, and each of the offset bits includes 4 bits.

4. The sequence of program codes further includes a second time parameter; the second time parameter indicates a time later than the first time parameter; 4. The method according to claim 1, wherein the trigger condition comprises a predefined time window including a start time for triggering the trace event based on the first time parameter and a stop time for stopping the trace event based on the second time parameter.

5. A method according to any one of claims 1 to 4, wherein each of the components includes one or more performance counters configured to collect count data to analyze the performance of the sequence of program code, the one or more performance counters including at least one of an activity counter, a stall counter, a statistical counter, or a sampling counter.

6. The one or more performance counters configured to collect count data include: collecting said count data including at least one of instruction tracing event data, memory mux tracing event data, or direct memory access event data; 6. A method according to any preceding claim, comprising storing said count data in one or more count registers or control / status registers.

7. The method of claim 1, wherein the subset of event data describes one or more memory access operations performed by the processor; 6. The method of claim 5, wherein the one or more memory access operations include moving a vector element from a first memory of a first component to a second memory of a second component of the processor.

8. An event collection system for collecting event data, comprising: one or more processing devices; and one or more non-transitory machine-readable storage devices for storing instructions executable by the one or more processing devices to perform operations, the operations including: executing a sequence of program code to perform an operation using components of a processor, each of said components having a time counter indicating a time value, each time counter including a respective offset bit used to compensate for phase changes between said time counters while performing said operation, said sequence of program code including a first time parameter as a trigger condition for triggering a trace event across two or more components of the processor while performing said operation, one of said time counters being selected as a global time counter, said operation further comprising: determining a global time value represented by the global time counter; updating, using the respective offset bits, at least the time values ​​represented by corresponding time counters not selected as the global time counter based on the global time value; determining whether the trigger condition is satisfied by comparing the time value to the first time parameter in the sequence of program code; and in response to determining that the trigger condition is satisfied, triggering the trace event to generate event data for the two or more of the components of the processor, the event data being synchronized based on the global time value, the operations further comprising: providing the event data to a host where it is used to analyze performance of the sequence of program code.

9. The method of claim 8, further comprising: updating, using the respective offset bits, at least the time values ​​represented by corresponding time counters that are not selected as the global time counter, based on the global time value, the method comprising: broadcasting the global time value represented by the global time counter to the corresponding time counters that are not selected as the global time counter; calculating a difference between the time value represented by the corresponding time counter and the global time value; and adjusting said corresponding time counter using said respective offset bits to reduce said difference.

10. A system as described in claim 8 or 9, wherein each time counter includes fixed-size binary data, the fixed-size binary data including a multi-bit counter and the respective offset bits, and each offset bit includes 4 bits.

11. The sequence of program codes further comprising a second time parameter; the second time parameter indicates a time later than the first time parameter; 11. The system of claim 8, wherein the trigger condition comprises a predefined time window including a start time for triggering the trace event based on the first time parameter and a stop time for stopping the trace event based on the second time parameter.

12. The system of any one of claims 8 to 11, wherein each of the components includes one or more performance counters configured to collect count data to analyze the performance of the sequence of program code, the one or more performance counters including at least one of an activity counter, a stall counter, a statistical counter, or a sampling counter.

13. The one or more performance counters configured to collect count data, collecting said count data including at least one of instruction tracing event data, memory mux tracing event data, or direct memory access event data; Storing the count data in one or more count registers or control / status registers.

14. The subset of event data describing one or more memory access operations performed by the processor; 13. The system of claim 12, wherein the one or more memory access operations include moving a vector element from a first memory of a first component to a second memory of a second component of the processor.

15. A program storing instructions executable by one or more processing devices to perform operations for collecting event data, the operations comprising: executing a sequence of program code to perform an operation using components of a processor, each of said components having a time counter indicating a time value, each time counter including a respective offset bit used to compensate for phase changes between said time counters while performing said operation, said sequence of program code including a first time parameter as a trigger condition for triggering a trace event across two or more components of the processor while performing said operation, one of said time counters being selected as a global time counter, said operation further comprising: determining a global time value represented by the global time counter; updating, using the respective offset bits, at least the time values ​​represented by corresponding time counters not selected as the global time counter based on the global time value; determining whether the trigger condition is satisfied by comparing the time value to the first time parameter in the sequence of program code; and in response to determining that the trigger condition is satisfied, triggering the trace event to generate event data for the two or more of the components of the processor, the event data being synchronized based on the global time value, the operations further comprising: providing the event data to a host for use in analyzing performance of the sequence of program code.

16. The method of claim 15, further comprising: updating, using the respective offset bits, at least the time values ​​represented by corresponding time counters that are not selected as the global time counter, based on the global time value, the method comprising: broadcasting the global time value represented by the global time counter to the corresponding time counters that are not selected as the global time counter; calculating a difference between the time value represented by the corresponding time counter and the global time value; and adjusting the corresponding time counter using the respective offset bit to reduce the difference.

17. A program as described in claim 15 or 16, wherein each time counter includes fixed-size binary data, the fixed-size binary data including a multi-bit counter and each of the offset bits, and each of the offset bits includes 4 bits.

18. The sequence of program codes further comprising a second time parameter; the second time parameter indicates a time later than the first time parameter; 18. The program of claim 15, wherein the trigger condition comprises a predefined time window including a start time for triggering the trace event based on the first time parameter and a stop time for stopping the trace event based on the second time parameter.

19. The program of any one of claims 15 to 18, wherein each of the components includes one or more performance counters configured to collect count data to analyze the performance of the sequence of program code, the one or more performance counters including at least one of an activity counter, a stall counter, a statistical counter, or a sampling counter.

20. The one or more performance counters configured to collect count data, collecting said count data including at least one of instruction tracing event data, memory mux tracing event data, or direct memory access event data; and storing the count data in one or more count registers or control / status registers.

Images