Synchronous hardware event collection

Abstract

The present disclosure relates to synchronous hardware event collection. A computer-implemented method includes monitoring execution of program code by first and second processor components. A computing system detects that a trigger condition is satisfied by: i) identifying an operand in a portion of the program code; or ii) determining that a current time of a clock of the computing system indicates a predefined time value. The operand and the predefined time value are used to initiate a trace event. When the trigger condition is satisfied, the system initiates the trace event, which generates trace data that identifies a respective hardware event that occurred across the computing system. The system uses the trace data to generate a correlated trace data set. The correlated trace data indicates a time-ordered sequence of the respective hardware events. The system uses the correlated trace data set to analyze performance of the executing program code.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201711292833.X, filed on December 8, 2017. Technical Field

[0003] This manual relates to the execution of analysis program code. Background Technology

[0004] Effective performance analysis of distributed software executing within distributed hardware components can be a complex task. Distributed hardware components can be corresponding processor cores of two or more central processing units (CPUs) (or graphics processing units (GPUs)) that cooperate and interact to execute large software programs or portions of program code.

[0005] From a hardware perspective (e.g., within a CPU or GPU), there are typically two types of information or features that can be applied to performance analysis: 1) hardware performance counters; and 2) hardware event traces. Summary of the Invention

[0006] Generally, an innovative aspect of the subject matter described in this specification can be incorporated into a computer-implemented method executed by one or more processors, the method comprising: monitoring the execution of program code performed by a first processor component, the first processor component being configured to execute at least a first portion of the program code; and monitoring the execution of the program code performed by a second processor component, the second processor component being configured to execute at least a second portion of the program code.

[0007] The method further includes: the computing system detecting that a triggering condition is met based on at least one of the following: i) identifying an operand appearing in at least a first part or a second part of the program code, the operand including a first parameter value for initiating one or more trace events, or ii) determining that the current time of at least one clock of the computing system indicates a predefined time value for initiating one or more trace events.

[0008] In response to detecting that a trigger condition is met, the method includes: initiating at least a first tracking event by a computing system, which generates tracking data identifying corresponding hardware events occurring across distributed processor units, said distributed processor units including at least a first processor component and a second processor component. For each of the corresponding hardware events, the tracking data includes at least one hardware event timestamp. The method further includes: using the tracking data by the computing system to generate an associated tracking dataset indicating at least a time-ordered sequence of the corresponding hardware events generated when the trigger condition is met.

[0009] These and other implementations may optionally include one or more of the following features. For example, in some implementations, the first trace event is a synchronized trace event that generates trace data that identifies hardware events occurring across the distributed processor unit, the trace data also identifying a unique trace identifier for the corresponding hardware event, and wherein the hardware events include a plurality of synchronized hardware events, and the two hardware events are synchronized when the two hardware events share a global hardware event timestamp.

[0010] In some implementations, detecting that a trigger condition is met includes: the computing system detecting one of the following: i) a first parameter value of the operand exceeds a first limit value of the register, or ii) a predefined time value indicated by the current time exceeds the first limit value of the register; and in response to detecting that the trigger condition is met, the computing system initiating a second trace event to generate trace data, wherein the trace data identifies at least one attribute shared among corresponding hardware events occurring across the distributed processor units.

[0011] In some embodiments, the method further includes: the computing system detecting one of the following: the second parameter value of the operand exceeds a second limit value of the register, or a second predefined time value indicated by the current time exceeds the second limit value of the register; and in response to the detection, the computing system stopping the second tracking event when the second parameter value of the operand exceeds the second limit value, or when the second predefined time value exceeds the second limit value.

[0012] In some implementations, the operands further include at least one of the following: a parameter indicating a specific sequence of steps in the program code; or a global control parameter indicating a specific performance state of the distributed processor unit; and the predefined time value includes at least one of the following: a specific time value indicated by the global clock of the distributed processor unit; or a specific time value of a predefined time window associated with the global clock.

[0013] In some implementations, the operand has a first binary data structure, and a first parameter value of the first operand corresponds to a tracking tag, a predefined time value has a second binary data structure, and the current time is indicated by a global clock, wherein the global clock is used by the distributed processor unit to generate one or more hardware event timestamps.

[0014] In some implementations, the method further includes: having the compiler of the computing system insert an operand of the trigger condition into at least a first portion of the program code executed by the first processor component; and having the compiler of the computing system insert at least one predefined time value of the trigger condition into at least a second portion of the program code executed by the second processor component.

[0015] In some implementations, initiating at least one of the first or second tracking events includes: generating a first control signal by a computing system, which is received by a first counter register of a first processor core, the first control signal causing data associated with the first hardware event to be stored in the first counter register; and generating a second control signal by a computing system, which is received by a second counter register of a second processor core, the second control signal causing data associated with the second hardware event to be stored in the second counter register.

[0016] In some implementations, the data associated with one of the first hardware event or the second hardware event includes at least one of the following: the number of bytes written to a specific memory buffer of a specific processor core in the distributed processor unit; or the number of instructions executed by a specific processor core in the distributed processor unit.

[0017] In some embodiments, the method further includes: the computing system identifying the presence of a second operand in at least a portion of program code executed by a first or second processor component, the second operand including a second parameter value; the computing system determining that a filtering condition is met based on the second parameter value of the second operand exceeding or falling below a specific threshold value of a register; and in response to determining that the filtering condition is met, the computing system filtering one or more trace events, wherein filtering one or more trace events includes excluding the storage of trace data associated with one or more hardware events.

[0018] Another innovative aspect of the subject matter described in this specification can be embodied in a hardware event collection system comprising: one or more processors, including one or more processor cores; one or more machine-readable storage units for storing instructions executable by the one or more processors to perform operations, including: monitoring the execution of program code performed by a first processor component, the first processor component being configured to execute at least a first portion of the program code; and monitoring the execution of the program code performed by a second processor component, the second processor component being configured to execute at least a second portion of the program code.

[0019] The method further includes: the computing system detecting that a triggering condition is met based on at least one of the following: i) identifying an operand appearing in at least a first part or a second part of the program code, the operand including a first parameter value for initiating one or more trace events, or ii) determining that the current time of at least one clock of the computing system indicates a predefined time value for initiating one or more trace events.

[0020] In response to detecting that a trigger condition is met, the method includes: initiating at least a first tracking event by a computing system, which generates tracking data identifying corresponding hardware events occurring across distributed processor units, said distributed processor units including at least a first processor component and a second processor component. For each hardware event in the corresponding hardware events, the tracking data includes at least one hardware event timestamp. The method further includes: using the tracking data by the computing system to generate an associated tracking dataset indicating at least a time-ordered sequence of corresponding hardware events generated when the trigger condition is met.

[0021] Generally, an innovative aspect of the subject matter described in this specification can be included in a computer-implemented method executed by one or more processors, the method comprising: monitoring the execution of program code carried out by a processor component configured to execute at least a first portion of the program code.

[0022] The method further includes: the computing system satisfying a triggering condition based on at least one of the following detections: i) identifying an operand in at least a first part of the program code, the operand including a first parameter value for initiating one or more trace events, or ii) determining that the current time of at least one clock of the computing system indicates a predefined time value for initiating the one or more trace events.

[0023] In response to the detection that a trigger condition is met, the method further includes: generating a control signal received by a counter register of a processor component by a computing system, the control signal causing counter data associated with a hardware event to be stored in the counter register; and generating a data structure by the computing system indicating one or more performance attributes associated with executable program code, the data structure being generated based on one or more counter parameters of the stored counter data.

[0024] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, the count register is one of a plurality of performance counters configured to store count data about the performance of one or more processor components of a computing system, and wherein at least one performance counter includes one of the following: an activity counter, a pause counter, a statistics counter, or a sampling counter.

[0025] In some implementations, one or more counting parameters indicate one of the following: i) the number of instructions received by a particular processor component; ii) the number of instructions processed by a particular processor component; iii) the number of instructions executed by a particular processor component; or iv) the number of memory reads or memory writes performed by a particular processor component.

[0026] In some implementations, the performance attributes associated with executing program code include one of the following: i) the pause frequency of a particular processor component executing the program code; ii) an indication that the utilization of the particular processor component is below a threshold utilization; or iii) an indication that the data storage queue used by the particular processor component is at or below a queue occupancy threshold.

[0027] Other embodiments of this and other aspects include corresponding systems, apparatuses, and computer programs configured to perform actions of methods encoded on computer storage devices. A system of one or more computers can be configured, in operation, to cause the system to perform actions by means of software, firmware, hardware, or a combination thereof installed on the system. One or more computer programs can be configured, by means of instructions, to cause an apparatus to perform actions when executed by a data processing device.

[0028] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages. The described hardware tracking system improves computational efficiency by using dynamic triggering conditions executed via hardware knobs / features. These triggering conditions can synchronously capture events in part based on a global time counter, embedded logic operands, and hardware registers, rather than purely through encoded sequences, which typically require system resources to monitor and execute event capture.

[0029] Similarly, system memory usage is optimized when hardware triggering is tuned to capture short-lived events and simultaneous synchronous global captures, rather than irrelevant and inefficient event captures. Control over short-lived synchronous event captures alleviates information overload, thereby keeping some memory resources unused and available for other system processes.

[0030] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0031] Figure 1 The figure shows a block diagram of an example computer system used for distributed hardware tracing.

[0032] Figure 2 The diagram illustrates a block diagram of the tracing chain and corresponding nodes of an example computing system used for distributed hardware tracing.

[0033] Figure 3 The diagram illustrates the design architecture and data structure of an example trace multiplexer.

[0034] Figure 4It is a block diagram indicating the tracing activities performed by the example computing system used for distributed hardware tracing of direct memory access tracing events.

[0035] Figure 5 The diagram illustrates an example data structure for the Global Time Counter (GTC) and a table indicating the use cases of the GTC by an example computing system used for distributed hardware tracing.

[0036] Figure 6 The diagram illustrates an example time counter and a block diagram of the hardware features associated with components of an example computing system used for distributed hardware tracing.

[0037] Figure 7 This is a flowchart of an example process for distributed hardware tracing.

[0038] Similar icon numbers and markings in the figures indicate similar elements. Detailed Implementation

[0039] The subject matter described in this specification generally relates to distributed hardware tracing. Specifically, the computing system monitors the execution of program code performed by a first processor core and the execution of program code performed by a second processor core. The computing system stores a timeline of hardware events in a memory buffer. The stored events span distributed processor units, including at least the first and second processor cores.

[0040] For each hardware event, the timeline includes an event timestamp and metadata characterizing the hardware event. The system generates a data structure that includes the hardware events from the timeline. The system stores this data structure in a repository on the host device and uses this data structure to evaluate the performance of program code executed by the first or second processor core. In this event tracing context, this specification describes methods for hardware event tracing in distributed computing systems, such as... Figures 1 to 4 As shown.

[0041] This specification further describes methods for collecting synchronous hardware events based on one or more triggering mechanisms, such as... Figures 5 to 7 As shown below, several aspects of the computing system described in this specification relate at least to coordinating / synchronizing the collection of trace data and event counts. In particular, at least one aspect includes a system and method for synchronously collecting hardware performance counter data and trace event data within independent and distributed systems. Synchronized event collection enhances the analysis of performance data and debug data from distributed program code. This enhanced analysis is achieved in part through the correlation of events occurring in response to the execution of connected software components / modules analyzed by system 100.

[0042] Figure 1The figure illustrates a block diagram of an example computer system 100 for distributed hardware tracing. As used herein, distributed hardware system tracing corresponds to events occurring within components and sub-components of an example processor microchip. Additionally, as used herein, a distributed hardware system (or tracing system) corresponds to a collection of processor microchips that cooperate in executing corresponding portions of software / program code configured for distributed execution among the collection of processor microchips. In some embodiments, different processor chips of system 100 can form corresponding nodes in the distributed hardware system. In alternative embodiments, a single processor chip can include one or more processor cores and hardware features that can each form a corresponding node in the processor chip.

[0043] For example, in a central processing unit (CPU) context, the processor chip may include at least two nodes, and each node may be a corresponding core of the CPU. Alternatively, in a graphics processing unit (GPU) context, the processor chip may include at least two nodes, and each node may be a corresponding streaming multiprocessor of the GPU. The computing system 100 may include multiple processor components. In some embodiments, a processor component may be at least one of a processor chip, a processor core, a memory access engine, or at least one hardware component of the overall computing system 100.

[0044] In some instances, a processor component, such as a processor core, can be a fixed-function component configured to perform at least one specific operation based on at least one instruction issued from the execution program code. In other instances, a processor component, such as a memory access engine (MAE), can be configured to execute program code at a lower level of detail or granularity than program code executed by other processor components of system 100.

[0045] For example, program code executed by the processor core can cause the generation of an MAE descriptor and its transmission / sending to the MAE. Upon receiving the descriptor, the MAE can perform data transfer operations based on it. In some implementations, data transfers performed by the MAE can, for example, involve moving data to and from certain components of system 100 via some data path or interface component of the system, or issuing a data request to an example configuration bus of system 100.

[0046] In some implementations, each tensor node of the exemplary processor chip of system 100 can have at least two "front ends," which can be hardware blocks / features that process program instructions. As detailed below, a first front end can correspond to a first processor core 104, and a second front end can correspond to a second processor core 106. Therefore, the first and second processor cores can also be described herein as a first front end 104 and a second front end 106.

[0047] As used in this specification, the trace chain can be a specific physical data communication bus on which trace entries are placed for transmission to the example chip manager within system 100. The received trace entries can be data words / structures comprising multiple bytes and multiple binary values ​​or bits. Therefore, the descriptor "word" indicates a fixed-size piece of binary data that can be processed as a unit by the hardware devices of the example processor core.

[0048] In some implementations, the processor chip of the distributed hardware tracing system is a multi-core processor (i.e., having multiple cores), each executing a portion of program code within its respective core. In some implementations, the portion of program code can correspond to vectorized computations for an inference workload exemplifying a multilayer neural network. In alternative implementations, the portion of program code can generally correspond to software modules associated with a conventional programming language.

[0049] The computing system 100 generally includes a node manager 102, a first processor core (FPC) 104, a second processor core (SPC) 106, a node organization structure (NF) 110, a data router 112, and a host interface block (HIB) 114. In some embodiments, the system 100 may include a memory multiplexer 108 configured to perform signal switching, multiplexing, and demultiplexing functions. The system 100 further includes a tensor core 116, which includes the FPC 104 disposed therein. The tensor core 116 may be an example computing device configured to perform vectorized computations on a multidimensional data array. The tensor core 116 may include a vector processing unit (VPU) 118 that interacts with a matrix unit (MXU) 120, a transpose unit (XU) 122, and a reduction and permutation unit (RPU) 124. In some embodiments, the computing system 100 may include one or more execution units of a conventional CPU or GPU, such as a load / store unit, an arithmetic logic unit (ALU), and a vector unit.

[0050] The components of System 100 generally include a large set of hardware performance counters and support hardware that facilitates tracing activities within the components. As described in more detail below, program code executed by the corresponding processor core of System 100 can include embedded trigger conditions for simultaneously enabling multiple performance counters during code execution. Generally, detected trigger conditions prompt the generation of trace data for one or more tracing events. The trace data can correspond to incremental parameter counts stored in the counters and capable of being analyzed to identify performance characteristics of the program code. Data for corresponding tracing events can be stored in a sample storage medium (e.g., a hardware buffer) and can include timestamps generated in response to the detection of trigger conditions.

[0051] Additionally, tracking data can be generated for a variety of events occurring within the hardware components of system 100. Example events can include inter-node and cross-node communication operations, such as direct memory access (DMA) operations and synchronization flag updates (both described in detail below). In some embodiments, system 100 can include a global synchronization timestamp counter, commonly referred to as a global time counter (“GTC”). In other embodiments, system 100 can include other types of global clocks, such as a Lamport clock.

[0052] GTC can be used to precisely correlate program code execution with the performance of software / program code executed in a distributed processing environment. Furthermore, in part related to GTC, in some implementations, system 100 can include one or more triggering mechanisms used by distributed software programs to start and stop data tracing in a highly coordinated manner within the distributed system.

[0053] In some implementations, host system 126 compiles program code that can include embedded operands that, upon detection, trigger the capture and storage of trace data associated with a hardware event. In some implementations, host system 126 provides the compiled program code to one or more processor chips of system 100. In alternative implementations, the program code can be compiled (with embedded triggering) using an example external compiler and loaded into one or more processor chips of system 100. In some instances, the compiler can set one or more trace bits (described below) associated with certain trigger conditions in portions of the embedded software instructions. The compiled program code can be a distributed software program executed by one or more components of system 100.

[0054] Host system 126 may include a monitoring engine 128 configured to monitor the execution of program code by one or more components of system 100. In some embodiments, monitoring engine 128 enables host system 126 to monitor the execution of program code by at least FPC 104 and SPC 106. For example, during code execution, host system 126 may monitor the performance of the executed code via monitoring engine 128, at least by receiving a periodic timeline of hardware events based on generated trace data. Although a single block is shown for host system 126, in some embodiments, system 126 may include multiple hosts (or host subsystems) associated with multiple processor chips or chip cores of system 100.

[0055] In other implementations, cross-node communication involving at least three processor cores can cause host system 126 to monitor one or more intermediate "hops" of data traffic as data traffic traverses the communication path between FPC 104 and the example third processor core / node. For example, FPC 104 and the third processor core may be the only cores executing program code for a given time period. Therefore, when data is transferred from FPC 104 to the third processor core, the data transfer from FPC 104 to the third processor core can generate trace data for intermediate hops at SPC 106. In other words, during data routing in system 100, data from the first processor chip to the third processor chip may need to traverse the second processor chip, so the execution of data routing operations may result in the generation of trace entries for routing activity in the second chip.

[0056] After the compiled program code is executed, the components of System 100 can interact to generate a timeline of hardware events occurring in the distributed computer system. Hardware events can include intra-node and cross-node communication events. See below for reference. Figure 2 The example nodes of the distributed hardware system and their associated communications are described in more detail. In some implementations, a data structure is generated that identifies a set of hardware events for at least one hardware event timeline. The timeline is capable of reconstructing events that occurred in the distributed system. In some implementations, event reconstruction can include correcting event ordering based on analysis of timestamps generated during the occurrence of a particular event.

[0057] Generally, an example distributed hardware tracing system can include the components of system 100 described above, as well as at least one host controller associated with host system 126. The performance or debugging of data obtained from a distributed tracing system can be beneficial when event data is correlated, for example, in a chronological or ordered manner. In some implementations, data correlation can occur when multiple stored hardware events corresponding to connected software modules are stored and then ordered for structured analysis of host system 126. For implementations including multiple host systems, the correlation of data obtained via different hosts can be performed, for example, by the host controller.

[0058] In some embodiments, FPC 104 and SPC 106 are both different cores of a single multi-core processor chip; in other embodiments, FPC 104 and SPC 106 are cores of different multi-core processor chips. As indicated above, system 100 may include a distributed processor unit having at least FPC 104 and SPC 106. In some embodiments, the distributed processor unit of system 100 may include one or more hardware or software components configured to execute at least a portion of a larger distributed software program or program code.

[0059] Data router 112 is an inter-chip interconnect (ICI) that provides data communication paths between components of system 100. Specifically, router 112 provides communication coupling or connection between FPC 104 and SPC 106, and between corresponding components associated with cores 104 and 106. Node organization 110 interacts with data router 112 to move data packets within the distributed hardware components and sub-components of system 100.

[0060] Node manager 102 is a high-level device that manages the low-level node functions in a multi-node processor chip. As detailed below, one or more nodes of the processor chip can include a chip manager controlled by node manager 102 to manage and store hardware event data in a local entry log. Memory multiplexer 108 is a multiplexing device capable of performing switching, multiplexing, and demultiplexing operations on data signals provided to or received from an external high-bandwidth memory (HBM).

[0061] In some implementations, when multiplexer 108 switches between FPC 104 and SPC 106, example trace entries (described below) can be generated by multiplexer 108. Memory multiplexer 108 can potentially affect the performance of specific processor cores 104, 106 that cannot access multiplexer 108. Therefore, trace entry data generated by multiplexer 108 can help understand latency spikes in certain system activities associated with the corresponding cores 104, 106. In some implementations, hardware event data initiated within multiplexer 108 (e.g., trace points described below) can be grouped together with event data from node organization structure 110 on the example hardware event timeline. Event grouping can occur when certain tracing activities cause event data from multiple hardware components to be stored in an example hardware buffer (e.g., trace entry log 218 described below).

[0062] In system 100, the performance analysis hardware includes FPC 104, SPC 106, multiplexer 108, node organization structure 110, data router 112, and HIB 114. Each of these hardware components or units includes hardware performance counters and hardware event tracking conditions and functions. In some embodiments, VPU 118, MXU 120, XU 122, and RPU 124 do not include their own dedicated performance hardware. Instead, in this embodiment, FPC 104 can be configured to provide the necessary counters for VPU 118, MXU 120, XU 122, and RPU 124.

[0063] The VPU 118 can include an internal design architecture that supports local high-bandwidth data processing and arithmetic operations associated with vector elements of the example matrix vector processor. The MXU 120 is a matrix multiplication unit configured to perform up to 128×128 matrix multiplications, for example, on a vector dataset of multiplicands.

[0064] XU 122 is a transpose unit configured to perform up to 128×128 matrix transpose operations on vector data associated with matrix multiplication, for example. RPU 124 can include sigma units and permutation units. The sigma unit performs sequential reduction on the vector data associated with matrix multiplication. Reduction can include summation and various types of comparison operations. The permutation unit can completely permutate or copy all elements of the vector data associated with matrix multiplication.

[0065] In some implementations, program code executed by components of system 100 can represent machine learning, neural network inference computation, and / or one or more direct memory access functions. Components of system 100 can be configured to execute one or more software programs, including instructions that cause the system's processing units or devices to perform one or more functions. The term "component" is intended to include any data processing or storage device, such as a control status register, or any other device capable of processing and storing data.

[0066] System 100 may generally include multiple processing units or devices, which may include one or more processors (e.g., microprocessors or central processing units (CPUs)), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or combinations of different processors. In alternative embodiments, system 100 may each include additional computing resources / devices (e.g., cloud-based servers) that provide additional processing options for performing calculations related to the hardware tracing functions described herein.

[0067] The processing unit or device may further include one or more memory cells or repositories (e.g., registers / counters). In some embodiments, the processing unit executes programmed instructions stored in memory on the device of system 100 for one or more functions described herein. The memory cells / repositories may include one or more non-transitory machine-readable storage media. Non-transitory machine-readable storage media may include solid-state memory, magnetic disks and optical disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information.

[0068] Figure 2The diagram illustrates an example tracing chain and corresponding example nodes 200, 201 for distributed hardware tracing performed by system 100. In some embodiments, nodes 200, 201 of system 100 can be different nodes within a single multi-core processor. In other embodiments, node 200 can be a first node in a first multi-core processor chip, and node 201 can be a second node in a second multi-core processor chip.

[0069] Although Figure 2 The implementation depicts two nodes, but in an alternative implementation, system 100 can include multiple nodes. For implementations involving multiple nodes, cross-node data transfers can generate trace data at intermediate hops along an example data path traversing multiple nodes. For example, an intermediate hop can correspond to data transfers through different nodes in a specific data transfer path. In some instances, trace data associated with ICI tracing / hardware events can be generated for one or more intermediate hops occurring during cross-node data transfers through one or more nodes.

[0070] In some implementations, node 0 and node 1 are tensor nodes used for vectorized computations associated with portions of the program code used for inference workloads. As used in this specification, a tensor is a multidimensional geometric object, and example multidimensional geometric objects include matrices and data arrays.

[0071] like Figure 2 As shown in the implementation, node 200 includes a tracing chain 203 that interacts with at least a subset of the components of system 100. Similarly, node 201 includes a tracing chain 205 that interacts with at least a subset of the components of system 100. In some implementations, nodes 200 and 201 are example nodes of the same subset of components, while in other implementations, nodes 200 and 201 are corresponding nodes of different subsets of components. The data router / ICI 112 includes a tracing chain 207, which typically converges with tracing chains 203 and 205 to provide tracing data to the chip manager 216.

[0072] exist Figure 2In some implementations, nodes 200 and 201 can each include at least a corresponding subset of components having FPC 104, SPC 106, node organization structure 110, and HIB 114. Each component of nodes 200 and 201 includes one or more tracking multiplexers configured to group tracking points (described below) generated by a particular component of the node. FPC 104 includes tracking multiplexer 204, node organization structure 110 includes tracking multiplexers 210a / b, SPC 106 includes tracking multiplexers 206a / b / c / d, HIB 214 includes tracking multiplexer 214, and ICI 212 includes tracking multiplexer 212. In some implementations, the tracking control register for each tracking multiplexer allows enabling and disabling individual tracking points. In some instances, for one or more tracking multiplexers, their corresponding tracking control registers can include individual enable bits and broader tracking multiplexer control.

[0073] Generally, the trace control register can be a regular control status register (CSR) that receives and stores trace instruction data. Regarding broader trace multiplexer control, in some implementations, trace can be enabled and disabled based on CSR writes performed by system 100. In some implementations, trace is dynamically started and stopped by system 100 based on the value of the global time counter (GTC), the value of the example trace flag register in FPC 104 (or core 116), or the value of the example trace flag register in SPC 106.

[0074] Reference Figures 5 to 7 The implementation methods are described in more detail, with additional details and descriptions relating to the systems and methods for dynamically starting and stopping tracking activities and synchronizing hardware event collection.

[0075] In some implementations, for core 116, FPC 104 can use trace control parameters to define a trace window associated with event activity occurring within core 116. The trace control parameters allow the trace window to be defined based on the lower and upper bounds of the GTC and the lower and upper bounds of the trace flag register.

[0076] In some implementations, system 100 may include features that can reduce the number of generated trace entries, such as trace event filtering features. For example, FPC 104 and SPC 106 may each include filtering features that limit the rate at which each core sets trace bits in an example-generated trace descriptor (described below). HIB 114 may include similar filtering features, such as an example DMA rate limiter that limits the trace bits associated with capturing certain DMA trace events. Furthermore, HIB 114 may include controls (e.g., via enable bits) for limiting which queues initiate DMA trace entries.

[0077] In some implementations, the descriptor used for DMA operations can have trace bits set by a sample compiler of the host system 126. When the trace bits are set, a sample trace event is performed using hardware features / knobs that determine and generate trace data. In some instances, the final trace bits in DMA can be a logical OR operation between trace bits statically inserted by the compiler and trace bits dynamically determined by specific hardware components. Therefore, in some instances, in addition to filtering, the trace bits generated by the compiler can also provide a mechanism to reduce the total amount of trace data generated.

[0078] For example, the compiler of host system 126 can decide to set the trace bits for only one or more remote DMA operations (e.g., DMA across at least two nodes) and clear the trace bits for one or more local DMA operations (e.g., DMA within a specific tensor node such as node 200). In this way, by limiting the trace activity to cross-node (i.e., remote) DMA operations, rather than trace activities that include both cross-node and local DMA operations, the amount of trace data generated can be reduced.

[0079] In some implementations, at least one trace event initiated by system 100 can be associated with a memory access operation that includes multiple intermediate operations occurring across system 100. The descriptor for the memory access operation (e.g., a MAE descriptor) can include trace bits that cause data associated with the multiple intermediate operations to be stored in one or more memory buffers. Therefore, as packets traverse system 100, the trace bits can be used to "mark" intermediate memory operations and generate multiple trace events in the middle of DMA operations.

[0080] In some implementations, ICI 112 may include a set of enable bits and a set of packet filters that provide control over each ingress and egress port of a specific component of nodes 200, 201. These enable bits and packet filters allow ICI 112 to enable and disable tracking points associated with specific components of nodes 200, 201. In addition to enabling and disabling tracking points, ICI 112 can be configured to filter tracking data based on event source, event destination, and tracking event packet type.

[0081] In some implementations, in addition to using GTC or trace markers, each trace control register of processor cores 104, 106, and HIB 114 can also include a "per person" trace mode. This "per person" trace mode allows trace across the entire processor chip to be controlled by trace multiplexer 204 or trace multiplexer 206a. In per person trace mode, trace multiplexers 204 and 206a can send "in-window" trace control signals specifying whether a particular trace multiplexer (multiplexer 204 or multiplexer 206a) is within a trace window.

[0082] The in-window tracking control signal can be broadcast or universally sent to all other tracking multiplexers, for example, within the processor chip or across multiple processor chips. Broadcasting to other tracking multiplexers while multiplexer 204 or multiplexer 206a is performing tracking activity can prompt the activation of all tracking. In some embodiments, the tracking multiplexers associated with processor cores 104, 106, and HIB 114 each include a tracking window control register that specifies when and / or how the "everyone tracks" control signal is generated.

[0083] In some implementations, tracking activity in tracking multiplexers 210a / b and tracking multiplexer 212 is generally enabled based on whether the tracking bit is set in the data word used for DMA operations or in the control message traversing ICI / data router 112. The DMA operation or control message can be a fixed-size binary data structure that may have tracking bits set within the binary data packet based on certain conditions or software requirements.

[0084] For example, when a DMA operation is initiated in an FPC 104 (or SPC 106) with a trace-type DMA instruction and the initiator (processor core 104 or 106) is within the trace window, the trace bit will be set in that particular DMA. In another example, for FPC 104, if FPC 104 is within the trace window and a trace point that causes trace data to be stored is enabled, a control message for writing data to another component within system 100 will complete the trace bit setting.

[0085] In some implementations, zero-length DMA operations provide examples of a broader range of DMA implementations within system 100. For example, some DMA operations are capable of generating non-DMA activities within system 100. It is also possible to track the execution of non-DMA activities (e.g., generating trace data) as if the non-DMA activity were a DMA operation (e.g., a DMA activity that includes a non-zero-length operation). For example, a DMA operation initiated at a source location but without any data to be sent or transmitted (e.g., zero-length) can instead send a control message to the destination location. The control message will indicate that there is no data to be received or processed at the destination, and the control message itself will be tracked by system 100 because the non-zero-length DMA operation will be tracked.

[0086] In some instances, for SPC 106, a zero-length DMA operation can generate a control message, and the track bit associated with the message can only be set after the DMA has completed track bit setting, i.e., when the control message does not have zero length. Generally, if HIB 114 is within the track window, a DMA operation initiated from host system 126 will complete the track bit setting.

[0087] exist Figure 2 In this implementation, tracing chain 203 receives tracing entry data for a subset of components aligned to node 0, while tracing chain 205 receives tracing entry data for a subset of components aligned to node 1. Each tracing chain 203, 205, and 207 is a different data communication path used by the corresponding nodes 200, 201, and ICI 112 to provide tracing entry data to the example tracing entry data log 218 of the chip manager 216. Therefore, the endpoints of tracing chains 203, 205, and 207 are the chip manager 216, where tracing events can be stored in the example memory unit.

[0088] In some implementations, at least one memory cell of the chip manager 216 can be 128 bits wide and have a memory depth of at least 20,000 tracking entries. In alternative implementations, at least one memory cell can have a larger or smaller bit width and a memory depth capable of storing more or fewer entries.

[0089] In some implementations, chip manager 216 may include at least one processing device that executes instructions to manage received trace entry data. For example, chip manager 216 may execute instructions to scan / analyze timestamp data of corresponding hardware events in trace data received via trace chains 203, 205, 207. Based on the analysis, chip manager 216 may populate trace entry log 218 to include time-ordered sequence data that can be used to identify (or generate) hardware trace events. When the processing unit of system 100 executes an example distributed software program, hardware trace events may correspond to data packet movements occurring at the component and subcomponent levels.

[0090] In some implementations, the hardware units of system 100 can generate trace entries (and corresponding timestamps) that fill the example hardware trace buffer in an unordered manner (i.e., out of order). For example, chip manager 216 can cause multiple trace entries with generated timestamps to be inserted into entry log 218. The corresponding trace entries of the multiple inserted trace entries may not be ordered chronologically relative to each other. In this implementation, the unordered trace entries can be received by the example host buffer of host system 126. After being received by the host buffer, host system 126 can execute instructions related to performance analysis / monitoring software to scan / analyze the timestamp data of the corresponding trace entries. The executed instructions can be used to sort the trace entries and construct / generate a timeline of hardware trace events.

[0091] In some implementations, tracking entries can be deleted from the entry log 218 during a tracking session via host DMA operations. In some instances, the host system 126 may not delete a tracking entry from the tracking entry log 218 as quickly as adding it to the log. In other implementations, the entry log 218 may include a predefined memory depth. If the memory depth limit of the entry log 218 is reached, added tracking entries may be lost. To control which tracking entries are discarded, the entry log 218 can operate in a first-in-first-out (FIFO) mode or alternatively in an overwrite mode.

[0092] In some implementations, the overwrite logging mode can be used by system 100 to support performance analysis associated with post-processing debugging. For example, program code can be executed during specific time periods when tracing activity is enabled and overwrite logging mode is enabled. In response to a post-processing software event (e.g., a program crash) within system 100, monitoring software executed by host system 126 can analyze the data contents of an example hardware trace buffer to gain insight into hardware events that occurred prior to the program crash. As used in this specification, post-processing debugging involves the analysis or debugging of program code that has crashed or is generally not executing / operating as expected.

[0093] In FIFO mode, if the entry log 218 is full, and if the host system 126 determines to delete log entries saved within a certain time frame to save memory resources, new trace entries may not be saved to the memory cells of the chip manager 216. In overwrite mode, if the entry log 218 is full, because the host system 126 determines to delete log entries already saved within a certain time frame to save memory resources, new trace entries can overwrite the oldest trace entry stored in the entry log 218. In some embodiments, trace entries are moved to the host system 126's memory in response to a DMA operation using the processing features of HIB 114.

[0094] As used in this specification, a trace point is a generator of trace entries and associated data received by chip manager 216 and stored in trace entry log 218. In some embodiments, a multi-core, multi-node processor microchip can include three trace chains within the chip, such that a first trace chain receives trace entries from chip node 0, a second trace chain receives trace entries from chip node 1, and a third trace chain receives trace entries from the chip's ICI router.

[0095] Each tracking point has a unique tracking identifier number within its tracking chain, which is inserted into the header of the tracking entry. In some implementations, each tracking entry identifies the tracking chain from which it originates in a header indicated by one or more bytes / bits of data words. For example, each tracking entry can include a data structure with a defined field format (e.g., header, payload, etc.) that conveys information about a specific tracking event. Each field in the tracking entry corresponds to useful data that can be applied to the tracking point that generated the tracking entry.

[0096] As described above, each trace entry can be written to or stored in the memory unit of the chip manager 216 associated with the trace entry log 218. In some implementations, trace points can be individually enabled or disabled, and multiple trace points can generate trace entries of the same type but with different trace point identifiers.

[0097] In some implementations, each trace entry type may include a trace name, a trace description, and a header encoded to identify specific fields and / or sets of fields within the trace entry. The name, description, and header together provide a description of what the trace entry represents. From the perspective of chip manager 216, this description may also identify the specific trace chains 203, 205, 207 that a particular trace entry within a specific processor chip participates in. Therefore, the fields within a trace entry represent data slices (e.g., in bytes / bits) associated with the description and can be trace entry identifiers used to determine which trace point generated the specific trace entry.

[0098] In some implementations, the trace entry data associated with one or more of the stored hardware events can partially correspond to data communications occurring at least between: a) nodes 0 and 1; b) between components at least within node 0; and c) between components at least within node 1. For example, the stored hardware events can partially correspond to data communications occurring between at least one of: 1) FPC 104 of node 0 and FPC 104 of node 1; FPC 104 of node 0 and SPC 106 of node 0; 2) SPC 106 of node 1 and SPC 106 of node 1.

[0099] Figure 3 The diagram illustrates a block diagram of an example trace multiplexer design architecture 300 and an example data structure 320. The trace multiplexer design 300 generally includes a trace bus input 302, a bus arbiter 304 and a local trace point arbiter 306, a bus FIFO 308, at least one local trace event queue 310, a shared trace event FIFO 312, and a trace bus output 314.

[0100] Multiplexer design 300 corresponds to an example tracking multiplexer arranged within a component of system 100. Multiplexer design 300 can include the following functions: The bus in 302 can be associated with local tracking point data temporarily stored in bus FIFO 308 until time arbitration logic (e.g., arbitrator 304) can place the tracking data onto the example tracking chain. One or more tracking points of the component can insert tracking event data into at least one local tracking event queue 310. Arbitrator 306 provides first-level arbitration and can select events from the local tracking events stored in queue 310. The selected events are placed in a shared tracking event FIFO 312, which also serves as a storage queue.

[0101] Arbitrator 304 provides a second-level arbitration, receiving local tracking events from FIFO queue 312 and merging them into specific tracking chains 203, 205, 207 via tracking bus output 314. In some implementations, tracking entries may be pushed into local queue 310 faster than they can be merged into shared FIFO 312, or alternatively, tracking entries may be pushed into shared FIFO 312 faster than they can be merged into tracking bus 314. When these scenarios occur, the corresponding queues 310 and 312 will become full of tracking data.

[0102] In some implementations, when any queue 310 or 312 becomes full of tracking data, system 100 can be configured to discard the latest tracking entry and not store or merge it into a particular queue. In other implementations, when some queues are full (e.g., queues 310, 312), instead of discarding tracking entries, system 100 can be configured to stop the sample processing pipeline until a refilled queue has available queue space to receive entries.

[0103] For example, the processing pipelines of queues 310 and 312 can be stopped until a sufficient or threshold number of tracking entries are merged onto the tracking bus 314. The sufficient or threshold number can correspond to a specific number of merged tracking entries, creating available queue space for queues 310 and 312 to receive one or more tracking entries. Implementations that stop the processing pipelines before downstream queue space becomes available can provide higher fidelity tracking data by retaining rather than discarding certain tracking entries.

[0104] In some implementations, the local tracking queue has the same width as the tracking entry, so that each tracking entry occupies only one point in the local queue 310. However, the shared tracking FIFO queue 312 can use unique tracking entry line codes, allowing some tracking entries to occupy two positions in the shared queue 312. In some implementations, when any data in a tracking packet is discarded, the entire packet is discarded, so that partial packets do not appear in the tracking entry log 218.

[0105] Generally, a trace is a timeline of activities or hardware events associated with a specific component of system 100. Unlike performance counters (described below) which are aggregated data, a trace contains detailed event data, providing insights into hardware activity occurring during a specified trace window. The described hardware system broadly supports distributed hardware tracing, including generating trace entries, temporarily storing trace entries in hardware-managed buffers, statically and dynamically enabling one or more trace types, and streaming trace entry data to host system 126.

[0106] In some implementations, traces can be generated for hardware events performed by components of system 100, such as generating DMA operations, executing DMA operations, issuing / executing certain instructions, or updating synchronization flags. In some instances, tracing activities can be used to trace DMA over the system or to trace instructions executed on a specific processor core.

[0107] System 100 can be configured to generate at least one data structure 320 that identifies one or more hardware events 322, 324 from a timeline of hardware events. In some embodiments, data structure 320 arranges one or more hardware events 322, 324 in a time-ordered sequence of events associated with at least FPC 104 and SPC 106. In some instances, system 100 can store data structure 320 in a repository of the host control device of host system 126. Data structure 320 can be used to evaluate the performance of program code executed by at least processor cores 104 and 106.

[0108] As illustrated in hardware event 324, in some implementations, a specific trace identifier (ID) number (e.g., trace ID "003") can be associated with multiple hardware events occurring across distributed processor units. These multiple hardware events can correspond to specific memory access operations (e.g., DMA), and the specific trace ID number is used to associate one or more hardware events.

[0109] For example, as indicated by event 324, a single trace ID for a DMA operation can include multiple time steps corresponding to multiple different points in the DMA. In some instances, trace ID “003” can have an “issued” event, an “executed” event, and a “completed” event identified as being spaced a time interval from each other. Therefore, in this respect, trace IDs can be further used to determine the latency attributes of memory access operations based on correlation and according to time steps.

[0110] In some implementations, generating data structure 320 may include, for example, comparing the event timestamps of corresponding events in a first subset of hardware events with the event timestamps of corresponding events in a second subset of hardware events. Generating data structure 320 may further include, in part, providing an associated set of hardware events for representation in the data structure based on the comparison between the first and second subsets of events.

[0111] like Figure 3 As shown, data structure 320 can identify at least one parameter indicating a delay attribute of a specific hardware event 322, 324. The delay attribute can at least indicate the duration of the specific hardware event. In some embodiments, data structure 320 is generated by software instructions executed by a control device of host system 126. In some instances, structure 320 can be generated in response to the control device storing tracking entry data to a memory disk / cell of host system 126.

[0112] Figure 4 This is a block diagram 400 indicating an example tracing activity of a Direct Memory Access (DMA) tracing event performed by system 100. For DMA tracing, data from an example DMA operation initiated from a first processor node to a second processor node can travel through ICI 112 and intermediate hops of the ICI / router can be generated along the data path. As the DMA operation traverses ICI 112, the DMA operation will generate tracing entries at each node within the processor chip and along each hop. Information from each of these generated tracing entries is captured to reconstruct the time progression of the DMA operation along the nodes and hops.

[0113] Example DMA operations can be performed with Figure 4 The process steps depicted in the implementation are related. For this operation, local DMA transfers data from virtual memory 402 (vmem 402) associated with at least one of processor cores 104, 106 to HBM 108. The numbers depicted in FIG400 correspond to the steps in Table 404 and generally represent activities in or initiated by node organization structure 110.

[0114] Table 404 broadly describes the associated trace points. For this DMA, the example operation will generate six trace entries. Step one includes an initial DMA request from the processor core to node organization structure 110, which generates one trace point in the node organization structure. Step two includes a read command, in which node organization structure 110 requests data transfer from the processor core, which generates another trace point in node organization structure 110. When vmem 402 completes the read from node organization structure 110, the example operation does not have the trace entry of step three.

[0115] Step four includes node organization structure 110 performing a read resource update to cause a synchronization flag update in the processor core, which generates a trace point in the processor core. Step five includes a write command, in which node organization structure 110 notifies memory multiplexer 108 of the upcoming data to be written to HBM. A trace point is generated in node organization structure 110 via the notification of the write command, and in step six, the completion of writing to HBM also generates a trace point in node organization structure 110. In step seven, node organization structure 110 performs a write resource update to cause a synchronization flag update in the processor core, which generates a trace point in the processor core (e.g., FPC 104). In addition to the write resource update, node organization structure 110 can perform an acknowledgment update (“ack update”), in which the data completion of the DMA operation is signaled back to the processor core. The ack update can generate a trace entry similar to the trace entry generated by the write resource update.

[0116] In another example DMA operation, a first trace entry is generated when a DMA command is issued in the node organization structure 110 of the originating node. Additional trace entries can be generated in the node organization structure 110 to capture the time for reading data from the DMA and writing data to the output queue. In some implementations, the node organization structure 110 can divide the DMA data into smaller data blocks. For data divided into smaller blocks, read and write trace entries can be generated for the first and last blocks. Optionally, all data blocks except the first and last can be configured to generate trace entries.

[0117] For remote / non-local DMA operations that may require ICI hops, the first and last data blocks can generate additional trace entries at the entry and exit points of each intermediate hop along the ICI / router 112. When the DMA data arrives at the destination node, a trace entry similar to the entry in the previous node organization structure 110 is generated at the destination node (e.g., reading / writing the first and last data blocks). In some implementations, the final step of the DMA operation can include execution instructions associated with the DMA, causing a synchronization flag to be updated at the destination node. When the synchronization flag is updated, a trace entry indicating the completion of the DMA operation can be generated.

[0118] In some implementations, when each component is in tracing mode, DMA tracing is initiated by FPC 104, SPC 106, or HIB 114 to enable the execution of tracing points. Components of system 100 can enter tracing mode based on global control in FPC 104 or SPC 106 via a triggering mechanism. Tracing points are triggered in response to a specific action or condition associated with the execution of program code by a component of system 100. For example, portions of the program code can include embedded triggering functionality detectable by at least one hardware component of system 100.

[0119] Components of system 100 can be configured to detect triggering functions associated with portions of program code executed by at least one of FPC 104 or SPC 106. In some instances, the triggering function can correspond to at least one of: 1) a specific sequence of steps in a portion or module of the executed program code; or 2) a specific timing parameter indicated by GTC used by the distributed processor unit of system 100.

[0120] In response to the detection of a trigger function, a specific component of system 100 can initiate, trigger, or execute at least one trace point (e.g., a trace event), which causes trace entry data associated with one or more hardware events to be stored in at least one memory buffer of the hardware component. As described above, the stored trace data can then be provided to chip manager 216 via at least one trace chain 203, 205, 207.

[0121] As mentioned above, Figures 1 to 4 The method for hardware event tracing in a distributed computing system has been illustrated. The remainder... Figures 5 to 7 This describes a method for collecting synchronous hardware events in a distributed computing system. The synchronous collection of hardware events can be based, at least in part, on one or more triggering mechanisms.

[0122] Figure 5Figures 504 and 506 illustrate example data structures for the Global Time Counter (GTC) 502 and tables 504 and 506 indicating the use cases of the GTC 502 by example computing systems (e.g., system 100) for distributed hardware tracing. Figure 5 In one implementation, GTC 502 is a 64-bit value comprising a 60-bit counter and a 4-bit offset. In an alternative implementation, GTC 502 can be a fixed-size binary data slice having a bit size ranging from less than 64 bits to greater than 64 bits.

[0123] The data structures associated with GTC 502 are used in the global master counter and are described below. Figure 6 The implementation described herein is a local time counter (LTC). The high 60 bits of GTC 502 correspond to the actual time counter, which increments by 1 every cycle unless "append" or "subtract" is used to account for GTC phase changes (described below). For example, during an update of the local GTC 502 (i.e., the LTC), the local GTC 502 may not "mark" for several cycles in order to "catch up" with the main global GTC 502. In these scenarios, the lower four "offset" bits are used to compensate for GTC phase changes that may occur during program code execution.

[0124] When resolving GTC phase transitions, the last four bits can increment in these cases to partially preserve the difference between timestamps of local hardware events that would otherwise have the same GTC. In most other instances, the 4-bit offset of the GTC 502 is clear and therefore not used. In some implementations, the offset bits can count up to 15 and then stop. However, in alternative implementations, the offset bits can be counted to any integer value as needed by a specific counter design. In some instances, the offset bits are cleared when the high 60 bits of the GTC 502 begin marking again.

[0125] Regarding the configuration and management of GTCs 502 and 602 (described below), in some implementations, the executed program code can configure or establish the GTC by configuring one or more processor chips of the system 100. Configuring the processor chip can include specifying a global primary GTC for the system and specifying a local primary GTC (i.e., LTC) for the corresponding node or component of the system 100.

[0126] The component can include executable program code / software used to calculate and compensate for clock delays and execute one or more local GTCs (i.e., Figure 6The system 100 includes a hardware knob / feature for manual tuning of the LTC (Local Time Tolerance). LTC compensation for clock delay and manual tuning is performed relative to the main GTC to minimize or mitigate phase changes. In some implementations, system 100 may include an automatic pilot mode for delay compensation, where component hardware features automatically determine link delay via a ping-and-echo mechanism.

[0127] In some implementations, system 100 may include a linker / software stack that includes portions or subsets of program code for adjusting parameter values ​​of a sample control / status register (CSR). In some instances, control signals generated in response to the execution of the linker stack program code cause one or more parameter values ​​to be written to the CSR. These CSR parameter values ​​can be used to configure various aspects of the GTC 502, 602, or LTC.

[0128] For example, one or more parameters can include: 1) the GTC_config parameter, which prompts the selection of a primary global GTC, a local primary GTC, or a local secondary GTC and selects the update sample size; 2) the GTC_sync parameter, which is used to reset the primary GTC or clear "ping" measurements; 3) the GTC_latency_compensation_control parameter, which configures manual latency compensation; 4) the GTC_link_ping parameter, which indicates the minimum ping latency; 5) the GTC_count parameter, which indicates the actual GTC value; and / or 6) the GTC_max_diff parameter, which indicates the maximum observed difference between the local GTC and the primary GTC.

[0129] As described above, system 100 can use the global master GTC 502 and one or more local GTCs 502 to obtain event timestamps and determine the sequential order of hardware events occurring within the system. In addition to the order between activities, the GTC 502 can be used to determine the latency of activities. In some implementations, a particular GTC 502 can exhibit phase changes; therefore, the determined latency may be inaccurate over one or more cycles.

[0130] As described above, depending on whether the local GTC differs from the global master, i.e., when a phase change occurs, the increment rate of some GTC 502 can be changed. In a steady state, when there is no phase change, the main global GTC 502 increments once per cycle and can be used to count local activities (e.g., a component) and the delay and sequence of the LTC of a particular component.

[0131] In some implementations, the main GTC 502 count may be slower than a certain LTC. In these cases, the offset bits of GTC 502 increment continuously once per local clock cycle. As shown in Tables 504 and 506, given the unique function of the GTC offset bits, an extended GTC (“GTC-ext”) can be defined for use by system 100. In some implementations, GTC-ext can be constructed by adding an upper 60 bits (GTC:top) and a lower four offset bits (GTC:offset), while in other implementations, program code executing in a particular processor can calculate that GTC-ext is equal to the sum of GTC:top and the fractional GTC:offset value.

[0132] In some implementations, and in response to the increment of the offset bits, the GTC-ext parameter can be used to determine event sequencing and the latency of local operations. Tables 504 and 506 show use cases where the GTC-ext parameter may prove useful when determining hardware event sequencing and event operation latency.

[0133] Table 504 describes example GTC usage for determining the order between activities or hardware events. In some implementations, the rate at which GTC 502 increments during activity analysis may be less relevant than when GTC 502 is used to determine the global order, such as the order of one or more activities between at least two different compute nodes (e.g., different nodes each having a primary GTC). However, when the local GTC 502 at a particular node is slower than the primary GTC of that node, GTC-ext can be used to distinguish between two local activities, or the timing of two local activities initiating writes to remote resources, as shown in feature 508 of Table 504.

[0134] Table 506 describes an example of GTC usage for calculating the delay of local activities / events or activities spanning two or more nodes. In some implementations, system parameters may not indicate whether the GTC 502 at a particular chip is being flagged at a normal rate (i.e., synchronized with the specified master global clock) or at a rate asynchronous with the specified master global clock. Therefore, in some instances, the LTC can be used for event delay determination of local activities, as shown in feature 510 of Table 506. However, in other implementations, the GTC or GTC-ext can be used to measure the delay of one or more operations.

[0135] Generally, hardware trace events occurring at various nodes or components of system 100 can include GTC parameters. In some implementations, when traced hardware events / activities span multiple nodes of system 100, the latency associated with local activities should be calculated via GTC rather than LTC. In these implementations, using GTC or GTC-ext may be a preferred counter / clock solution because LTCs across two or more different nodes may not be synchronized or timed / marked at the same rate.

[0136] Figure 6 The diagram illustrates an example time counter and one or more counters associated with a corresponding component of an example computing system (e.g., system 100) used for distributed hardware tracing. The time counter can include a global time counter (GTC) 602 and multiple local time counters (LTCs), each associated with a corresponding component of system 100.

[0137] For clarity, although Figure 6 A single LTC block coupled to a specific component of system 100 is depicted, but in an alternative implementation, each component can include one or more local time counters / LTCs. Additionally, although Figure 6 A single master GTC 602 is depicted, but this feature can also correspond to or represent a specific LTC selected to act as the global master. For example, in some implementations, GTC 602 corresponds to counter selection logic implemented by host system 126 to select a specific LTC to act as the global master clock / counter.

[0138] In some instances, system 100 can select an example node of ICI 112 to act as the global master clock, and the example node's LTC is marked with 1 per clock cycle. However, in an alternative implementation, GTC 602 is a global time counter that acts as the global master clock and broadcasts specific timing parameters to the components of system 100. In either implementation, GTC 602 can be a global synchronization clock used to correlate hardware events occurring across system 100.

[0139] In some implementations, as described above, the GTC 602 may have small phase variations across system 100, but can be configured to avoid long-term time drift. For example, the GTC 602 can be flagged or counted based on an example oscillator for a specific processor chip (e.g., an oscillator selected as the global host). Generally, there is no long-term drift or phase variation for the selected oscillator of the processor chip in system 100; however, drift or variation may occur over time relative to oscillators across different processor chips. Therefore, at processor chips that are not the global host, the GTC 602 may increment (i.e., flag) faster or slower than the local clock (LTC).

[0140] As a global host, the current host value of GTC 602 can be broadcast to all nodes of ICI 112. Example ICI nodes can correspond to one or more components of system 100. Receiving nodes can calculate the difference between the local values ​​of the example GTC used for local operations for a specific component / node. The minimum difference between the host value and the local GTC value received within the example sampling period can be used to update the local GTC. Over time, based on counter adjustments implemented using one or more offset bits as described above, the phase change between the host value and the local GTC value (if present) can be "appended".

[0141] like Figure 6 As shown in the implementation, each component of system 100 may include one or more performance counters, and the set or subset of performance counters within system 100 is described herein as a plurality of counters 600. In particular, and for each component, FPC 104 may include a plurality of counters 604, SPC 106 may include a plurality of counters 606, multiplexed memory (HBM) 108 may include a plurality of counters 608, node organization structure 110 may include a plurality of counters 610, ICI 112 may include a plurality of counters 612, and HIB 114 may include a plurality of counters 614.

[0142] In system 100, performance analysis of program code executed by one or more processors can be enabled / disabled based on one or more hardware performance counters associated with specific components of the system. These performance counters can correspond to... Figure 6 Multiple corresponding counters, and capable of including at least one of the following: 1) an activity counter; 2) a pause counter; 3) a statistics counter; and 4) a sampling counter.

[0143] In some implementations, system 100 includes various other types of performance counters that can be programmed to increment and store specified count data associated with technical aspects of the executed program code. Additionally, by using one or more counters, hardware support for tracking activity can include functions such as: capturing instruction trace event data; capturing memory multiplexer trace event data; capturing DMA trace event data; and capturing event data in a trace buffer.

[0144] In some implementations, at least a subset of the plurality of counters 600 may be CSRs or counting registers accessible to the host system 126 via HIB 114. These counting registers / CSRs may be storage devices configured to store various types of count data identifying instructions received, processed, or executed by a particular hardware component, as well as various other types of data / information associated with technical aspects of the executing program code. In some instances, data is stored in the form of counts or increments associated with a particular instruction or processor operation (e.g., counting parameters). In some implementations, the plurality of counters 600 of system 100 may correspond to hundreds or thousands of performance counters / counting registers.

[0145] Regarding the at least four types of performance counters mentioned above, in some implementations, the activity counter can be used to store and analyze data associated with utilization parameters of different hardware units. The activity counter can also be used to analyze portions of the executed program code or to analyze the quantity or type of data transferred during code execution. For example, some processor cores can include activity counters used to derive instruction mix ratios in various instruction streams. For example, the activity counter can be configured to store counts associated with executing a specific instruction type in an exemplary instruction stream.

[0146] In some implementations, the activity counter may include a subset of counters, such as: a release counter that increments after certain instruction types are issued; a memory activity counter that is available on a memory path and increments in response to reads / writes associated with inter-memory transfers (e.g., between multiplexer 108 and VMEM 402); and one or more other general activity counters that increment in response to instructions such as interrupt, synchronization flag, and alarm counters.

[0147] Regarding pause counters, in some implementations, these counters can provide a pause count, indicating the number of cycles a hardware unit / component has stopped for a specific pause reason, instead of performing useful work associated with instruction processing or data computation. For example, when the utilization of a particular hardware unit is below a threshold utilization or the program code does not execute at the expected threshold performance level, the pause counter can be used by system 100 to obtain relevant pause information for root cause analysis of utilization and performance problems (e.g., below-threshold utilization / performance).

[0148] In some implementations, the pause counter can provide information related to: pauses based on the detection that some normally available memory path is indicated as unavailable; pauses based on the delayed execution of one or more synchronization flag instructions; and / or pauses caused by one or more normally available execution units (e.g., XU 122 or RPU 124) indicated as unavailable.

[0149] In some implementations, FPC 104 may include at least one scalar unit that provides a scalar for vectorized computation performed by VPU 118. A pause counter associated with the scalar unit can provide information about pauses caused by one or more hazards that may occur within the scalar unit. In some instances, various hazard types may occur, and FPC 104 may include a pause counter for each hazard type. Example hazard types may include DMA latency hazard, scalar fence latency hazard, and / or scalar latency instruction hazard.

[0150] Regarding statistical counters, in some implementations, components of system 100 may include one or more data queues (e.g., storage media), and the activity counter may be configured to provide information related to queue utilization. However, in some instances, the activity counter for queues may be insufficient to provide complete information about queue utilization effectiveness by executing program code. In these cases, a statistical counter may be available in one or more components to obtain certain data counts in order to determine calculations related to average queue occupancy and the time spent in certain queues.

[0151] In some implementations, the queue statistics counter can provide queue-related statistics for queue occupancy, queue insertion status, queue full status, or threshold occupancy status. In some instances, when the queue statistics counter is active, the occupancy counter associated with the component queue increments based on the current queue occupancy in each processor / instruction cycle.

[0152] In other instances, when the active queue statistics counter provides counts related to the queue full state, the counter increments (e.g., by 1) in the period when the queue is fully occupied. Additionally, when the active queue statistics counter provides counts related to the queue insertion state, the counter increments in response to the data block placed into the queue (i.e., queue insertion). Furthermore, for queues in a threshold occupancy state, the counter can increment (e.g., by 1) in every period where the current queue occupancy rate is greater than a specific queue threshold occupancy rate.

[0153] In some implementations, components of system 100 may include a set of sampling counters that can be periodically read from host system 126 to generate data structures (e.g., structures 320 or 620). The data structures may include sampling statistics of detailed data processing information about hardware events, tracing activities, or activities / events occurring in the distributed processor units of system 100. In some implementations, the data structures generated based on the sampling statistics may include, for example, structure 320 (as described above) or structure 620 (as described below). In some instances, the generated data structures are used to construct one or more monitoring tools for monitoring engine 128 of host system 126 to analyze the performance of program code executed by components of system 100.

[0154] As described above, components of system 100 can initiate event tracing based on global control associated with one or more triggering mechanisms. In some embodiments, multiple counters 600 can be enabled, disabled, or otherwise controlled to initiate triggering mechanisms that cause tracing points to execute and generate tracing entries including specific data related to one or more hardware events. In some embodiments, tracing entries related to hardware events are at least partially based on event data populated from aggregated count data from one or more of the aforementioned multiple counters 600.

[0155] Global control generally includes the use of CSR control (i.e., specific CSR hardware features / knobs) and triggering mechanisms that allow counter increments and enable trace generation in a coordinated and synchronized manner. In some implementations, coordinated and synchronized control can occur across distributed processor units of system 100. As mentioned above, synchronized trace can be enabled in part by using a globally synchronized counter that serves as a global master clock (e.g., GTC 602).

[0156] In some implementations, system 100 triggers synchronization event tracking based on a specific time parameter (e.g., time parameter 09:01.13) indicated by the global clock of the distributed processor unit. For example, system 100 can use the global master clock value of GTC 602 as a trigger to start and stop all counters and accurately track the duration of a specific operation.

[0157] In some instances, the sample software compiler of System 100 is able to embed or insert instructions within the executable program code, including operand or parameter values ​​based on a triggering mechanism associated with a specific predefined time window of GTC 602. The predefined time window can include an example trace duration having a trace event start time (e.g., a first GTC 602 time parameter) and a trace event end time (e.g., a second GTC 602 time parameter that is later in time than the first GTC time parameter).

[0158] like Figure 6 As shown in the implementation, the components of system 100 can include multiple performance counters 600. Since they are related to triggering mechanisms, performance counter control can include mechanisms for enabling and disabling counters, as well as for clearing and pausing counters. In addition to the statistical counter functions described above, some hardware units can include statistical counters with associated selectors (e.g., CSRs), which are used to select a subset of hardware blocks within the hardware unit whose events are being counted.

[0159] As described above with reference to the trace multiplexer control, in some implementations, tracing can be enabled and disabled based on CSR writes performed by system 100. In some instances, tracing can be dynamically started and stopped by system 100 based on the value of the example trace flag (TM) register in FPC 104 (or core 116) or based on the value of the example TM register in SPC 106.

[0160] In some implementations, example trace control parameters can be used to define a specific trace window. For example, trace control parameters can allow the trace window to be defined based on the lower and upper bounds of GTC 602 (e.g., start and stop time clock values) and the lower and upper bounds of the example trace flag register in FPC 104 or SPC 106.

[0161] In some implementations, the program code executing within system 100 can include triggering processing logic to trigger a trace event in each processor cycle within one of FPC 104 or SPC 106. In some instances, the triggering logic is associated with a specific "set trace flag" instruction, which can be inserted by the executing software / program code into one or more instruction streams.

[0162] For example, regarding the triggering logic of FPC104, when a sample instruction to set a trace marker is issued within FPC104, the triggering logic can include analyzing the trace marker operand of the instruction and comparing the parameter value of the operand with the value of the sample trace marker window. This analysis and comparison is used to determine whether the operand parameter value is equal to a lower bound or falls within the lower to upper bound of a specific trace marker window.

[0163] If the operand parameter value falls within the defined range of the tracking marker window, a trigger can be activated (e.g., a trigger condition is met), and FPC 104 can enter tracking or counting mode and activate one or more of the multiple counters 604. In some implementations, when a subsequent "Set Tracking Marker" instruction includes an operand parameter value, wherein the operand parameter value, based on the operand value being equal to or exceeding the upper bound of the example tracking marker window, causes a stop counting trigger to be activated, and FPC 104 exits tracking / counting mode.

[0164] In some implementations, system 100 can detect the fulfillment of a triggering condition based on at least one of the following: i) identifying an example trace marker operand appearing in at least a first portion of the executing program code. The trace marker operand can include parameter values ​​(e.g., code step sequence / values) for initiating one or more trace events and / or performance counts. System 100 can also detect the fulfillment of a triggering condition based on determining a predefined time value indicated by the current time GTC 602 for initiating one or more trace events and / or performance counts.

[0165] In response to the detection that a trigger condition has been met, the processor component of system 100 can generate a control signal, which is received by a counter register of the processor component. The control signal enables various count data associated with one or more hardware events to be stored in the counter register. In some embodiments, system 100 can be configured or programmed, for example, via host system 126 to generate data structures indicating one or more performance attributes associated with executable program code. For example, the data structures can be generated based on one or more count parameters of the stored count data. In some embodiments, the counter register is one of a plurality of performance counters 600 configured to store count data relating to the performance of one or more processor components of system 100.

[0166] To further illustrate trace mark triggering, in the example code sequence, the trace mark setting instruction can include a 32-bit trigger operand that identifies a computational step in a distributed software architecture (e.g., a sample neural network inference workload). The FPC 104 can include a dedicated CSR, identified in the program code as "tracemark_limits". While a 32-bit operand is described in this example code sequence, in alternative implementations, the trigger operand or register limit operand / parameter value can be a binary data structure with fewer or more than 32 bits.

[0167] This dedicated CSR can encode at least two 32-bit values. The first encoded 32-bit value can correspond to a lower bound of the tracking marker used to start tracking / counting, while the second encoded 32-bit value can correspond to an upper bound of the tracking marker used to stop tracking / counting. In some implementations, the first and second encoded values ​​correspond to computational steps in a distributed software program / construct, such as an example neural network inference workload, or any other distributed program code.

[0168] Referring again to the example code sequence, when the "Set Trace Mark" instruction is executed, the 32-bit operand value is compared with the first and second encoded values ​​in the tracemark_limits register. Based on this comparison, if the trigger operand equals / matches the start value (i.e., the first encoded value) or has a higher value, a profiling activity is triggered. Similarly, if the trigger operand matches the stop value (i.e., the second encoded value) or has a higher value, the ongoing profiling activity is stopped.

[0169] Similar to trace flag triggering, when GTC 602 is used as a trigger (e.g., for FPC104 or SPC 106), the upper and lower bounds of the example GTC / LTC trigger register for a specific component can be evaluated against the current value of GTC 602. Therefore, the current time parameter or current value of GTC 602 can be compared with the lower bound of the example predefined time window to perform trace activity using GTC triggering. Similarly, after executing GTC triggering to start counting activity, the current time parameter or current value of GTC 602 can be compared with the upper bound of the GTC trigger register to determine whether GTC 602 matches or exceeds the upper bound value. In response to GTC 602 at least matching the upper bound value, a stop counting trigger is fired, causing the component to exit counting mode.

[0170] In some implementations, SPC 106 includes corresponding instructions to the set trace marker instructions of FPC 104. Therefore, the triggering logic of SPC 106 can operate in the same or substantially similar manner as FPC 104. For example, trace marker parameters can be included as operands of one or more instructions issued by SPC 106. In some instances, operand triggering evaluation occurs at the time of each instruction issuance, and the evaluation logic can be substantially similar to the way counting is enabled in FPC 104 (e.g., using trace marker operands). For example, in SPC 106, tracking is enabled when the trace marker operand included in the set trace marker instruction is within the lower and upper bounds of the example tracking window. Similarly, tracking is disabled when the trace marker operand included in the trace marker instruction is outside the lower and upper bounds of the tracking window.

[0171] As described above, in addition to using GTC 602 or trace markers, system 100 can include a "per-person" tracing mode. Generally, this per-person tracing mode allows tracing across the entire processor chip to be controlled by control signals generated, for example, by components such as FPC 104 or SPC 106. In some embodiments, the control signals can be global control parameters indicating specific performance states of the distributed processor units of system 100. In some instances, specific performance states correspond to the "per-person" tracing mode.

[0172] In some implementations, one or more hardware components (e.g., HIB 114 or ICI 112) can be programmed or configured to initiate event tracking in response to activation of the "everyone" tracking mode, or simply ignore propagation control signals indicating the tracking mode for each person. Therefore, while system 100 can include signaling mechanisms for propagating each tracking control signal, system 100 can also include mechanisms that cause at least one hardware component to ignore control signals.

[0173] For example, a user of system 100 (e.g., a performance analyst) might want to use HIB114 to trace packets indicating all host communications. The user can, for example, insert example program instructions via an external compiler to put HIB114 into "always-tracking" mode while also using per-person tracking mode to capture certain single-step data sequences from other hardware components of system 100. Therefore, when programmed to execute in always-tracking mode, HIB114 can ignore per-person tracking control signals used to generate single-step sequence tracking data from other hardware components.

[0174] In some implementations, a trace point can be triggered in response to a specific action or condition associated with the execution of program code by a component of system 100. For example, the distributed software program can include multiple software modules, each comprising shared or overlapping sequences or steps of code. For example, processor cores 104 and 106 can each receive and execute portions of the distributed software program, and each program portion can include example code sequences 1-100.

[0175] As described above, in some embodiments, system 100 can be configured to include one or more triggering conditions / mechanisms having or based on specific sequence steps of distributed program code. Therefore, system 100 can be configured to trigger a trace point in response to the arrival of a certain encoded sequence / step (e.g., code sequence step 33) in each of processor cores 104, 106 during code execution. In some instances, portions of the executed distributed program code can include embedded triggering functionality detectable by at least one hardware component of system 100.

[0176] For example, as described above, system 100 can be configured to detect the satisfaction of one or more trigger conditions based on the presence of operands in at least a portion of program code executed by either FPC 104 or SPC 106. In response to detecting the satisfaction of at least one trigger condition, system 100 can initiate one or more tracing events that generate tracing entries / tracing data or count data. In some embodiments, the generated tracing entries can include at least one attribute (e.g., shared timestamps or code sequence steps) shared between corresponding hardware events occurring across distributed processor units.

[0177] For example, as shown in data structure 620, the generated trace entry 622 can correspond to a synchronous trace event, and the trace data can include a unique trace identifier for the corresponding hardware event and a global hardware event timestamp (e.g., 09:01:13) shared between at least two corresponding hardware events. Therefore, for trace entry 622, at least one attribute shared between the corresponding hardware events can be the hardware event timestamp. Similarly, when system 100 triggers a trace point based on a specific code sequence, for trace entry 624, at least one attribute shared between the corresponding hardware events can be the code sequence or a trace marker step.

[0178] In some implementations, a tracking entry may include a timestamp header or field. The timestamp header may include a 48-bit timestamp that identifies the time the tracking entry was generated. In some instances, the timestamp header may correspond to the lower 48 bits of the GTC 502.

[0179] As described above, in some embodiments, the compiler of system 100 is capable of inserting one or more operands for multiple trigger conditions into portions of program code executed by the example processor core of system 100. For example, the compiler associated with FPC 104 is capable of inserting or embedding one or more operands for trigger conditions into portions of program code executed by FPC 104. Similarly, the compiler associated with SPC 106 is capable of inserting or embedding one or more operands for trigger conditions into portions of program code executed by SPC 106.

[0180] Generally, as described above, system 100 is configured to detect one or more trigger conditions and initiate at least one tracing event in response to the detection of a condition. In some embodiments, initiating at least one tracing event may include: system 100 providing, for example, a first control signal to a first performance counter / register (e.g., a CSR such as an activity counter) of FPC 104. This first control signal enables data associated with at least one hardware event to be stored in the first performance counter.

[0181] Similarly, in some embodiments, initiating at least one trace event may further include: system 100 providing, for example, a second control signal to a second performance counter / register (e.g., a statistical counter such as a CSR) of SPC 106. This second control signal enables data associated with at least one hardware event to be stored in the second performance counter.

[0182] In some implementations, the data associated with at least one hardware event may include one of the following: 1) the number of bytes written to a specific memory buffer of a processor core of a distributed processor unit of system 100; or 2) the number of instructions executed by a specific processor core of a distributed processor unit of system 100.

[0183] Generally, the described tracing system includes a hardware-based triggering mechanism. Triggers can be pre-configured to be activated based on software semantics. For example, operands for trigger conditions can be inserted or embedded in the preamble and end of a software component (e.g., function headers, loop headers, or other appropriate locations in a distributed software system). As described above, in some instances, the insertion of operands for trigger conditions is done by the compiler in system 100, while in other instances, the trigger and trigger operands can be embedded / inserted by the compiler before the code is executed by system 100, or by the host system 126 during real-time code execution.

[0184] For example, in some implementations, performance analysis software running on host system 126 can establish conditions across distributed hardware systems (e.g., in each chip / node of system 100) indicating when a trigger should occur. When a trigger occurs, hardware tracing and performance counting are performed in a specific hardware unit, and similar trigger conditions are set to stop tracing data collection.

[0185] In other implementations, the trigger in system 100 can be a monotonically increasing value as an operand of an instruction. This instruction can be inserted into the introduction and conclusion of important software structures by a sample compiler of the processor core. Generally, for executing program code, the performance penalty caused by keeping the trigger operand in the instruction always present is negligible because performance data is not collected before the trigger fires.

[0186] For example, trigger conditions can be set by writing to a sample register in the hardware component. To evaluate or detect a trigger, the hardware component can compare the value in that register with the operand of the trigger condition in the instruction whenever a trigger instruction is encountered. Generally, trigger conditions should be set before certain coded sequence steps so that processor chips / nodes in a distributed system do not reach the trigger conditions prematurely.

[0187] In some instances, hardware triggering implementations require propagating trigger signals across different components of the example processor chip / node. At least one benefit of the triggering mechanism is that it enables a distributed software program to execute an activity (e.g., code sequence step 33) on a first node at a time significantly different from the related activity (e.g., code sequence step 33) at another (second) node. In this embodiment, system 100 is able to maintain accurate collection of hardware performance data on the same or substantially related activities executed by different hardware components or processor nodes at different time periods.

[0188] In some implementations, example triggering use cases can involve system 100 collecting multiple sets of trace data without reprogramming, resetting, or re-inserting specific triggering conditions. For example, the executing program code can include a software structure that generates a sequence of instructions in which the same operand that sets the trace marker appears multiple times. In some instances, the same operand that sets the trace marker may appear multiple times as the processor of system 100 executes a portion of the program code in a loop. During the execution of the program loop, event tracing can begin and end according to the iteration of the program loop.

[0189] For example, based on the operand value appearing at the start of a loop iteration, a triggering condition can be met, such as triggering an event and initiating event tracing. Similarly, based on the operand value appearing at the end of a loop iteration, the triggering condition can no longer be met, such as event tracing stopping. Generally, the start and end of a loop iteration can correspond to the boundaries of a program loop for a given software structure. Therefore, a tracing session can begin and end at each execution of the software structure (e.g., each loop iteration).

[0190] In some implementations, the appropriate trigger operation value (e.g., GTC or trace flag) can be determined in the following ways. As described above, the distributed software program can include software modules, each comprising shared or overlapping sequences or steps of code. During the execution of the distributed program code within system 100, a performance analyst can read or analyze the CSR value of the trace flag register via the computing console of system 100 to, for example, determine the current code sequence or step number.

[0191] In some instances, analysts repeatedly read CSR values ​​to determine the average growth rate of a sequence or step. The determined growth rate can then be used to predictively set one or more trigger conditions based on the occurrence of, for example, a tracking marker operation value. In some implementations, when using GTC, performance analysts can read or analyze the system / machine (or hardware component) clock speed to predict or determine the rate of GTC increase. Analysts can then use the determined rate to set one or more trigger conditions based on the occurrence of a specific GTC value.

[0192] Figure 7 This is a process flowchart of an example process 700 using computing system 100 and one or more nodes 200, 201 of system 100 for distributed hardware tracing. Therefore, process 700 can be implemented using one or more of the aforementioned computing resources of system 100, including the resources of nodes 200, 201.

[0193] Process 700 begins at block 702 and includes a computing system 100 monitoring the execution of program code executed by a first processor core. In some embodiments, the first processor core is configured to execute at least a first portion of the monitored program code. At block 704 of process 700, system 100 monitors the execution of program code executed by a second processor core. In some embodiments, the second processor core is configured to execute at least a second portion of the monitored program code.

[0194] In block 706, system 100 detects that at least one trigger condition is met. In some embodiments, system 100 detects that the trigger condition is met based on the presence of an operand in at least a portion of program code executed by one of the first or second processor cores. In some embodiments, the program code executed within system 100 includes trigger processing logic for triggering a traced event.

[0195] In some instances, the triggering logic is associated with a "Set Trace Mark" instruction inserted into one or more instruction streams. The triggering logic can include identifying the trace mark operand of the instruction that appears, analyzing the trace mark operand, and comparing the operand with the value of a sample trace mark window. In some instances, this analysis and comparison are used to detect that a first triggering condition is met.

[0196] In other instances, the triggering logic is associated with predefined time values ​​that can be inserted into one or more instruction streams by an external user of system 100 or by a sample compiler of the system. The triggering logic may include determining a predefined time value that indicates the current time of at least one clock of the computing system to initiate one or more trace events. In some implementations, determining the predefined time value may include receiving the predefined time value and comparing it with the time value of an example predefined time window.

[0197] In block 708, in response to detecting that a trigger condition is met, system 100 initiates at least one first tracing event, which generates tracing data. The generated tracing data identifies corresponding hardware events occurring across distributed processor units, including at least a first processor core and a second processor core.

[0198] For example, in response to detecting that a trigger condition is met, system 100 can initiate a synchronization tracking event, which generates tracking data that identifies hardware events occurring across FPC 104 and SPC 106. In some implementations, the tracking data identifies a unique tracking identifier for the corresponding hardware event.

[0199] In some instances, a hardware event comprises multiple synchronized hardware events, and when the events share a global hardware event timestamp, at least two hardware events of the multiple synchronized events are synchronized.

[0200] In some implementations, multiple triggering conditions can be associated with executable code.

[0201] In some instances, a specific triggering condition can be satisfied based on the identification of a specific operand appearing in at least a portion of program code executed by at least a first or second processor core. This specific operand can include one of a plurality of parameter values, each of which can be used to initiate a second tracing event in one or more tracing events. In this embodiment, detecting that a specific triggering condition is satisfied can include the system 100 detecting that a specific parameter value exceeds a first limit value of a register.

[0202] In some implementations, in response to detecting that a specific triggering condition is met, system 100 initiates a trace event to generate trace data that identifies at least one attribute shared among corresponding hardware events occurring across distributed processor units. In some implementations, the attribute of the corresponding hardware event may include at least a source memory address, a destination memory address, or a sequence / program computation step. The at least one shared attribute may include a specific sequence step or program step of program code.

[0203] In some implementations, one or more triggering conditions can be associated with at least one of the following: 1) a specific sequence of steps in the program code; 2) a global control parameter indicating a specific performance state of the distributed processor unit; 3) the current time or a specific time parameter indicated by the global clock / counter of the distributed processor unit; or 4) a predefined time window associated with the global clock.

[0204] In some implementations, system 100 can also detect that a specific parameter value or a specific predefined time value of a particular operand exceeds a second limit value of the register. In some instances, in response to this detection, system 100 can stop the second tracking event when a specific parameter value of a particular operand or a predefined time value exceeds the second limit value of the register.

[0205] At block 710 of process 700, system 100 provides the host device with an associated trace dataset including corresponding hardware events. In some embodiments, the associated trace dataset indicates at least one time-ordered sequence of corresponding hardware events generated when a first triggering condition is met. For each of the corresponding hardware events, the trace data includes at least a hardware event timestamp of the hardware event and metadata characterizing the hardware event. In some instances, the associated trace dataset corresponds at least partially to data structure 620.

[0206] In block 712, computing system 100 uses associated trace datasets to analyze the performance of program code executed by at least the first processor core and the second processor core. In some embodiments, host system 126 uses data structure 620 (i.e., corresponding to the associated trace data) to analyze the performance of program code executed by at least processor cores 104 and 106. Similarly, host system 126 can use data structure 620 to analyze the performance of at least one component of system 100.

[0207] For example, host system 126 or example user can analyze data structure 620 to detect or determine whether there are performance problems associated with a specific software module executing program code. Indicative problems may include the software module failing to complete the execution of certain instruction streams within the allocated execution time window, or experiencing latency below a threshold.

[0208] Additionally, the user or host device / system 126 can detect or determine whether a specific component of system 100 is operating above or below a threshold performance level. Example problems related to component performance may include specific hardware components generating result data that exceeds an acceptable threshold result parameter range. In some implementations, the generated result data may be inconsistent with result data generated by other relevant components of system 100 performing substantially similar instructions / operations.

[0209] For example, during the execution of program code, a first component of system 100 may be required to perform an operation and produce a result. Similarly, a second component of system 100 may be required to perform a substantially similar operation and produce substantially similar results. Analysis of the associated tracking dataset can indicate that the results produced by the second component are significantly different from those produced by the first component. Likewise, example data structures of associated tracking entries can indicate that the result parameter values ​​of the second component are outside the acceptable parameter range. These results can indicate potential performance problems with the second component.

[0210] Embodiments and functional operations of the subject matter described herein can be implemented in digital circuits, in tangibly embodied computer software or firmware, in computer hardware, or in a combination of one or more of these, including the structures described herein and their structural equivalents. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by or control of their operation by a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electronic, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0211] The processes and logic flows described in this specification can be executed by a programmable computer that executes one or more computer programs to perform functions by manipulating input data and generating data. The processes and logic flows can also be executed by dedicated logic circuits, and the device can be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units).

[0212] Computers suitable for executing computer programs include, for example, central processing units (CPUs) capable of being based on general-purpose or special-purpose microprocessors or both, or any other type. Generally, the CPU receives instructions and data from read-only memory or random access memory or both. The basic elements of a computer are the CPU for implementing or executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes, or is operatively coupled to, one or more mass storage devices for storing data, to receive data from or transfer data to, or both, such as magnetic magneto-optical disks or optical disks. However, a computer need not have such devices.

[0213] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor storage devices including, for example, EPROM, EEPROM, and flash memory devices, and disks such as internal hard disks or removable disks. Processors and memories can be augmented with or combined with dedicated logic circuitry.

[0214] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments can also be implemented in combination within a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as operating in certain combinations or even as initially claimed features, one or more features from a claimed combination can be excluded from said combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0215] Similarly, although operations are depicted in a specific order in the diagrams, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all the illustrated operations to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, but rather it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0216] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and the desired result can still be obtained. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to obtain the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for collecting event data about neural network computation for a neural network having multiple neural network layers, the method comprising: Execute program code to perform neural network computations using components of a processor configured to implement the neural network; The processor includes a global time counter, and each of the components of the processor includes a corresponding local time counter. Wherein, the global time counter and the corresponding local time counter each include at least one corresponding offset bit, and the at least one corresponding offset bit is used to reduce the phase change between one or more of the corresponding local time counters and the global time counter; The program code includes a first time parameter as a triggering condition to trigger a tracking event across two or more components of the processor during the execution of the neural network computation. The following is used to synchronize the corresponding current time value indicated by the corresponding local time counter based on the current time value indicated by the global time counter: at least one corresponding offset bit of each corresponding local time counter in two or more of the components of the processor; By comparing the corresponding current time value that is synchronized with the first time parameter in the program code, it is determined whether the triggering condition for triggering the tracking event is met; In response to determining that the triggering condition is met, the tracking event is triggered to generate event data for two or more components in the components of the processor; The event data is synchronized based on the corresponding current time value of the global time counter; and The event data is provided to a host used to analyze the program code during the execution of the neural network computation.

2. The method according to claim 1, comprising: The event data is stored in one or more of a plurality of trace buffers set in the first component of the processor.

3. The method according to claim 1, wherein, Each of the global time counter and the corresponding local time counter includes a fixed-size binary data, which comprises a 60-bit counter and a 4-bit offset. The program code also includes a second time parameter; Wherein, the second time parameter indicates a time later than the first time parameter; and The triggering condition includes a predefined time window associated with the global time counter, wherein the predefined time window includes a start time for triggering the tracking event based on the first time parameter and an end time for stopping the tracking event based on the second time parameter.

4. The method according to claim 1, wherein: Executing the program code to perform the neural network computation includes executing a sequence of operations to process vector elements of the inference workload using the neural network.

5. The method according to claim 1, wherein, Synchronizing the corresponding current time value indicated by the corresponding local time counter based on the current time value indicated by the global time counter further includes: For each of the respective local time counters, calculate the difference between the corresponding current time value indicated by the local time counter and the current time value indicated by the global time counter; and Determine whether the difference satisfies the minimum value within the sampling period.

6. The method according to claim 4, wherein: The subset of the event data describes multiple memory access operations, which are executed by the processor to perform the neural network computation. and The operation sequence includes moving the vector element from the first memory of the first component to the second memory of the second component of the processor.

7. The method according to claim 6, wherein, The event data that triggers the tracing event to generate the event data of two or more components in the processor includes: During the process of moving the vector elements from the first memory of the first component to the second memory of the second component, the subset of event data synchronized between the first component and the second component is generated.

8. The method according to claim 1, further comprising: In response to compiling the program code for execution at the processor, the first timing parameter is embedded by the host. and The host loads the compiled program code, which includes the first time parameter embedded in the compiled program code.

9. A time-based data collection system for collecting event data about neural network computations for a neural network having multiple neural network layers, the system comprising: One or more processing devices; and One or more non-transitory machine-readable storage devices for storing instructions, said instructions being executable by said one or more processing devices to cause operations to be performed, said operations including: Execute program code to perform neural network computations using components of a processor configured to implement the neural network; The processor includes a global time counter, and each of the components of the processor includes a corresponding local time counter. Wherein, the global time counter and the corresponding local time counter each include at least one corresponding offset bit, and the at least one corresponding offset bit is used to reduce the phase change between one or more of the corresponding local time counters and the global time counter; The program code includes a first time parameter as a triggering condition to trigger a tracking event across two or more components of the processor during the execution of the neural network computation. The following is used to synchronize the corresponding current time value indicated by the corresponding local time counter based on the current time value indicated by the global time counter: at least one corresponding offset bit of each corresponding local time counter in two or more of the components of the processor; By comparing the corresponding current time value that is synchronized with the first time parameter in the program code, it is determined whether the triggering condition for triggering the tracking event is met; In response to determining that the triggering condition is met, the tracking event is triggered to generate event data for two or more components in the components of the processor; The event data is synchronized based on the corresponding current time value of the global time counter; and The event data is provided to a host used to analyze the program code during the execution of the neural network computation.

10. The system according to claim 9, wherein, The operation includes: The event data is stored in one or more of a plurality of trace buffers set in the first component of the processor.

11. The system according to claim 9, wherein, Each of the global time counter and the corresponding local time counter includes a fixed-size binary data, which comprises a 60-bit counter and a 4-bit offset. The program code also includes a second time parameter; The second time parameter indicates a time later than the first time parameter; The triggering conditions include a predefined time window associated with the global time counter; and The predefined time window includes a start time for triggering the tracking event based on the first time parameter, and an end time for stopping the tracking event based on the second time parameter.

12. The system according to claim 9, wherein: Executing the program code to perform the neural network computation includes executing a sequence of operations to process vector elements of the inference workload using the neural network.

13. The system according to claim 9, wherein, Synchronizing the corresponding current time value indicated by the corresponding local time counter based on the current time value indicated by the global time counter further includes: For each of the respective local time counters, calculate the difference between the corresponding current time value indicated by the local time counter and the current time value indicated by the global time counter; and Determine whether the difference satisfies the minimum value within the sampling period.

14. The system according to claim 12, wherein: The subset of the event data describes multiple memory access operations, which are executed by the processor to perform the neural network computation. and The operation sequence includes moving the vector element from the first memory of the first component to the second memory of the second component of the processor.

15. The system according to claim 14, wherein, The event data that triggers the tracing event to generate the event data of two or more components in the processor includes: During the process of moving the vector elements from the first memory of the first component to the second memory of the second component, the subset of event data synchronized between the first component and the second component is generated.

16. The system according to claim 9, wherein, The operation includes: In response to compiling the program code for execution at the processor, the first timing parameter is embedded by the host; and The host loads the compiled program code, which includes the first time parameter embedded in the compiled program code.

17. One or more non-transitory machine-readable storage devices for storing instructions executable by one or more processing devices to perform operations for collecting event data about neural network computations for a neural network having multiple neural network layers, the operations including: Execute program code to perform neural network computations using components of a processor configured to implement the neural network; The processor includes a global time counter, and each of the components of the processor includes a corresponding local time counter. Wherein, the global time counter and the corresponding local time counter each include at least one corresponding offset bit, and the at least one corresponding offset bit is used to reduce the phase change between one or more of the corresponding local time counters and the global time counter; The program code includes a first time parameter as a triggering condition to trigger a tracking event across two or more components of the processor during the execution of the neural network computation. The following is used to synchronize the corresponding current time value indicated by the corresponding local time counter based on the current time value indicated by the global time counter: at least one corresponding offset bit of each corresponding local time counter in two or more of the components of the processor; By comparing the corresponding current time value that is synchronized with the first time parameter in the program code, it is determined whether the triggering condition for triggering the tracking event is met; In response to determining that the triggering condition is met, the tracking event is triggered to generate event data for two or more components in the components of the processor; The event data is synchronized based on the corresponding current time value of the global time counter; and The event data is provided to a host used to analyze the program code during the execution of the neural network computation.

18. The machine-readable storage device according to claim 17, wherein, The operation includes: The event data is stored in one or more of a plurality of trace buffers set in the first component of the processor.

19. The machine-readable storage device according to claim 17, wherein, Each of the global time counter and the corresponding local time counter includes binary data of a fixed size, the corresponding fixed-size binary data including a 60-bit counter and a 4-bit offset; The program code also includes a second time parameter; Wherein, the second time parameter indicates a time later than the first time parameter; and The triggering condition includes a predefined time window associated with the global time counter, wherein the predefined time window includes a start time for triggering the tracking event based on the first time parameter and an end time for stopping the tracking event based on the second time parameter.

20. The machine-readable storage device of claim 17, wherein: Executing the program code to perform the neural network computation includes executing a sequence of operations to process vector elements of the inference workload using the neural network.

21. The machine-readable storage device according to claim 17, wherein, Synchronizing the corresponding current time value indicated by the corresponding local time counter based on the current time value indicated by the global time counter further includes: For each of the respective local time counters, calculate the difference between the corresponding current time value indicated by the local time counter and the current time value indicated by the global time counter; and Determine whether the difference satisfies the minimum value within the sampling period.

Images