Distributed hardware tracing

Abstract

This disclosure relates to distributed hardware tracing. A computer-implemented method executed by one or more processors includes: monitoring the execution of program code executed by a first processor component; and monitoring the execution of program code executed by a second processor component. The computing system stores data identifying hardware events in a memory buffer. The stored events occur across processor units including at least first and second processor components. Each hardware event includes an event timestamp and metadata characterizing the event. The system generates a data structure identifying the hardware events. The data structure arranges the events in a time-ordered sequence and associates the events with at least the first or second processor component. The system stores the data structure in a repository on a host device and uses the data structure to analyze the performance of program code executed by the first or second processor component.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201711294123.0, filed on December 8, 2017.

[0003] Cross-reference to related applications

[0004] This application relates to U.S. Patent Application No. 15 / 472,932, filed March 29, 2017, entitled "Synchronous Hardware Event Collection," Attorney General No. 16113-8129001. The entire disclosure of U.S. Patent Application No. 15 / 472,932 is hereby incorporated herein by reference. Technical Field

[0005] This manual relates to the execution of analysis program code. Background Technology

[0006] Effective performance analysis of distributed software executing within distributed hardware components can be a complex task. Distributed hardware components can be corresponding processor cores of two or more central processing units (CPUs) (or graphics processing units (GPUs)) that cooperate and interact to execute large software programs or portions of program code.

[0007] From a hardware perspective (e.g., within a CPU or GPU), there are typically two types of information or features that can be applied to performance analysis: 1) hardware performance counters; and 2) hardware event traces. Summary of the Invention

[0008] Generally, one aspect of the subject matter described in this specification can be embodied as a computer-implemented method executed by one or more processors. The method includes: monitoring the execution of program code performed by a first processor component, the first processor component being configured to execute at least a first portion of the program code; and monitoring the execution of program code performed by a second processor component, the second processor component being configured to execute at least a second portion of the program code.

[0009] The method further includes: storing data identifying one or more hardware events occurring across processor units including a first processor component and a second processor component in at least one memory buffer by a computing system. Each hardware event represents at least one of data communication associated with a memory access operation of program code, a published instruction of program code, or an executed instruction of program code. The data identifying each of the one or more hardware events includes a hardware event timestamp and metadata characterizing the hardware event. The method includes: generating a data structure by the computing system that identifies the one or more hardware events, the data structure being configured to arrange the one or more hardware events in a time-ordered sequence of events associated with at least the first processor component and the second processor component.

[0010] The method further includes: storing the generated data structures in a repository on a host device by the computing system for use in analyzing the performance of program code executed by at least a first processor component or a second processor component.

[0011] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, the method further includes: detecting, by the computing system, a triggering function associated with a portion of program code executed by at least one of a first processor component or a second processor component; and, in response to detecting the triggering function, initiating at least one trace event by the computing system, such that data associated with one or more hardware events is stored in at least one memory buffer.

[0012] In some implementations, the triggering function corresponds to at least one of a specific sequence step in the program code or a specific time parameter indicated by a global clock used by the processor unit; and initiating at least one trace event includes determining that a trace bit is set to a specific value, the at least one trace event being associated with a memory access operation including a plurality of intermediate operations occurring across the processor unit, and in response to determining that the trace bit is set to a specific value, data associated with the plurality of intermediate operations being stored in one or more memory buffers.

[0013] In some implementations, storing data identifying one or more hardware events further includes storing a first subset of the data identifying the hardware events among the one or more hardware events in a first memory buffer of the first processor component. This storage occurs in response to the first processor component executing hardware trace instructions associated with at least a first portion of the program code.

[0014] In some implementations, storing data identifying one or more hardware events further includes storing a second subset of the data identifying the hardware events among the one or more hardware events in a second memory buffer of the second processor component. This storage occurs in response to the second processor component executing hardware tracing instructions associated with at least a second portion of the program code.

[0015] In some implementations, generating the data structure further includes: comparing, by a computing system, at least a hardware event timestamp of a corresponding event in a first subset of the data identifying the hardware event with at least a hardware event timestamp of a corresponding event in a second subset of the data identifying the hardware event; and providing an associated set of hardware events to be presented in the data structure, partly based on the comparison between the corresponding events in the first subset and the corresponding events in the second subset.

[0016] In some implementations, the generated data structure identifies at least one parameter indicating a latency attribute of a particular hardware event, the latency attribute indicating at least the duration of the particular hardware event. In some implementations, at least one processor of the computing system is a multi-core, multi-node processor having one or more processor components, and the one or more hardware events partially correspond to data transfers occurring between at least a first processor component of a first node and a second processor component of a second node.

[0017] In some implementations, the first processor component and the second processor component are one of the following: a processor, processor core, memory access engine, or hardware feature of a computing system, and one or more hardware events partially correspond to the movement of data packets between a source and a destination; and the metadata characterizing the hardware events corresponds to at least one of a source memory address, a destination memory address, a unique trace identifier number, or a size parameter associated with direct memory access (DMA) tracking.

[0018] In some implementations, a specific trace (ID) number is associated with multiple hardware events occurring across processor units, wherein the multiple hardware events correspond to a specific memory access operation, and the specific trace ID number is used to associate one or more of the multiple hardware events and to determine the latency attributes of the memory access operation based on the association.

[0019] Another aspect of the subject matter described in this specification can be embodied as a distributed hardware tracing system comprising: one or more processors, including one or more processor cores; one or more machine-readable storage units for storing instructions executable by the one or more processors to perform operations, including: monitoring the execution of program code performed by a first processor component, the first processor component being configured to execute at least a first portion of the program code; and monitoring the execution of program code performed by a second processor component, the second processor component being configured to execute at least a second portion of the program code.

[0020] The method further includes: storing data identifying one or more hardware events occurring across processor units including a first processor component and a second processor component in at least one memory buffer by a computing system. Each hardware event represents at least one of data communication associated with a memory access operation of program code, a published instruction of program code, or an executed instruction of program code. The data identifying each of the one or more hardware events includes a hardware event timestamp and metadata characterizing the hardware event. The method includes: generating a data structure by the computing system that identifies the one or more hardware events, the data structure being configured to arrange the one or more hardware events in a time-ordered sequence of events associated with at least the first processor component and the second processor component.

[0021] The method further includes: storing the generated data structures in a repository on a host device by the computing system for use in analyzing the performance of program code executed by at least a first processor component or a second processor component.

[0022] Other embodiments of this and other aspects include corresponding systems, apparatuses, and computer programs configured to perform actions of methods encoded on computer storage devices. A system of one or more computers can be configured, in operation, to cause the system to perform actions by means of software, firmware, hardware, or a combination thereof installed on the system. One or more computer programs can be configured, by means of instructions, to cause an apparatus to perform actions when executed by a data processing device.

[0023] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. The hardware tracing system is capable of efficiently correlating hardware events occurring during the execution of distributed software programs by distributed processing units including multi-node, multi-core processors. The hardware tracing system further includes mechanisms for collecting and correlating hardware event / tracing data across multiple cross-node configurations.

[0024] Hardware tracing systems improve computational efficiency by using dynamic triggering conditions executed via hardware knobs / features. Furthermore, by utilizing event descriptors such as unique trace identifiers, event timestamps, event source addresses, and event destination addresses, hardware events can be sequentially ordered by time. These descriptors help software programmers and processor design engineers efficiently debug and analyze potential software and hardware performance issues that may arise during source code execution.

[0025] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0026] Figure 1 The figure shows a block diagram of an example computer system used for distributed hardware tracing.

[0027] Figure 2 The diagram illustrates a block diagram of the tracing chain and corresponding nodes of an example computing system used for distributed hardware tracing.

[0028] Figure 3 The diagram illustrates the design architecture and data structure of an example trace multiplexer.

[0029] Figure 4 It is a block diagram indicating the tracing activities performed by the example computing system used for distributed hardware tracing of direct memory access tracing events.

[0030] Figure 5 This is a flowchart of an example process for distributed hardware tracing.

[0031] Similar icon numbers and markings in the figures indicate similar elements. Detailed Implementation

[0032] The subject matter described in this specification generally relates to distributed hardware tracing. Specifically, a computing system monitors the execution of program code executed by one or more processor cores. For example, the computing system is capable of monitoring the execution of program code executed by a first processor core and the execution of program code executed by at least a second processor core. The computing system stores data identifying one or more hardware events in a memory buffer. The stored data identifying the events corresponds to events occurring in a distributed processor unit encompassing at least the first and second processor cores.

[0033] For each hardware event, the stored data includes an event timestamp and metadata characterizing the hardware event. The system generates a data structure to identify the hardware events. The data structure arranges the events in chronological order and associates each event with at least one first or second processor core. The system stores the data structure in a repository on the host device and uses the data structure to analyze the performance of program code executed by the first or second processor core.

[0034] Figure 1 The diagram illustrates a block diagram of an example computing system 100 for distributed hardware tracing. As used herein, distributed hardware system tracing corresponds to the storage of data identifying events occurring within components and sub-components of an example processor microchip. Furthermore, as used herein, a distributed hardware system (or tracing system) corresponds to a collection of processor microchips or processing units that cooperate to execute portions of software / program code configured for distributed execution within the processor microchips or distributed processing units.

[0035] System 100 can be a distributed processing system having one or more processors or processing units that execute software programs in a distributed manner, i.e., by executing program code on different parts or portions of the program code on different processing units of system 100. A processing unit can include two or more processors, processor microchips, or processing units, for example, at least a first processing unit and a second processing unit.

[0036] In some implementations, two or more processing units can be distributed processing units when a first processing unit receives and executes a first part of the program code of the distributed software program, and when a second processing unit receives and executes a second part of the program code of the same distributed software program.

[0037] In some implementations, different processor chips of system 100 can form corresponding nodes in a distributed hardware system. In alternative implementations, a single processor chip can include one or more processor cores and hardware features, each of which can form a corresponding node of the processor chip.

[0038] For example, in a central processing unit (CPU) context, the processor chip may include at least two nodes, and each node may be a corresponding core of the CPU. Alternatively, in a graphics processing unit (GPU) context, the processor chip may include at least two nodes, and each node may be a corresponding streaming multiprocessor of the GPU. The computing system 100 may include multiple processor components. In some embodiments, a processor component may be at least one of a processor chip, a processor core, a memory access engine, or at least one hardware component of the overall computing system 100.

[0039] In some instances, a processor component, such as a processor core, can be a fixed-function component configured to perform at least one specific operation based on at least one instruction issued from the execution program code. In other instances, a processor component, such as a memory access engine (MAE), can be configured to execute program code at a lower level of detail or granularity than program code executed by other processor components of system 100.

[0040] For example, program code executed by the processor core can cause the generation of an MAE descriptor and its transmission / sending to the MAE. Upon receiving the descriptor, the MAE can perform data transfer operations based on it. In some implementations, data transfers performed by the MAE can, for example, involve moving data to and from certain components of system 100 via some data path or interface component of the system, or issuing a data request to an example configuration bus of system 100.

[0041] In some implementations, each tensor node of the exemplary processor chip of system 100 can have at least two "front ends," which can be hardware blocks / features that process program instructions. As detailed below, a first front end can correspond to a first processor core 104, and a second front end can correspond to a second processor core 106. Therefore, the first and second processor cores can also be described herein as a first front end 104 and a second front end 106.

[0042] As used in this specification, the trace chain can be a specific physical data communication bus on which trace entries are placed for transmission to the example chip manager within system 100. The received trace entries can be data words / structures comprising multiple bytes and multiple binary values ​​or bits. Therefore, the descriptor "word" indicates a fixed-size piece of binary data that can be processed as a unit by the hardware devices of the example processor core.

[0043] In some implementations, the processor chip of the distributed hardware tracing system is a multi-core processor (i.e., having multiple cores), each executing a portion of program code within its respective core. In some implementations, the portion of program code can correspond to vectorized computations for an inference workload exemplifying a multilayer neural network. In alternative implementations, the portion of program code can generally correspond to software modules associated with a conventional programming language.

[0044] The computing system 100 generally includes a node manager 102, a first processor core (FPC) 104, a second processor core (SPC) 106, a node organization structure (NF) 110, a data router 112, and a host interface block (HIB) 114. In some embodiments, the system 100 may include a memory multiplexer 108 configured to perform signal switching, multiplexing, and demultiplexing functions. The system 100 further includes a tensor core 116, which includes the FPC 104 disposed therein. The tensor core 116 may be an example computing device configured to perform vectorized computations on a multidimensional data array. The tensor core 116 may include a vector processing unit (VPU) 118 that interacts with a matrix unit (MXU) 120, a transpose unit (XU) 122, and a reduction and permutation unit (RPU) 124. In some embodiments, the computing system 100 may include one or more execution units of a conventional CPU or GPU, such as a load / store unit, an arithmetic logic unit (ALU), and a vector unit.

[0045] The components of System 100 generally include a large set of hardware performance counters and support hardware that facilitates tracing activities within the components. As described in more detail below, program code executed by the corresponding processor core of System 100 can include embedded trigger conditions for simultaneously enabling multiple performance counters during code execution. Generally, detected trigger conditions prompt the generation of trace data for one or more tracing events. The trace data can correspond to incremental parameter counts stored in the counters and capable of being analyzed to identify performance characteristics of the program code. Data for corresponding tracing events can be stored in a sample storage medium (e.g., a hardware buffer) and can include timestamps generated in response to the detection of trigger conditions.

[0046] Additionally, tracking data can be generated for a variety of events occurring within the hardware components of system 100. Example events can include inter-node and cross-node communication operations, such as direct memory access (DMA) operations and synchronization flag updates (both described in detail below). In some embodiments, system 100 can include a global synchronization timestamp counter, commonly referred to as a global time counter (“GTC”). In other embodiments, system 100 can include other types of global clocks, such as a Lamport clock.

[0047] GTC can be used to precisely correlate program code execution with the performance of software / program code executed in a distributed processing environment. Furthermore, in part related to GTC, in some implementations, system 100 can include one or more triggering mechanisms used by distributed software programs to start and stop data tracing in a highly coordinated manner within the distributed system.

[0048] In some implementations, host system 126 compiles program code that can include embedded operands that, upon detection, trigger the capture and storage of trace data associated with a hardware event. In some implementations, host system 126 provides the compiled program code to one or more processor chips of system 100. In alternative implementations, the program code can be compiled (with embedded triggering) using an example external compiler and loaded into one or more processor chips of system 100. In some instances, the compiler can set one or more trace bits (described below) associated with certain trigger conditions in portions of the embedded software instructions. The compiled program code can be a distributed software program executed by one or more components of system 100.

[0049] Host system 126 may include a monitoring engine 128 configured to monitor the execution of program code by one or more components of system 100. In some embodiments, monitoring engine 128 enables host system 126 to monitor the execution of program code executed by at least FPC 104 and SPC 106. For example, during code execution, host system 126 may monitor the performance of the executed code via monitoring engine 128, at least by receiving a periodic timeline of hardware events based on generated trace data. Although a single block is shown for host system 126, in some embodiments, system 126 may include multiple hosts (or host subsystems) associated with multiple processor chips or chip cores of system 100.

[0050] In other implementations, cross-node communication involving at least three processor cores can cause host system 126 to monitor one or more intermediate "hops" of data traffic as data traffic traverses the communication path between FPC 104 and the example third processor core / node. For example, FPC 104 and the third processor core may be the only cores executing program code for a given time period. Therefore, when data is transferred from FPC 104 to the third processor core, the data transfer from FPC 104 to the third processor core can generate trace data for intermediate hops at SPC 106. In other words, during data routing in system 100, data from the first processor chip to the third processor chip may need to traverse the second processor chip, so the execution of data routing operations may result in the generation of trace entries for routing activity in the second chip.

[0051] After the compiled program code is executed, the components of System 100 can interact to generate a timeline of hardware events occurring in the distributed computer system. Hardware events can include intra-node and cross-node communication events. See below for reference. Figure 2The example nodes of the distributed hardware system and their associated communications are described in more detail. In some implementations, a data structure is generated that identifies a set of hardware events for at least one hardware event timeline. The timeline is capable of reconstructing events that occurred in the distributed system. In some implementations, event reconstruction can include correcting event ordering based on analysis of timestamps generated during the occurrence of a particular event.

[0052] Generally, an example distributed hardware tracing system can include the components of system 100 described above, as well as at least one host controller associated with host system 126. The performance or debugging of data obtained from a distributed tracing system can be beneficial when event data is correlated, for example, in a chronological or ordered manner. In some implementations, data correlation can occur when multiple stored hardware events corresponding to connected software modules are stored and then ordered for structured analysis of host system 126. For implementations including multiple host systems, the correlation of data obtained via different hosts can be performed, for example, by the host controller.

[0053] In some embodiments, FPC 104 and SPC 106 are both different cores of a single multi-core processor chip; in other embodiments, FPC 104 and SPC 106 are cores of different multi-core processor chips. As indicated above, system 100 may include a distributed processor unit having at least FPC 104 and SPC 106. In some embodiments, the distributed processor unit of system 100 may include one or more hardware or software components configured to execute at least a portion of a larger distributed software program or program code.

[0054] Data router 112 is an inter-chip interconnect (ICI) that provides data communication paths between components of system 100. Specifically, router 112 provides communication coupling or connection between FPC 104 and SPC 106, and between corresponding components associated with cores 104 and 106. Node organization 110 interacts with data router 112 to move data packets within the distributed hardware components and sub-components of system 100.

[0055] Node manager 102 is a high-level device that manages the low-level node functions in a multi-node processor chip. As detailed below, one or more nodes of the processor chip can include a chip manager controlled by node manager 102 to manage and store hardware event data in a local entry log. Memory multiplexer 108 is a multiplexing device capable of performing switching, multiplexing, and demultiplexing operations on data signals provided to or received from an external high-bandwidth memory (HBM).

[0056] In some implementations, when multiplexer 108 switches between FPC 104 and SPC 106, example trace entries (described below) can be generated by multiplexer 108. Memory multiplexer 108 can potentially affect the performance of specific processor cores 104, 106 that cannot access multiplexer 108. Therefore, trace entry data generated by multiplexer 108 can help understand latency spikes in certain system activities associated with the corresponding cores 104, 106. In some implementations, hardware event data initiated within multiplexer 108 (e.g., trace points described below) can be grouped together with event data from node organization structure 110 on the example hardware event timeline. Event grouping can occur when certain tracing activities cause event data from multiple hardware components to be stored in an example hardware buffer (e.g., trace entry log 218 described below).

[0057] In system 100, the performance analysis hardware includes FPC 104, SPC 106, multiplexer 108, node organization structure 110, data router 112, and HIB 114. Each of these hardware components or units includes hardware performance counters and hardware event tracking conditions and functions. In some embodiments, VPU 118, MXU 120, XU 122, and RPU 124 do not include their own dedicated performance hardware. Instead, in this embodiment, FPC 104 can be configured to provide the necessary counters for VPU 118, MXU 120, XU 122, and RPU 124.

[0058] The VPU 118 can include an internal design architecture that supports local high-bandwidth data processing and arithmetic operations associated with vector elements of the example matrix vector processor. The MXU 120 is a matrix multiplication unit configured to perform up to 128×128 matrix multiplications, for example, on a vector dataset of multiplicands.

[0059] XU 122 is a transpose unit configured to perform up to 128×128 matrix transpose operations on vector data associated with matrix multiplication, for example. RPU 124 can include sigma units and permutation units. The sigma unit performs sequential reduction on the vector data associated with matrix multiplication. Reduction can include summation and various types of comparison operations. The permutation unit can completely permutate or copy all elements of the vector data associated with matrix multiplication.

[0060] In some implementations, program code executed by components of system 100 can represent machine learning, neural network inference computation, and / or one or more direct memory access functions. Components of system 100 can be configured to execute one or more software programs, including instructions that cause the system's processing units or devices to perform one or more functions. The term "component" is intended to include any data processing or storage device, such as a control status register, or any other device capable of processing and storing data.

[0061] System 100 may generally include multiple processing units or devices, which may include one or more processors (e.g., microprocessors or central processing units (CPUs)), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or combinations of different processors. In alternative embodiments, system 100 may each include additional computing resources / devices (e.g., cloud-based servers) that provide additional processing options for performing calculations related to the hardware tracing functions described herein.

[0062] The processing unit or device may further include one or more memory cells or repositories (e.g., registers / counters). In some embodiments, the processing unit executes programmed instructions stored in memory on the device of system 100 for one or more functions described herein. The memory cells / repositories may include one or more non-transitory machine-readable storage media. Non-transitory machine-readable storage media may include solid-state memory, magnetic disks and optical disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information.

[0063] Figure 2 The diagram illustrates an example tracing chain and corresponding example nodes 200, 201 for distributed hardware tracing performed by system 100. In some embodiments, nodes 200, 201 of system 100 can be different nodes within a single multi-core processor. In other embodiments, node 200 can be a first node in a first multi-core processor chip, and node 201 can be a second node in a second multi-core processor chip.

[0064] Although Figure 2The implementation depicts two nodes, but in an alternative implementation, system 100 can include multiple nodes. For implementations involving multiple nodes, cross-node data transfers can generate trace data at intermediate hops along an example data path traversing multiple nodes. For example, an intermediate hop can correspond to data transfers through different nodes in a specific data transfer path. In some instances, trace data associated with ICI tracing / hardware events can be generated for one or more intermediate hops occurring during cross-node data transfers through one or more nodes.

[0065] In some implementations, node 0 and node 1 are tensor nodes used for vectorized computations associated with portions of the program code used for inference workloads. As used in this specification, a tensor is a multidimensional geometric object, and example multidimensional geometric objects include matrices and data arrays.

[0066] like Figure 2 As shown in the implementation, node 200 includes a tracing chain 203 that interacts with at least a subset of the components of system 100. Similarly, node 201 includes a tracing chain 205 that interacts with at least a subset of the components of system 100. In some implementations, nodes 200 and 201 are example nodes of the same subset of components, while in other implementations, nodes 200 and 201 are corresponding nodes of different subsets of components. The data router / ICI 112 includes a tracing chain 207, which typically converges with tracing chains 203 and 205 to provide tracing data to the chip manager 216.

[0067] exist Figure 2 In some implementations, nodes 200 and 201 can each include at least a corresponding subset of components having FPC 104, SPC 106, node organization structure 110, and HIB 114. Each component of nodes 200 and 201 includes one or more tracking multiplexers configured to group tracking points (described below) generated by a particular component of the node. FPC 104 includes tracking multiplexer 204, node organization structure 110 includes tracking multiplexers 210a / b, SPC 106 includes tracking multiplexers 206a / b / c / d, HIB 214 includes tracking multiplexer 214, and ICI 212 includes tracking multiplexer 212. In some implementations, the tracking control register for each tracking multiplexer allows enabling and disabling individual tracking points. In some instances, for one or more tracking multiplexers, their corresponding tracking control registers can include individual enable bits and broader tracking multiplexer control.

[0068] Generally, the trace control register can be a regular control status register (CSR) that receives and stores trace instruction data. Regarding broader trace multiplexer control, in some implementations, trace can be enabled and disabled based on CSR writes performed by system 100. In some implementations, trace is dynamically started and stopped by system 100 based on the value of the global time counter (GTC), the value of the example trace flag register in FPC 104 (or core 116), or the value of the flag in SPC 106.

[0069] The relevant U.S. Patent Application No. 15 / 472,932, filed March 29, 2017, entitled "Synchronizing Hardware Event Collection" and Agent's Case No. 16113-8129001, describes a system and method for dynamically starting and stopping tracking activities, as well as related details and descriptions. The entire disclosure of U.S. Patent Application No. 15 / 472,932 is hereby incorporated herein by reference in its entirety.

[0070] In some implementations, for core 116, FPC 104 can use trace control parameters to define a trace window associated with event activity occurring within core 116. The trace control parameters allow the trace window to be defined based on the lower and upper bounds of the GTC and the lower and upper bounds of the trace flag register.

[0071] In some implementations, system 100 may include features that can reduce the number of generated trace entries, such as trace event filtering features. For example, FPC 104 and SPC 106 may each include filtering features that limit the rate at which each core sets trace bits in an example-generated trace descriptor (described below). HIB 114 may include similar filtering features, such as an example DMA rate limiter that limits the trace bits associated with capturing certain DMA trace events. Furthermore, HIB 114 may include controls (e.g., via enable bits) for limiting which queues initiate DMA trace entries.

[0072] In some implementations, the descriptor used for DMA operations can have trace bits set by a sample compiler of the host system 126. When the trace bits are set, a sample trace event is performed using hardware features / knobs that determine and generate trace data. In some instances, the final trace bits in DMA can be a logical OR operation between trace bits statically inserted by the compiler and trace bits dynamically determined by specific hardware components. Therefore, in some instances, in addition to filtering, the trace bits generated by the compiler can also provide a mechanism to reduce the total amount of trace data generated.

[0073] For example, the compiler of host system 126 can decide to set the trace bits for only one or more remote DMA operations (e.g., DMA across at least two nodes) and clear the trace bits for one or more local DMA operations (e.g., DMA within a specific tensor node such as node 200). In this way, by limiting the trace activity to cross-node (i.e., remote) DMA operations, rather than trace activities that include both cross-node and local DMA operations, the amount of trace data generated can be reduced.

[0074] In some implementations, at least one trace event initiated by system 100 can be associated with a memory access operation that includes multiple intermediate operations occurring across system 100. The descriptor for the memory access operation (e.g., a MAE descriptor) can include trace bits that cause data associated with the multiple intermediate operations to be stored in one or more memory buffers. Therefore, as packets traverse system 100, the trace bits can be used to "mark" intermediate memory operations and generate multiple trace events in the middle of DMA operations.

[0075] In some implementations, ICI 112 may include a set of enable bits and a set of packet filters that provide control over each ingress and egress port of a specific component of nodes 200, 201. These enable bits and packet filters allow ICI 112 to enable and disable tracking points associated with specific components of nodes 200, 201. In addition to enabling and disabling tracking points, ICI 112 can be configured to filter tracking data based on event source, event destination, and tracking event packet type.

[0076] In some implementations, in addition to using step markers, GTC, or trace markers, each trace control register of processor cores 104, 106, and HIB 114 can also include a "per person" trace mode. This "per person" trace mode allows trace across the entire processor chip to be controlled by trace multiplexer 204 or trace multiplexer 206a. In per person trace mode, trace multiplexers 204 and 206a can send "in-window" trace control signals specifying whether a particular trace multiplexer (multiplexer 204 or multiplexer 206a) is within a trace window.

[0077] The in-window tracking control signal can be broadcast or universally sent to all other tracking multiplexers, for example, within the processor chip or across multiple processor chips. Broadcasting to other tracking multiplexers while multiplexer 204 or multiplexer 206a is performing tracking activity can prompt the activation of all tracking. In some embodiments, the tracking multiplexers associated with processor cores 104, 106, and HIB 114 each include a tracking window control register that specifies when and / or how the "everyone tracks" control signal is generated.

[0078] In some implementations, tracking activity in tracking multiplexers 210a / b and tracking multiplexer 212 is generally enabled based on whether the tracking bit is set in the data word used for DMA operations or in the control message traversing ICI / data router 112. The DMA operation or control message can be a fixed-size binary data structure that may have tracking bits set within the binary data packet based on certain conditions or software requirements.

[0079] For example, when a DMA operation is initiated in an FPC 104 (or SPC 106) with a trace-type DMA instruction and the initiator (processor core 104 or 106) is within the trace window, the trace bit will be set in that particular DMA. In another example, for FPC 104, if FPC 104 is within the trace window and a trace point that causes trace data to be stored is enabled, a control message for writing data to another component within system 100 will complete the trace bit setting.

[0080] In some implementations, zero-length DMA operations provide examples of a broader range of DMA implementations within system 100. For example, some DMA operations are capable of generating non-DMA activities within system 100. It is also possible to track the execution of non-DMA activities (e.g., generating trace data) as if the non-DMA activity were a DMA operation (e.g., a DMA activity that includes a non-zero-length operation). For example, a DMA operation initiated at a source location but without any data to be sent or transmitted (e.g., zero-length) can instead send a control message to the destination location. The control message will indicate that there is no data to be received or processed at the destination, and the control message itself will be tracked by system 100 because the non-zero-length DMA operation will be tracked.

[0081] In some instances, for SPC 106, a zero-length DMA operation can generate a control message, and the track bit associated with the message can only be set after the DMA has completed track bit setting, i.e., when the control message does not have zero length. Generally, if HIB 114 is within the track window, a DMA operation initiated from host system 126 will complete the track bit setting.

[0082] exist Figure 2In this implementation, tracing chain 203 receives tracing entry data for a subset of components aligned to node 0, while tracing chain 205 receives tracing entry data for a subset of components aligned to node 1. Each tracing chain 203, 205, and 207 is a different data communication path used by the corresponding nodes 200, 201, and ICI 112 to provide tracing entry data to the example tracing entry data log 218 of the chip manager 216. Therefore, the endpoints of tracing chains 203, 205, and 207 are the chip manager 216, where tracing events can be stored in the example memory unit.

[0083] In some implementations, at least one memory cell of the chip manager 216 can be 128 bits wide and have a memory depth of at least 20,000 tracking entries. In alternative implementations, at least one memory cell can have a larger or smaller bit width and a memory depth capable of storing more or fewer entries.

[0084] In some implementations, chip manager 216 may include at least one processing device that executes instructions to manage received trace entry data. For example, chip manager 216 may execute instructions to scan / analyze timestamp data of corresponding hardware events in trace data received via trace chains 203, 205, 207. Based on the analysis, chip manager 216 may populate trace entry log 218 to include time-ordered sequence data that can be used to identify (or generate) hardware trace events. When the processing unit of system 100 executes an example distributed software program, hardware trace events may correspond to data packet movements occurring at the component and subcomponent levels.

[0085] In some implementations, the hardware units of system 100 can generate trace entries (and corresponding timestamps) that fill the example hardware trace buffer in an unordered manner (i.e., out of order). For example, chip manager 216 can cause multiple trace entries with generated timestamps to be inserted into entry log 218. The corresponding trace entries of the multiple inserted trace entries may not be ordered chronologically relative to each other. In this implementation, the unordered trace entries can be received by the example host buffer of host system 126. After being received by the host buffer, host system 126 can execute instructions related to performance analysis / monitoring software to scan / analyze the timestamp data of the corresponding trace entries. The executed instructions can be used to sort the trace entries and construct / generate a timeline of hardware trace events.

[0086] In some implementations, tracking entries can be deleted from the entry log 218 during a tracking session via host DMA operations. In some instances, the host system 126 may not delete a tracking entry from the tracking entry log 218 as quickly as adding it to the log. In other implementations, the entry log 218 may include a predefined memory depth. If the memory depth limit of the entry log 218 is reached, added tracking entries may be lost. To control which tracking entries are discarded, the entry log 218 can operate in a first-in-first-out (FIFO) mode or alternatively in an overwrite mode.

[0087] In some implementations, the overwrite log mode can be used by system 100 to support performance analysis associated with post-processing debugging. For example, program code can be executed during specific time periods when tracing activity is enabled and overwrite log mode is enabled. In response to a post-processing software event (e.g., a program crash) within system 100, monitoring software executed by host system 126 can analyze the data contents of an example hardware trace buffer to gain insight into hardware events that occurred prior to the program crash. As used in this specification, post-processing debugging involves the analysis or debugging of program code that has crashed or is generally not executing / operating as expected.

[0088] In FIFO mode, if the entry log 218 is full, and if the host system 126 determines to delete log entries saved within a certain time frame to save memory resources, new trace entries may not be saved to the memory cells of the chip manager 216. In overwrite mode, if the entry log 218 is full, because the host system 126 determines to delete log entries already saved within a certain time frame to save memory resources, new trace entries can overwrite the oldest trace entry stored in the entry log 218. In some embodiments, trace entries are moved to the host system 126's memory in response to a DMA operation using the processing features of HIB 114.

[0089] As used in this specification, a trace point is a generator of trace entries and associated data received by chip manager 216 and stored in trace entry log 218. In some embodiments, a multi-core, multi-node processor microchip can include three trace chains within the chip, such that a first trace chain receives trace entries from chip node 0, a second trace chain receives trace entries from chip node 1, and a third trace chain receives trace entries from the chip's ICI router.

[0090] Each tracking point has a unique tracking identifier number within its tracking chain, which is inserted into the header of the tracking entry. In some implementations, each tracking entry identifies the tracking chain from which it originates in a header indicated by one or more bytes / bits of data words. For example, each tracking entry can include a data structure with a defined field format (e.g., header, payload, etc.) that conveys information about a specific tracking event. Each field in the tracking entry corresponds to useful data that can be applied to the tracking point that generated the tracking entry.

[0091] As described above, each trace entry can be written to or stored in the memory unit of the chip manager 216 associated with the trace entry log 218. In some implementations, trace points can be individually enabled or disabled, and multiple trace points can generate trace entries of the same type but with different trace point identifiers.

[0092] In some implementations, each trace entry type may include a trace name, a trace description, and a header encoded to identify specific fields and / or sets of fields within the trace entry. The name, description, and header together provide a description of what the trace entry represents. From the perspective of chip manager 216, this description may also identify the specific trace chains 203, 205, 207 that a particular trace entry within a specific processor chip participates in. Therefore, the fields within a trace entry represent data slices (e.g., in bytes / bits) associated with the description and can be trace entry identifiers used to determine which trace point generated the specific trace entry.

[0093] In some implementations, the trace entry data associated with one or more of the stored hardware events can partially correspond to data communications occurring at least between: a) nodes 0 and 1; b) between components at least within node 0; and c) between components at least within node 1. For example, the stored hardware events can partially correspond to data communications occurring between at least one of: 1) FPC 104 of node 0 and FPC 104 of node 1; FPC 104 of node 0 and SPC 106 of node 0; 2) SPC 106 of node 1 and SPC 106 of node 1.

[0094] Figure 3 The diagram illustrates a block diagram of an example trace multiplexer design architecture 300 and an example data structure 320. The trace multiplexer design 300 generally includes a trace bus input 302, a bus arbiter 304 and a local trace point arbiter 306, a bus FIFO 308, at least one local trace event queue 310, a shared trace event FIFO 312, and a trace bus output 314.

[0095] Multiplexer design 300 corresponds to an example tracking multiplexer arranged within a component of system 100. Multiplexer design 300 can include the following functions: The bus in 302 can be associated with local tracking point data temporarily stored in bus FIFO 308 until time arbitration logic (e.g., arbitrator 304) can place the tracking data onto the example tracking chain. One or more tracking points of the component can insert tracking event data into at least one local tracking event queue 310. Arbitrator 306 provides first-level arbitration and can select events from the local tracking events stored in queue 310. The selected events are placed in a shared tracking event FIFO 312, which also serves as a storage queue.

[0096] Arbitrator 304 provides a second-level arbitration, receiving local tracking events from FIFO queue 312 and merging them into specific tracking chains 203, 205, 207 via tracking bus output 314. In some implementations, tracking entries may be pushed into local queue 310 faster than they can be merged into shared FIFO 312, or alternatively, tracking entries may be pushed into shared FIFO 312 faster than they can be merged into tracking bus 314. When these scenarios occur, the corresponding queues 310 and 312 will become full of tracking data.

[0097] In some implementations, when any queue 310 or 312 becomes full of tracking data, system 100 can be configured to discard the latest tracking entry and not store or merge it into a particular queue. In other implementations, when some queues are full (e.g., queues 310, 312), instead of discarding tracking entries, system 100 can be configured to stop the sample processing pipeline until a refilled queue has available queue space to receive entries.

[0098] For example, the processing pipelines of queues 310 and 312 can be stopped until a sufficient or threshold number of tracking entries are merged onto the tracking bus 314. The sufficient or threshold number can correspond to a specific number of merged tracking entries, creating available queue space for queues 310 and 312 to receive one or more tracking entries. Implementations that stop the processing pipelines before downstream queue space becomes available can provide higher fidelity tracking data by retaining rather than discarding certain tracking entries.

[0099] In some implementations, the local tracking queue has the same width as the tracking entry, so that each tracking entry occupies only one point in the local queue 310. However, the shared tracking FIFO queue 312 can use unique tracking entry line codes, allowing some tracking entries to occupy two positions in the shared queue 312. In some implementations, when any data in a tracking packet is discarded, the entire packet is discarded, so that partial packets do not appear in the tracking entry log 218.

[0100] Generally, a trace is a timeline of activities or hardware events associated with a specific component of system 100. Unlike performance counters (described below) which are aggregated data, a trace contains detailed event data, providing insights into hardware activity occurring during a specified trace window. The described hardware system broadly supports distributed hardware tracing, including generating trace entries, temporarily storing trace entries in a hardware-managed buffer, statically and dynamically enabling one or more trace types, and streaming trace entry data to host system 126.

[0101] In some implementations, traces can be generated for hardware events performed by components of system 100, such as generating DMA operations, executing DMA operations, issuing / executing certain instructions, or updating synchronization flags. In some instances, tracing activities can be used to trace DMA over the system or to trace instructions executed on a specific processor core.

[0102] System 100 can be configured to generate at least one data structure 320 that identifies one or more hardware events 322, 324 from a timeline of hardware events. In some embodiments, data structure 320 arranges one or more hardware events 322, 324 in a time-ordered sequence of events associated with at least FPC 104 and SPC 106. In some instances, system 100 can store data structure 320 in a repository of the host control device of host system 126. Data structure 320 can be used to evaluate the performance of program code executed by at least processor cores 104 and 106.

[0103] As illustrated in hardware event 324, in some implementations, a specific trace identifier (ID) number (e.g., trace ID "003") can be associated with multiple hardware events occurring across distributed processor units. These multiple hardware events can correspond to specific memory access operations (e.g., DMA), and the specific trace ID number is used to associate one or more hardware events.

[0104] For example, as indicated by event 324, a single trace ID for a DMA operation can include multiple time steps corresponding to multiple different points in the DMA. In some instances, trace ID “003” can have an “issued” event, an “executed” event, and a “completed” event identified as being spaced a time interval from each other. Therefore, in this respect, trace IDs can be further used to determine the latency attributes of memory access operations based on correlation and according to time steps.

[0105] In some implementations, generating data structure 320 may include, for example, comparing the event timestamps of corresponding events in a first subset of hardware events with the event timestamps of corresponding events in a second subset of hardware events. Generating data structure 320 may further include, in part, providing an associated set of hardware events for representation in the data structure based on the comparison between the first and second subsets of events.

[0106] like Figure 3 As shown, data structure 320 can identify at least one parameter indicating a delay attribute of a specific hardware event 322, 324. The delay attribute can at least indicate the duration of the specific hardware event. In some embodiments, data structure 320 is generated by software instructions executed by a control device of host system 126. In some instances, structure 320 can be generated in response to the control device storing tracking entry data to a memory disk / cell of host system 126.

[0107] Figure 4 This is a block diagram 400 indicating an example tracing activity of a Direct Memory Access (DMA) tracing event performed by system 100. For DMA tracing, data from an example DMA operation initiated from a first processor node to a second processor node can travel through ICI 112 and intermediate hops of the ICI / router can be generated along the data path. As the DMA operation traverses ICI 112, the DMA operation will generate tracing entries at each node within the processor chip and along each hop. Information from each of these generated tracing entries is captured to reconstruct the time progression of the DMA operation along the nodes and hops.

[0108] Example DMA operations can be performed with Figure 4 The process steps depicted in the implementation are related. For this operation, local DMA transfers data from virtual memory 402 (vmem 402) associated with at least one of processor cores 104, 106 to HBM 108. The numbers depicted in FIG400 correspond to the steps in Table 404 and generally represent activities in or initiated by node organization structure 110.

[0109] Table 404 broadly describes the associated trace points. For this DMA, the example operation will generate six trace entries. Step one includes an initial DMA request from the processor core to node organization structure 110, which generates one trace point in the node organization structure. Step two includes a read command, in which node organization structure 110 requests data transfer from the processor core, which generates another trace point in node organization structure 110. When vmem 402 completes the read from node organization structure 110, the example operation does not have the trace entry of step three.

[0110] Step four includes node organization structure 110 performing a read resource update to cause a synchronization flag update in the processor core, which generates a trace point in the processor core. Step five includes a write command, in which node organization structure 110 notifies memory multiplexer 108 of the upcoming data to be written to HBM. A trace point is generated in node organization structure 110 via the notification of the write command, and in step six, the completion of writing to HBM also generates a trace point in node organization structure 110. In step seven, node organization structure 110 performs a write resource update to cause a synchronization flag update in the processor core, which generates a trace point in the processor core (e.g., FPC 104). In addition to the write resource update, node organization structure 110 can perform an acknowledgment update (“ack update”), in which the data completion of the DMA operation is signaled back to the processor core. The ack update can generate a trace entry similar to the trace entry generated by the write resource update.

[0111] In another example DMA operation, a first trace entry is generated when a DMA command is issued in the node organization structure 110 of the originating node. Additional trace entries can be generated in the node organization structure 110 to capture the time for reading data from the DMA and writing data to the output queue. In some implementations, the node organization structure 110 can divide the DMA data into smaller data blocks. For data divided into smaller blocks, read and write trace entries can be generated for the first and last blocks. Optionally, all data blocks except the first and last can be configured to generate trace entries.

[0112] For remote / non-local DMA operations that may require ICI hops, the first and last data blocks can generate additional trace entries at the entry and exit points of each intermediate hop along the ICI / router 112. When the DMA data arrives at the destination node, a trace entry similar to the entry in the previous node organization structure 110 is generated at the destination node (e.g., reading / writing the first and last data blocks). In some implementations, the final step of the DMA operation can include execution instructions associated with the DMA, causing a synchronization flag to be updated at the destination node. When the synchronization flag is updated, a trace entry indicating the completion of the DMA operation can be generated.

[0113] In some implementations, when each component is in tracing mode, DMA tracing is initiated by FPC 104, SPC 106, or HIB 114 to enable the execution of tracing points. Components of system 100 can enter tracing mode based on global control in FPC 104 or SPC 106 via a triggering mechanism. Tracing points are triggered in response to a specific action or condition associated with the execution of program code by a component of system 100. For example, portions of the program code can include embedded triggering functionality detectable by at least one hardware component of system 100.

[0114] Components of system 100 can be configured to detect triggering functions associated with portions of program code executed by at least one of FPC 104 or SPC 106. In some instances, the triggering function can correspond to at least one of: 1) a specific sequence of steps in a portion or module of the executed program code; or 2) a specific timing parameter indicated by GTC used by the distributed processor unit of system 100.

[0115] In response to the detection of a trigger function, a specific component of system 100 can initiate, trigger, or execute at least one trace point (e.g., a trace event), which causes trace entry data associated with one or more hardware events to be stored in at least one memory buffer of the hardware component. As described above, the stored trace data can then be provided to chip manager 216 via at least one trace chain 203, 205, 207.

[0116] Figure 5 This is a process flowchart of an example process 500 for distributed hardware tracing using the component characteristics of system 100 and one or more nodes 200, 201 of system 100. Therefore, process 500 can be implemented using one or more of the aforementioned computing resources of system 100, including nodes 200, 201.

[0117] Process 500 begins at block 702 and includes a computing system 100 that monitors the execution of program code executed by one or more processor components (including at least FPC 104 and SPC 106). In some embodiments, the execution of program code generating tracing activities can be monitored at least partially by multiple host systems or a subsystem of a single host system. Thus, in these implementations, system 100 is capable of executing multiple processes 500 related to the analysis of tracing activities for hardware events occurring on distributed processing units.

[0118] In some implementations, the first processor component is configured to execute at least a first portion of the monitored program code. At block 504, process 500 includes computing system 100 monitoring the execution of program code executed by a second processor component. In some implementations, the second processor component is configured to execute at least a second portion of the monitored program code.

[0119] Components of computing system 100 may each include at least one memory buffer. Block 506 of process 500 includes system 100 storing data of one or more hardware events in at least one memory buffer that identifies a particular component. In some implementations, hardware events occur across distributed processor units, which include at least a first processor component and a second processor component. The stored data identifying hardware events may each include a hardware event timestamp and metadata characterizing the hardware event. In some implementations, the collection of hardware events corresponds to timeline events.

[0120] For example, system 100 can store data identifying one or more hardware events, which partially correspond to the movement of data packets between a source hardware component within system 100 and a destination hardware component within system 100. In some embodiments, the stored metadata characterizing the hardware event can correspond to at least one of the following: 1) a source memory address, 2) a destination memory address, 3) a unique trace identifier associated with the trace entry that caused the hardware event to be stored, or 4) a size parameter associated with a direct memory access (DMA) trace entry.

[0121] In some implementations, storing data identifying a set of hardware events includes storing the event data in memory buffers of FPC 104 and / or SPC 106, corresponding to, for example, at least one local trace event queue 310. Storing the event data can indicate a subset of hardware event data that can be used to generate a larger hardware event timeline. In some implementations, the storage of the event data is in response to the occurrence of at least one of FPC 104 or SPC 106 executing a hardware trace instruction associated with a portion of program code executed by a component of system 100.

[0122] At block 508 of process 500, system 100 generates a data structure, such as structure 320, that identifies one or more hardware events from a set of hardware events. The data structure is capable of arranging the one or more hardware events in chronological order of events associated with at least a first processor component and a second processor component. In some embodiments, the data structure identifies a hardware event timestamp for a specific trace event, a source address associated with the trace event, or a memory address associated with the trace event.

[0123] At block 510 of process 500, system 100 stores the generated data structure in a repository of a host device associated with host system 126. In some embodiments, host system 126 can use the stored data structure to analyze the performance of program code executed by at least a first processor component or a second processor component. Similarly, host system 126 can use the stored data structure to analyze the performance of at least one component of system 100.

[0124] For example, the user or host system 126 can analyze data structures to detect or determine whether there are performance problems associated with the execution of a specific software module within the program code. Example problems might include a software module failing to complete execution within its allocated execution time window.

[0125] Furthermore, the user or host device 126 can detect or determine whether a specific component of system 100 is operating above or below a threshold performance level. Example problems related to component performance can include specific hardware components that perform a particular event but generate result data that exceeds an acceptable range of parameter parameters. In some implementations, the result data may be inconsistent with result data generated by other relevant components of system 100 performing substantially similar operations.

[0126] For example, during the execution of program code, a first component of system 100 may be required to perform an operation and produce a result. Similarly, a second component of system 100 may be required to perform a substantially similar operation and produce substantially similar results. Analyzing the resulting data structures can indicate that the results generated by the second component are significantly different from those generated by the first component. Similarly, data structures can indicate result parameter values ​​of the second component that are clearly outside the acceptable range of result parameters. These results may indicate potential performance problems of the second component of system 100.

[0127] Embodiments and functional operations of the subject matter described herein can be implemented in digital circuits, in tangibly embodied computer software or firmware, in computer hardware, or in a combination of one or more of these, including the structures described herein and their structural equivalents. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by or control of their operation by a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electronic, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0128] The processes and logic flows described in this specification can be executed by a programmable computer that executes one or more computer programs to perform functions by manipulating input data and generating data. The processes and logic flows can also be executed by dedicated logic circuits, and the device can be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units).

[0129] Computers suitable for executing computer programs include, for example, central processing units (CPUs) capable of being based on general-purpose or special-purpose microprocessors or both, or any other type. Generally, the CPU receives instructions and data from read-only memory or random access memory or both. The basic elements of a computer are the CPU for implementing or executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes, or is operatively coupled to, one or more mass storage devices for storing data, to receive data from or transfer data to, or both, such as magnetic magneto-optical disks or optical disks. However, a computer need not have such devices.

[0130] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor storage devices including, for example, EPROM, EEPROM, and flash memory devices, and disks such as internal hard disks or removable disks. Processors and memories can be augmented with or combined with dedicated logic circuitry.

[0131] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments can also be implemented in combination within a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as operating in certain combinations or even as initially claimed features, one or more features from a claimed combination can be excluded from said combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0132] Similarly, although operations are depicted in a specific order in the diagrams, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all the illustrated operations to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, but rather it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0133] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and the desired result can still be obtained. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to obtain the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method performed using a tracing system, the method comprising: Provide code to processors that include multiple nodes; The code is executed using each of the plurality of nodes of the processor; At each of the plurality of nodes, a tracking bit is detected to be set to a first value, the first value causing tracking data to be generated at the processor using a counter of the tracking system; In response to determining that the tracking bit is set to the first value, a corresponding portion of the tracking data is locally generated at each node; Locally, a corresponding portion of the tracking data generated at each node is stored, wherein the portion of the tracking data is stored in a corresponding hardware tracking buffer included at that node; and Each portion of the corresponding part of the tracking data generated at the processor is provided to the host for analysis at the host. Wherein, at least one of the plurality of nodes is a fixed-function component node, the fixed-function component node being operable to perform a specific operation, and the method further includes: Using the fixed-function component node, the specific operation is performed based on instructions issued by the code; and In response to detecting that the tracking bit is set to the first value, a corresponding portion of the tracking data is locally generated at the fixed functional component node based on the specific operation. Among them, the fixed functional component node is the memory access engine, and The specific operation includes performing the memory access operation using the memory access engine, and the memory access operation includes multiple intermediate operations involving at least one other node among the plurality of nodes.

2. The method according to claim 1, wherein, The corresponding part of generating the tracking data includes: Based on the instructions issued by the code, a descriptor for a memory access operation to be performed using the memory access engine is generated, wherein the descriptor includes the trace bit set to the first value, so that the trace data is generated at the processor.

3. The method according to claim 1, wherein: Performing the memory access operation causes the tracking data associated with the intermediate operation to be stored in the hardware tracking buffer included at the other nodes.

4. The method according to claim 1, wherein, Generating the tracking data at the processor includes: Data representing tracking entries is generated and provided to a tracking chain, which corresponds to a data communication bus coupling two or more of the plurality of nodes.

5. The method according to claim 4, further comprising: The tracking chain is used to provide the data representing the tracking entry for storage in the chip manager of the tracking system; and The data representing the trace entries is stored in the chip manager, which is operable to store multiple trace entries for multiple hardware events that occur at the processor during the execution of the code.

6. The method according to claim 1, further comprising: The host embeds operands in the code provided to the processor so that the processor's nodes store trace data describing hardware events that occur during the execution of the code at the processor.

7. The method according to claim 6, wherein, The code provided includes: In response to the provision of the code by compiling the code by the host, the host is coupled to the processor through a host interface and a node organization of the tracing system, wherein the host uses each of the host interface and the node organization to provide the compiled code to the processor.

8. The method according to claim 7, wherein, Compiling the code includes: The operands are embedded in the code provided to the processor so that performance analysis of the compiled code can be performed using the host's monitoring engine, and the host communicates with the host interface and the node organization structure of the tracing system.

9. The method according to claim 1, wherein, The processor implements a neural network comprising multiple layers, and the method includes: At each of the plurality of nodes, a tracking bit is detected and set to a first value, which causes tracking data to be generated at the processor during the computation of the workload of the neural network.

10. The method according to claim 9, wherein: The processor is a multi-core processor; At least one of the plurality of nodes is a core of the multi-core processor; and The method further includes: locally storing at the kernel a portion of the tracking data generated at the kernel during the computation of the workload of the neural network. The portion of the tracking data generated at the core is stored in a hardware tracking buffer included at the core.

11. A tracking system, the tracking system comprising: One or more processing devices; as well as A non-transitory machine-readable storage device for storing one or more instructions, said instructions being executable by said one or more processing devices to cause operations to be performed, said operations including: Provide code to processors that include multiple nodes; The code is executed using each of the plurality of nodes of the processor; At each of the plurality of nodes, a tracking bit is detected and set to a first value, the first value causing tracking data to be generated at the processor; In response to determining that the tracking bit is set to the first value, a corresponding portion of the tracking data is locally generated at each node; Locally, a corresponding portion of the tracking data generated at each node is stored, wherein the portion of the tracking data is stored in a corresponding hardware tracking buffer included at that node; and Each portion of the corresponding part of the tracking data generated at the processor is provided to the host for analysis at the host. Wherein, at least one of the plurality of nodes is a fixed-function component node, the fixed-function component node being operable to perform a specific operation, and the operation further includes: Using the fixed-function component node, the specific operation is performed based on instructions issued by the code; and In response to detecting that the tracking bit is set to the first value, a corresponding portion of the tracking data is locally generated at the fixed functional component node based on the specific operation. Among them, the fixed functional component node is the memory access engine, and The specific operation includes performing the memory access operation using the memory access engine, and the memory access operation includes multiple intermediate operations involving at least one other node among the plurality of nodes.

12. The tracking system according to claim 11, wherein, The corresponding part of generating the tracking data includes: Based on the instructions issued by the code, a descriptor for a memory access operation to be performed using the memory access engine is generated, wherein the descriptor includes the trace bit set to the first value, so that the trace data is generated at the processor.

13. The tracking system according to claim 11, wherein: Performing the memory access operation causes the tracking data associated with the intermediate operation to be stored in the hardware tracking buffer included at the other nodes.

14. The tracking system according to claim 11, wherein, Generating the tracking data at the processor includes: Data representing tracking entries is generated and provided to a tracking chain, which corresponds to a data communication bus coupling two or more of the plurality of nodes.

15. The tracking system according to claim 14, wherein, The operation includes: The tracking chain is used to provide the data representing the tracking entry for storage in the chip manager of the tracking system; and The data representing the trace entries is stored in the chip manager, which is operable to store multiple trace entries for multiple hardware events that occur at the processor during the execution of the code.

16. The tracking system according to claim 11, wherein, The operation includes: The host embeds operands in the code provided to the processor so that the processor's nodes store trace data describing hardware events that occur during the execution of the code at the processor.

17. The tracking system according to claim 16, wherein, The code provided includes: In response to the provision of the code by compiling the code by the host, the host is coupled to the processor through a host interface and a node organization of the tracing system, wherein the host uses each of the host interface and the node organization to provide the compiled code to the processor.

18. The tracking system according to claim 17, wherein, Compiling the code includes: The operands are embedded in the code provided to the processor so that performance analysis of the compiled code can be performed using the host's monitoring engine, and the host communicates with the host interface and the node organization structure of the tracing system.

19. The tracking system according to claim 11, wherein, The processor implements a neural network comprising multiple layers, and the operations include: At each of the plurality of nodes, a tracking bit is detected and set to a first value, which causes tracking data to be generated at the processor during the computation of the workload of the neural network.

20. A non-transitory machine-readable storage device for storing one or more instructions, said instructions being executable by one or more processing devices of a tracking system to cause operations to be performed, said operations including: Provide code to processors that include multiple nodes; The code is executed using each of the plurality of nodes of the processor; At each of the plurality of nodes, a tracking bit is detected and set to a first value, the first value causing tracking data to be generated at the processor; In response to determining that the tracking bit is set to the first value, a corresponding portion of the tracking data is locally generated at each node; Locally, a corresponding portion of the tracking data generated at each node is stored, wherein the portion of the tracking data is stored in a corresponding hardware tracking buffer included at that node; and Each portion of the corresponding part of the tracking data generated at the processor is provided to the host for analysis at the host. Wherein, at least one of the plurality of nodes is a fixed-function component node, the fixed-function component node being operable to perform a specific operation, and the operation further includes: Using the fixed-function component node, the specific operation is performed based on instructions issued by the code; and In response to detecting that the tracking bit is set to the first value, a corresponding portion of the tracking data is locally generated at the fixed functional component node based on the specific operation. Among them, the fixed functional component node is the memory access engine, and The specific operation includes performing the memory access operation using the memory access engine, and the memory access operation includes multiple intermediate operations involving at least one other node among the plurality of nodes.

21. The non-transitory machine-readable storage device according to claim 20, wherein, Generating the tracking data at the processor includes: Data representing tracking entries is generated and provided to a tracking chain, which corresponds to a data communication bus coupling two or more of the plurality of nodes; The data representing the tracking entry is received by the chip manager of the tracking system and stored in the chip manager; and In response to receiving, the data representing the tracking entry is stored in the chip manager of the tracking system, wherein the chip manager is operable to store multiple tracking entries of multiple hardware events that occur at the processor during the execution of the code.

Images