Performance analysis system and performance analysis method
By pre-embedding tracking circuits in the AI chip to collect hardware and software operation information and analyzing it on the host computer, the disconnect between hardware implementation and programming analysis in AI chip performance tuning is solved, enabling a deeper understanding of AI chip performance and improving tuning efficiency.
Patent Information
- Application Number
- CN202410979680.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-12
AI Technical Summary
In existing technologies, performance tuning of AI chips requires developers to have a deep understanding of the underlying hardware implementation. Programming analysis detached from hardware implementation is difficult, and the lack of connection between software programs and hardware metrics leads to obstacles in programming and performance analysis.
By embedding tracking circuits in the AI chip, hardware and software operation information is collected and analyzed by the host computer. The performance analysis graph is presented on the same timeline using hardware and software timestamps, providing a comparison of hardware and software operation information.
It enables in-depth analysis of AI chip performance, helping developers understand the relationship between programs and hardware, and improving the efficiency of programming and performance tuning.
Smart Images

Figure CN118942500A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an artificial intelligence (AI) chip, and more particularly to a performance analysis system and a performance analysis method for analyzing the artificial intelligence chip. Background Art
[0002] Computing devices such as artificial intelligence chips can provide huge computing power. The huge computing power of AI chips comes from a large number of internal hardware execution units (Execution Unit, EU, or execution core). An AI chip usually contains multiple programmable multiprocessors, such as a stream processor cluster (Stream Processor Cluster, SPC). Each programmable multiprocessor contains multiple computing units (Compute Unit, CU, or computing cores), such as tensor cores, vector cores, and other computing cores, such as integer (Integer, INT) computing cores or floating point (Floating Point, FP) computing cores. Each computing unit usually contains multiple execution units. By organizing various types of computing units through programming, programmable multiprocessors can support general computing, scientific computing, and neural network computing.
[0003] AI chips are widely used in high-performance computing (HPC) and AI training and reasoning. The multi-core computing power of AI chips can accelerate the calculation of single instruction multiple data (SIMD) or single instruction multiple thread (SIMT) models. The multi-core programming model is different from the traditional single-core processor programming model. Performance tuning of AI chips requires developers to have a deeper understanding of the underlying implementation. Generally speaking, the programming model abstraction obscures the underlying hardware implementation details. There are great difficulties in analyzing performance without hardware implementation. The relevant technology has not established a connection from software programs to hardware indicators, which poses an obstacle to the programming and performance analysis of AI chip operator developers. Summary of the invention
[0004] The present invention is directed to a performance analysis system and a performance analysis method for analyzing the performance of an artificial intelligence chip.
[0005] In an embodiment according to the present invention, the performance analysis system includes an artificial intelligence chip and a host. The artificial intelligence chip is pre-embedded with a tracking circuit. The tracking circuit collects software operation information of the artificial intelligence chip from a task scheduling module of the artificial intelligence chip. The tracking circuit collects hardware operation information of multiple programmable multiprocessors from multiple programmable multiprocessors of the artificial intelligence chip. The host is coupled to the artificial intelligence chip. The host collects hardware operation information and software operation information from the tracking circuit of the artificial intelligence chip. Based on the hardware timestamp of the hardware operation information and the software timestamp of the software operation information, the host presents the hardware operation information and the software operation information on the same time axis of the performance analysis graph.
[0006] In an embodiment of the present invention, the performance analysis method includes: pre-embedding a tracing circuit in an artificial intelligence chip; collecting software operation information of the artificial intelligence chip from a task scheduling module in the artificial intelligence chip by the tracing circuit; collecting hardware operation information of multiple programmable multiprocessors from multiple programmable multiprocessors in the artificial intelligence chip by the tracing circuit; collecting hardware operation information and software operation information from the tracing circuit of the artificial intelligence chip by a host; and based on the hardware timestamp of the hardware operation information and the software timestamp of the software operation information, the host presents the hardware operation information and the software operation information on the same timeline of the performance analysis graph.
[0007] Based on the above, the artificial intelligence chip to be analyzed is pre-embedded with a tracking circuit. The tracking circuit can collect the hardware operation information and software operation information of the artificial intelligence chip and then provide it to the host. Based on the hardware timestamp of the hardware operation information and the software timestamp of the software operation information, the host can present the hardware operation information and the software operation information on the same timeline of the performance analysis chart. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 is a schematic diagram of a circuit block of a performance analysis system according to an embodiment.
[0009] Figure 2 It is a flowchart of a performance analysis method according to an embodiment of the present invention.
[0010] Figure 3 is a schematic diagram of a performance analysis diagram according to an embodiment of the present invention.
[0011] Description of Reference Numerals
[0012] 100: Performance Analysis System
[0013] 110: Host
[0014] 120: Artificial Intelligence (AI) Chip
[0015] 121: Task Scheduling Module
[0016] 122: Programmable Multiprocessor
[0017] 123: Tracking Circuits
[0018] 124: Tracking collection module
[0019] 125: Trace Buffer
[0020] 310: Software operation information
[0021] 320: Hardware operation information
[0022] S210, S220, S230, S240: Steps DETAILED DESCRIPTION
[0023] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.
[0024] The term "coupled (or connected)" used in the entire specification of this case (including the claims) may refer to any direct or indirect means of connection. For example, if the text describes a first device coupled (or connected) to a second device, it should be interpreted that the first device can be directly connected to the second device, or the first device can be indirectly connected to the second device through other devices or some connection means. The terms "first", "second", etc. mentioned in the entire specification of this case (including the claims) are used to name the components (element) or to distinguish different embodiments or scopes, and are not used to limit the upper or lower limit of the number of components, nor to limit the order of components. In addition, wherever possible, components / components / steps with the same number in the drawings and embodiments represent the same or similar parts. Components / components / steps using the same number or the same terminology in different embodiments can refer to the relevant descriptions of each other.
[0025] Figure 1 FIG. 1 is a schematic diagram of a circuit block of a performance analysis system 100 according to an embodiment of the present invention. The performance analysis system 100 is used to analyze the performance of an artificial intelligence (AI) chip. Figure 1 The performance analysis system 100 shown includes a host 110 and an AI chip 120. Based on different applications, Figure 1The AI chip 120 shown can be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU) or other computing chips. Computing devices such as artificial intelligence chips can provide huge computing power. The AI chip 120 includes a task scheduling module 121 and multiple programmable multiprocessors 122, such as a streaming processor cluster or other programmable multi-computing core groups. The task scheduling module 121 is a command processor. The task scheduling module 121 selectively sends programming commands to one or more of the multiple programmable multiprocessors 122. Each programmable multiprocessor 122 includes multiple computing units (not shown). Figure 1 ), such as tensor cores, vector cores and other computing cores, such as integer computing cores or floating point computing cores. Each computing unit includes multiple execution units (not shown in FIG. Figure 1 ). By programming and organizing various types of computing units, the programmable multiprocessor 122 can support general computing, scientific computing, and neural network computing. The host 110 is coupled to the AI chip 120. The host 110 can analyze the performance of the AI chip 120.
[0026] Figure 2 is a flow chart of a performance analysis method according to an embodiment of the present invention. Figure 1 and Figure 2 , in step S210, the tracing circuit 123 is pre-buried in the AI chip 120. In step S220, the tracing circuit 123 collects the software operation information of the AI chip 120 from the task scheduling module 121 of the AI chip 120. For example, the software operation information includes different execution times of different computing tasks in different hardware modules. The software timestamp of the software operation information can represent the different execution times. Based on actual applications, the computing work can be referred to as a thread grid (Grid), a thread block (Thread Block), a warp (warp) or a thread (thread), and the hardware module can be referred to as a programmable multiprocessor (which includes one or more computing units, such as a streaming processor cluster), a computing unit, an execution unit or a hardware thread. The hardware corresponding to the thread grid is one or more computing units. When all tasks are full, the hardware implementation corresponding to the thread grid is a whole GPU or GPGPU.
[0027] During the process of an operator being executed on hardware, the task scheduling module 121 sends the corresponding task of this operator to one or more programmable multiprocessors 122. For example, tasks task0 and task1 are sent to the first programmable multiprocessor in the programmable multiprocessor 122, while tasks task2 and task3 are sent to the second programmable multiprocessor in the programmable multiprocessor 122. Each task corresponds to a thread group, and a thread group has a thread group start (group start) and a thread group end (group end). A thread group can be divided into multiple thread blocks. A thread block is a scheduling unit. A thread block is executed in an execution unit. A thread block has a thread block start (warp start), multiple instructions and a thread block end (warp end). Based on the host 110 triggering the tracking and collection mode of the AI chip 120, the task scheduling module 121 can record the time when any one of the thread group start, thread group end, thread block start, thread block end and instruction is triggered. That is, based on actual applications, the "software operation information" (such as instruction information) collected by the tracing circuit 123 from the task scheduling module 121 may include: the instruction identification number (ID), timestamp (instruction trigger time) and index of the execution unit, or the thread block identification number, thread block start timestamp and index of the execution unit, or the thread block identification number, thread block end timestamp and index of the execution unit, or the thread group identification number, thread group start timestamp and index of the programmable multiprocessor, or the thread group identification number, thread group end timestamp and index of the programmable multiprocessor.
[0028] In step S220, the tracking circuit 123 also collects hardware operation information of the programmable multiprocessor 122 from the programmable multiprocessor 122 of the AI chip 120. For example, the hardware operation information includes the counting results of different performance counters of different hardware modules of the programmable multiprocessor 122 at different collection times, and the hardware timestamp of the hardware operation information is used to indicate the different collection times. Each hardware module in the programmable multiprocessor 122 corresponds to a plurality of different performance counters. For example, when a hardware module receives a read request, the latency counter for the read request is triggered to count the time spent by the hardware module for the read request, and the workload counter for the read request is triggered to count the number of times the hardware module executes the request. Performance counters are well-known counters, so they are not described in detail. The tracking circuit 123 collects the counting results of the performance counters at regular intervals, for example, once every 1000 cycles. Based on the regular collection, each collected counting result corresponds to a timestamp. The programmable multiprocessor 122 reports the hardware performance information (the information calculated by the counting result of the performance counter and the timestamp) of any hardware module to the tracking circuit 123. The host 110 uses the counting results of different performance counters of different hardware modules of the programmable multiprocessor 122 at different collection times to calculate the hardware indicators of different hardware modules. For example, the hardware indicator includes at least one of the utilization, latency and bandwidth of the hardware module.
[0029] exist Figure 1 In the illustrated embodiment, the trace circuit 123 includes a trace collection module 124 and a trace buffer 125. The trace collection module 124 is coupled to the task scheduling module 121 and the programmable multiprocessor 122. Based on the host 110 triggering the trace collection mode of the AI chip 120, during the operation of the AI chip 120, the task scheduling module 121 actively reports the software operation information to the trace collection module 124, and the programmable multiprocessor 122 actively reports the hardware operation information to the trace collection module 124. The trace buffer 125 is coupled to the trace collection module 124. The trace collection module 124 writes the software operation information and the hardware operation information into the trace buffer 125. The host 110 reads the software operation information and the hardware operation information of the AI chip 120 from the trace buffer 125.
[0030] For example (but not limited to this), the host 110 allocates a dedicated trace buffer (trace buffer 125) for caching tokens (also called tokens) during the initialization phase, and notifies the AI chip 120 to trigger trace collection (enter trace collection mode) by setting registers. During the operation of the AI chip 120, the task scheduling module 121 and the programmable multiprocessor 122 actively report the information of key events and instructions of the running state to the trace collection module 124. After obtaining the software operation information and the hardware operation information, the trace collection module 124 encodes (or compresses) the hardware operation information and the hardware timestamp into hardware information tokens, and encodes (or compresses) the software operation information and the software timestamp into software information tokens. The trace collection module 124 writes the hardware information tokens and the software information tokens into the trace buffer 125 in the encoding order.
[0031] The host 110 can issue a stop trace collection instruction by resetting the register to notify the AI chip 120 to stop trace collection (end the trace collection mode). After the AI chip 120 receives the stop trace collection instruction issued by the host 110, the trace collection module 124 flushes all hardware information tokens and all software information tokens to the trace buffer 125.
[0032] Please refer to Figure 1 and Figure 2 , in step S230, the host 110 collects the hardware operation information and the software operation information from the tracing circuit 123 of the AI chip 120. For example (but not limited to this), the host 110 reads the hardware information token and the software information token of the tracing buffer 125, and parses (or decompresses) these tokens according to the agreed format. After the host 110 reads the hardware information token from the tracing buffer 125, the host 110 parses (or decompresses) the hardware information token to restore the hardware operation information and the hardware timestamp. After the host 110 reads the software information token from the tracing buffer 125, the host 110 parses (or decompresses) the software information token to restore the software operation information and the software timestamp. Then, the host 110 restores the corresponding instruction information from the software operation information based on instruction assembly. By combining the software operation information with the hardware operation information (the counting results of the performance counters of the AI chip 120 collected at specified intervals (such as 1024 cycles) during the execution process), the host 110 can obtain the hardware indicators such as utilization, latency, bandwidth, etc. of the specific modules corresponding to the AI chip 120 when each programmable multiprocessor 122 executes specific instructions, and then conduct in-depth analysis of the internal behavior of the corresponding AI chip 120 during the execution of specific instructions.
[0033] In step S240, based on the hardware timestamp of the hardware operation information and the software timestamp of the software operation information, the host 110 presents the hardware operation information and the software operation information on the same time axis of the performance analysis graph. For example (but not limited to this), Figure 3 is a schematic diagram of a performance analysis diagram according to an embodiment of the present invention. Figure 3 The horizontal axis represents the time axis. Figure 1 and Figure 3 Based on the software timestamp of the software operation information 310 and the hardware timestamp of the hardware operation information 320, the host 110 may present the hardware operation information 320 and the software operation information 310 in Figure 3 Same timeline as the performance analysis graph shown. Figure 3 The software operation information 310 shown shows the different execution times of different computing tasks in different hardware modules. Figure 3 The hardware operation information 320 shows the counting results of different performance counters of different hardware modules of the programmable multiprocessor 122 at different collection times. The post-silicon platform directly generates a performance analysis chart of the AI chip 120 to facilitate performance analysis and tuning, solving the availability of pre-silicon simulation platforms and device resource limitations. Figure 3 The performance analysis chart shown is convenient for operator development users to analyze and optimize by combining specific program instructions with corresponding hardware indicators. Figure 3 The performance analysis chart shown makes it easy for software developers to understand the indicator details from the upper layer to the lower layer of the program.
[0034] In summary, the AI chip 120 to be analyzed is pre-embedded with a tracking circuit 123. The tracking circuit 123 can collect the hardware operation information and software operation information of the AI chip 120, and then provide it to the host 110. Based on the hardware timestamp of the hardware operation information and the software timestamp of the software operation information, the host 110 can present the hardware operation information and the software operation information on the same time axis of the performance analysis chart.
[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A performance analysis system for analyzing the performance of an artificial intelligence chip, characterized in that: The performance analysis system comprises: The artificial intelligence chip, wherein the artificial intelligence chip is pre-embedded with a tracking circuit, the tracking circuit collects software operation information of the artificial intelligence chip from a task scheduling module of the artificial intelligence chip, and the tracking circuit collects hardware operation information of multiple programmable multiprocessors from multiple programmable multiprocessors of the artificial intelligence chip; and A host coupled to the artificial intelligence chip, wherein the host collects the hardware operation information and the software operation information from the tracing circuit of the artificial intelligence chip, as well as a hardware timestamp of the hardware operation information and a software timestamp of the software operation information, and the host presents the hardware operation information and the software operation information on the same timeline of a performance analysis graph.
2. The performance analysis system according to claim 1, characterized in that: The software operation information includes different execution times of different computing tasks in different hardware modules, and the software timestamp of the software operation information is used to indicate the different execution times.
3. The performance analysis system according to claim 1, characterized in that: The hardware operation information includes counting results of different performance counters of different hardware modules of the multiple programmable multiprocessors at different collection times, and the hardware timestamp of the hardware operation information is used to indicate the different collection times.
4. The performance analysis system according to claim 3, characterized in that: The host calculates hardware indicators of the different hardware modules of the multiple programmable multiprocessors using the counting results of the different performance counters of the different hardware modules at the different collection times.
5. The performance analysis system according to claim 4, characterized in that: The hardware indicator includes at least one of utilization, latency and bandwidth.
6. The performance analysis system according to claim 1, characterized in that: The tracking circuit of the artificial intelligence chip includes: a tracing and collecting module coupled to the task scheduling module and the plurality of programmable multiprocessors of the artificial intelligence chip, wherein based on the host triggering the tracing and collecting mode of the artificial intelligence chip, during the operation of the artificial intelligence chip, the task scheduling module actively reports the software operation information to the tracing and collecting module, and the plurality of programmable multiprocessors actively reports the hardware operation information to the tracing and collecting module; and A trace buffer is coupled to the trace collection module, wherein the trace collection module writes the software operation information and the hardware operation information into the trace buffer, and the host reads the software operation information and the hardware operation information from the trace buffer.
7. The performance analysis system according to claim 6, characterized in that: After the tracking and collecting module obtains the software operation information and the hardware operation information, the tracking and collecting module encodes the hardware operation information and the hardware timestamp into hardware information tokens, and the tracking and collecting module encodes the software operation information and the software timestamp into software information tokens, and writes the hardware information tokens and the software information tokens into the tracking buffer in the encoding order.
8. The performance analysis system according to claim 7, characterized in that: After the artificial intelligence chip receives the stop tracing collection instruction issued by the host, the tracing collection module refreshes all the hardware information tokens and all the software information tokens to the tracing buffer.
9. The performance analysis system according to claim 7, characterized in that: After the host reads the hardware information token from the trace buffer of the artificial intelligence chip, the host parses the hardware information token to restore the hardware operation information and the hardware timestamp.
10. The performance analysis system according to claim 7, characterized in that: After the host reads the software information token from the trace buffer of the artificial intelligence chip, the host parses the software information token to restore the software operation information and the software timestamp, and the host restores the corresponding instruction information from the software operation information based on instruction assembly.
11. A performance analysis method for analyzing the performance of an artificial intelligence chip, characterized in that: The performance analysis method comprises: Pre-embed the tracking circuit in the artificial intelligence chip; The tracking circuit collects software operation information of the artificial intelligence chip from a task scheduling module in the artificial intelligence chip; The tracking circuit collects hardware operation information of the multiple programmable multiprocessors of the artificial intelligence chip; The host collects the hardware operation information and the software operation information from the tracking circuit of the artificial intelligence chip; and Based on the hardware timestamp of the hardware operation information and the software timestamp of the software operation information, the host presents the hardware operation information and the software operation information on the same time axis of the performance analysis graph.
12. The performance analysis method according to claim 11, characterized in that: The software operation information includes different execution times of different computing tasks in different hardware modules, and the software timestamp of the software operation information is used to indicate the different execution times.
13. The performance analysis method according to claim 11, characterized in that: The hardware operation information includes counting results of different performance counters of different hardware modules of the multiple programmable multiprocessors at different collection times, and the hardware timestamp of the hardware operation information is used to indicate the different collection times.
14. The performance analysis method according to claim 13, characterized in that: The host calculates hardware indicators of the different hardware modules of the multiple programmable multiprocessors using the counting results of the different performance counters of the different hardware modules at the different collection times.
15. The performance analysis method according to claim 14, characterized in that: The hardware indicator includes at least one of utilization, latency and bandwidth.
16. The performance analysis method according to claim 11, characterized in that: The performance analysis method further comprises: Based on the host triggering the tracking and collection mode of the artificial intelligence chip, during the operation of the artificial intelligence chip, the task scheduling module actively reports the software operation information to the tracking and collection module of the tracking circuit, and the multiple programmable multiprocessors actively report the hardware operation information to the tracking and collection module; The trace collection module writes the software operation information and the hardware operation information into a trace buffer of the trace circuit; and The software operation information and the hardware operation information are read from the trace buffer by the host.
17. The performance analysis method according to claim 16, characterized in that: The performance analysis method further comprises: After the tracking and collecting module obtains the software operation information and the hardware operation information, the tracking and collecting module encodes the hardware operation information and the hardware timestamp into a hardware information token, and the tracking and collecting module encodes the software operation information and the software timestamp into a software information token; and The hardware information token and the software information token are written into the trace buffer in a coding order by the trace collection module.
18. The performance analysis method according to claim 17, characterized in that: The performance analysis method further comprises: After the artificial intelligence chip receives the stop tracking and collection instruction issued by the host, the tracking and collection module refreshes all the hardware information tokens and all the software information tokens to the tracking buffer.
19. The performance analysis method according to claim 17, characterized in that: The performance analysis method further comprises: After the host reads the hardware information token from the tracking buffer of the artificial intelligence chip, the host parses the hardware information token to restore the hardware operation information and the hardware timestamp.
20. The performance analysis method according to claim 17, characterized in that: The performance analysis method further comprises: After the host reads the software information token from the trace buffer of the artificial intelligence chip, the host parses the software information token to restore the software operation information and the software timestamp; and The host restores corresponding instruction information from the software operation information based on instruction assembly.
Citation Information
Cited By
Activity scheduling method, system, terminal and storage medium based on high response ratio
US20220291959A1
