A heterogeneous NPU performance data processing system and method

Abstract

The application provides a heterogeneous NPU performance data processing system and method. The system comprises: a project import and file type identification module, which is used for receiving an import file or folder selected by a user and processing according to file types; a single database performance data initialization and version checking module, which is used for querying a version field in a temporary database table based on a feature identifier; a heterogeneous database to tracking event conversion module, which is used for converting multi-source data into a tracking event standard format; a multi-node data merging module, which is used for scanning all database files under a directory and integrating into a single tracking event array using a merging operation; a binary tracking analysis module, which is used for decoding an event string and organizing event output structured data according to a processing core; and an API type differentiation detail view module, which is used for searching a mapping table according to an API name. The system provided by the application significantly reduces the risk of memory overflow in a large data volume scenario.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence data processing technology, specifically to a heterogeneous NPU (Neural Network Processing Unit) performance data processing system and method. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, AI (Artificial Intelligence) accelerators such as neural network processors (NPUs) have been widely used in various computing scenarios, including image recognition, natural language processing, speech recognition, and autonomous driving. When performing neural network inference and training, NPUs generate a large amount of heterogeneous performance data, which is crucial for performance analysis, bottleneck identification, and system optimization.

[0003] Specifically, NPU performance data includes, but is not limited to, the following types: Runtime API (Application Programming Interface) data, i.e., runtime interface call sequences and time-consuming information; Stream data, i.e., computation flow and scheduling information; Task data, i.e., specific tasks and their submission, scheduling, and completion times; Kernel Function data, i.e., kernel function execution details; Core Meta data, i.e., NPU kernel-level metadata; Evamind (instruction-level trace) data, i.e., instruction-level trace data, including bin binary format; and CCL (Collection Communication Library) data, i.e., multi-rank (distributed node number) distributed communication data. This data is stored in various heterogeneous formats such as SQLite (lightweight relational database), JSON (data exchange format), and binary bin, resulting in complex table structures and diverse hierarchical relationships.

[0004] The existing technology has the following inherent defects: First, there is incompatibility due to heterogeneous formats. NPU performance data is stored using a multi-table structure, including runtime_api (runtime interface table), task (task table), stream (computation stream), kernel_function (kernel function table), core_meta (core-level metadata table), evamind, and ccl_device (collection communication device table). This storage format is incompatible with the Trace Event specification and cannot be directly parsed and displayed using common trace visualization tools such as Perfetto (performance analysis visualization tool) and ChromeDevTools (browser developer tools). Users need to manually convert the data or write dedicated parsing scripts, which is inefficient and prone to errors.

[0005] Second, large data volumes can lead to memory overflows. When the NPU performs large-scale neural network inference, loading millions of API records and kernel records can easily cause out-of-memory (OOM) errors. Existing solutions lack effective paging and streaming mechanisms, making it impossible to guarantee stable system operation when processing massive amounts of data, thus limiting the availability of performance analysis tools in large-scale scenarios.

[0006] Third, multi-rank (distributed node number) data is scattered. In multi-GPU or multi-rank distributed training scenarios using CCL (Collective Communication Library), the performance data for each rank is stored in different .db files, and existing technologies lack unified merging and analysis capabilities. Users need to open each rank's data file individually for analysis, making it impossible to obtain a global view of the performance data and hindering cross-rank performance comparisons and communication analysis.

[0007] Fourth, binary traces are not visible. Evamind (instruction-level tracing) stores instruction-level traces in binary bin format, which contains special encodings such as magic number, DataHeader (data header structure), and EventStr (event string encoding). Current technology lacks a solution for parsing this binary format and exporting it as JSON (data exchange format), which makes it impossible for front-end visualization tools to load and display instruction-level trace data.

[0008] Fifth, there is insufficient differentiation in API display. Different APIs (such as evMalloc and evMallocAsync) require different dimensions of detailed display (memory, stream, task, etc.). The existing general view cannot meet the needs of differentiated display, and users cannot quickly obtain key performance information for specific API types. Summary of the Invention

[0009] The objective of this invention is to provide a heterogeneous NPU performance data processing system and method. Through this system and / or method, the problems of incompatible heterogeneous NPU performance data formats, memory overflow during large-scale data processing, scattered multi-rank (distributed node number) data, unvisualized binary traces, and insufficient differentiated API display in existing technologies are addressed. The invention achieves unified conversion of heterogeneous NPU performance data, merging of multi-rank data, parsing of binary traces, and differentiated detailed display. It offers the advantages of controllable memory usage, complete mapping relationships, unified analysis of multi-rank data, parsable and visualized binary traces, and differentiated display of API details.

[0010] In a first aspect of the invention, the aforementioned task is solved by a heterogeneous NPU performance data processing system, the system comprising: The project import and file type identification module is configured to receive the imported files or folders selected by the user, calculate the hash value based on the file path, file size, modification time, and creation time as a feature identifier, query the system table to perform deduplication detection, and process them according to file type. The single database performance data initialization and version verification module is configured to query the version field in the temporary database table based on feature identifiers, compare it with the system's preset version information, and establish a formal data connection in the system directory after the verification is successful. The heterogeneous database to tracing event conversion module is configured to convert multi-source data into a tracing event specification format through time normalization, pagination scanning, and hierarchical mapping. The multi-source data includes a runtime call table, a task table, and a kernel function table. The multi-node data merging module is configured to scan all database files in the directory, extract node identifiers from filenames, establish connections to the database of each node and perform version verification, create merge database instances, traverse the source tables to obtain table structure information, add a data source distinction column to the target table, append the corresponding node identifier when inserting each row of data, instantiate a conversion processing program for each node to obtain multiple sets of trace data, and use the merging operation to integrate them into a single trace event array. The binary tracing and parsing module is configured to query the kernel function table for records where the binary filename is not empty, sequentially attempt to locate binary and structured files in the same directory as the database, directly read the content of the structured file, and call the parser to perform preset magic number checks and header field parsing on the binary file, decode event strings, and output structured data according to the core event processing. The API type-differentiated details view module is configured to look up the mapping table based on the API name when the details are called during front-end request runtime. If a mapping exists, the corresponding details logic is called; otherwise, the default processing logic is used.

[0011] In one embodiment of the present invention, in the heterogeneous database to tracking event conversion module, time normalization uses a unified reference time normalization method, with the global minimum start time as the reference value, and the normalized time of any event is calculated by the difference between the original time and the reference value.

[0012] In one embodiment of the present invention, in the heterogeneous database to tracking event conversion module, the pagination parameters are read through system configuration items. When not configured, the default value is used, and upper and lower limit constraints are used to ensure processing stability.

[0013] In one embodiment of the present invention, in the multi-node data merging module, version verification is used to ensure that the version information of all nodes is consistent. Merging is rejected when the version information is inconsistent. The version information includes field semantics, time unit and statistical caliber.

[0014] In one embodiment of the present invention, the binary tracing and parsing module performs protocol-driven parsing based on a defined protocol structure, and adapts to format upgrades through version identification and branch parsing.

[0015] In one embodiment of the present invention, the API name mapping table in the API type differentiation details view module supports configurable extension, and the extension is completed by adding new processing logic and route registration when adding a new API type.

[0016] In one embodiment of the present invention, the heterogeneous NPU performance data processing system further includes: The offset correction module is configured to calculate the clock offset by reading the system logs of each node and introduce a linear compensation term in time normalization.

[0017] In a second aspect of the invention, the aforementioned task is also solved by a heterogeneous NPU performance data processing method, wherein the heterogeneous NPU performance data processing system is used in the method, which includes the following steps: The system receives the imported file or folder selected by the user, calculates a hash value based on the file path, file size, modification time, and creation time as a feature identifier, queries the system table to perform deduplication detection, and processes the files according to their type. The version field in the temporary database table is queried based on the feature identifier and compared with the system's preset version information. After the verification is successful, a formal data connection is established in the system directory. Multi-source data is converted into a trace event specification format through time normalization, pagination scanning, and hierarchical mapping. The multi-source data includes a runtime call table, a task table, and a kernel function table. Scan all database files in the directory, extract node identifiers from filenames, establish connections to the database of each node and perform version verification, create a merge database instance, traverse the source table to obtain table structure information, add a data source distinction column to the target table, append the corresponding node identifier when inserting each row of data, instantiate the conversion processing program for each node to obtain multiple sets of trace data, and use the merge operation to integrate them into a single trace event array. The kernel function table is queried to find records where the binary filename is not empty. Within the same directory as the database, binary and structured files are located sequentially. The structured file's content is read directly. For the binary file, a parser is invoked to perform preset magic number checks and header field parsing, decoding event strings and outputting structured data according to the core event processing logic. When the front-end requests and calls the details, it looks up the mapping table based on the API name. If a mapping exists, the corresponding details logic is called; otherwise, the default processing logic is used.

[0018] In a third aspect, the present invention also provides an electronic device comprising: A processor, configured to execute computer-readable instructions; The graphics card is configured to train the heterogeneous NPU performance data processing method described above; and A memory configured to store computer-readable instructions that, when executed by a processor and / or a graphics card, perform the steps of the heterogeneous NPU performance data processing method.

[0019] In a fourth aspect, the present invention also provides a computer-readable storage medium having stored thereon computer-readable instructions which, when executed by a processor, perform the steps of the heterogeneous NPU performance data processing method.

[0020] The technical solution provided by this invention has the following advantages: 1. The heterogeneous NPU performance data processing system and method proposed in this invention reduces the peak memory usage from O(N) related to the total data volume to O(pageSize) related to the page size through a paging scanning and recursive processing mechanism. With a page size of 200 or 1000, the upper bound of memory usage is controllable, which significantly reduces the risk of Out Of Memory (OOM) in large data scenarios and improves the system's conversion stability and continuous operation capability in large-scale data scenarios.

[0021] 2. The heterogeneous NPU performance data processing system proposed in this invention achieves complete mapping through cross-table primary key association links. The core link is that the runtime_api runtime interface table is associated with the task table through api_seq (interface sequence number), and the task table is associated with the kernel_function table through task_uuid (unique task identifier). The association key and key identification information are retained to ensure that the subordinate relationship between each API and its derived Task and Kernel is traceable. This achieves lossless and accurate mapping from the complex data model inside the NPU to the flat, time-series Trace Event model.

[0022] 3. The heterogeneous NPU performance data processing system and method proposed in this invention realizes the unified merging and analysis of performance data of each rank in multi-card or multi-rank (distributed node number) scenarios. It distinguishes the data sources of different ranks by using the extend_rank_id (extended node identifier) ​​column, uses a unified reference time normalization method for time axis alignment, and uses concat (array merging) to merge into a single traceEvents array, providing users with a global view of performance data and supporting cross-rank performance comparison and communication analysis.

[0023] 4. The heterogeneous NPU performance data processing system proposed in this invention performs magic number verification, DataHeader parsing, and EventStr decoding on Evamind (instruction-level tracing) bin (binary) files, and outputs JSON (data exchange format) for front-end loading. This solves the technical problem of the unvisualability of binary traces, enabling instruction-level trace data to be loaded and displayed by front-end visualization tools.

[0024] 5. The heterogeneous NPU performance data processing system proposed in this invention routes different APIs to different detail assembly logics through apiNameDetailsMap (interface name detail mapping table), realizing differentiated detail views according to API type. For example, evMalloc (memory allocation interface) is mapped to malloc (memory allocation) detail logic, and evMallocAsync (asynchronous memory allocation interface) is mapped to mallocAsync detail logic (asynchronous memory allocation detail logic), which meets the needs of different API types for differentiated display dimensions. Attached Figure Description

[0025] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0026] Figure 1 A schematic diagram of a heterogeneous NPU performance data processing system according to an embodiment of the present invention is shown; Figure 2 This diagram illustrates a process for project import and file type identification, as well as single database performance data initialization and version verification, according to an embodiment of the present invention. Figure 3 A schematic diagram illustrating the conversion process from a heterogeneous database to tracing events according to an embodiment of the present invention is shown; Figure 4 A schematic diagram of the association mapping according to an embodiment of the present invention is shown; Figure 5 A schematic diagram of a multi-node data merging process according to an embodiment of the present invention is shown; Figure 6 A schematic diagram of a binary tracing and parsing process according to an embodiment of the present invention is shown; Figure 7 This diagram illustrates an API detail differential routing process according to an embodiment of the present invention. Figure 8 A flowchart illustrating a heterogeneous NPU performance data processing method according to an embodiment of the present invention is shown.

[0027] It should be noted that, Figure 2-7 The English names retained in the manual are mainly program interface names, function names, data table names, field names, and field names in the event tracing specification. These English names are all technical identifiers commonly used in this field. They are retained to maintain consistency with the system implementation and data specifications. Chinese explanations of their meanings are added in parentheses in the corresponding positions in the manual. Detailed Implementation

[0028] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of the embodiments of the invention. However, the invention is not limited to these specific details.

[0029] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0030] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0031] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by a corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by a corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by a corresponding output device (such as a display, speaker, etc.), and so on.

[0032] To address the problems of incompatible heterogeneous performance data formats in existing technologies, easy memory overflow during large-scale data processing, scattered multi-rank (distributed node number) data, unvisual binary traces, and insufficient differentiated API display, as well as the common challenges in the visualization and analysis of heterogeneous performance data, this invention provides a heterogeneous NPU performance data processing system. It offers a complete technical solution from heterogeneous data import, unified conversion, multi-rank merging, binary parsing to Trace Event generation, which has significant technical innovation and practical value.

[0033] Figure 1 A schematic diagram of a heterogeneous NPU performance data processing system according to an embodiment of the present invention is shown. Figure 1 As shown, the heterogeneous NPU performance data processing system includes: a project import and file type identification module 101, a single database performance data initialization and version verification module 102, a heterogeneous database to trace event conversion module 103, a multi-node data merging module 104, a binary trace parsing module 105, and an API type differential details view module 106. The following provides a detailed description of each module.

[0034] The project import and file type identification module 101 is the entry module of the system of the present invention. It is responsible for receiving the import file or folder selected by the user, performing file deduplication detection, and performing stream processing according to file type.

[0035] The single database performance data initialization and version verification module 102 is responsible for initializing the connection, verifying the version, and copying the data for a single .db Profiler data file.

[0036] Figure 2 The workflow of the project import and file type identification module 101 and the single database performance data initialization and version verification module 102 is shown. Steps 201-206 show the workflow of the project import and file type identification module 101, and steps 207-210 show the workflow of the single database performance data initialization and version verification module 102.

[0037] Step 201: The user selects the file or folder to be imported through the front-end interface. The imported object can be a single .json file, a single .db file, or a folder containing multiple .db files (corresponding to CCL multi-rank scenarios).

[0038] Step 202: Identify file types and process them separately according to file type.

[0039] Step 203: For each imported .db file or file path, the system calls the `getFileHashKey` function (the function to retrieve the file hash key), calculating the MD5 (Message Digest Algorithm) hash value based on four attributes: `path`, `size`, `mtime`, and `ctime`, which serves as the hash key (a feature-based hash key). Here, `path` is the file path, `size` is the file size, `mtime` is the file's last modification time, and `ctime` is the file's creation time. The combination of these four attributes uniquely identifies a file, accurately distinguishing them even if the filenames are the same but the content is different, or if the content is the same but the paths are different.

[0040] Step 204: If the imported object is a .json file, it is determined to be Timeline data and copied directly to the trace directory.

[0041] Step 205: If the imported object is a folder, it is determined to be CCL (Collection Communication Library) data and enters the multi-node data merging module (i.e., CCL multi-rank data merging module).

[0042] Step 206: Query the system table t_sys_file (system file table) to check if the hashKey (characteristic identifier hash key) already exists. If the hashKey already exists, it means the file has already been imported, and the system directly returns the existing project information to achieve deduplication and avoid processing the same data repeatedly. If the hashKey does not exist, proceed to the next processing step.

[0043] If the imported object is a .db file, it is determined to be Profiler data and enters the single database performance data initialization and version verification module (i.e., the single DB Profiler data initialization and version verification module).

[0044] The single database performance data initialization and version verification module is responsible for initializing the connection, verifying the version, and copying the data for a single .db Profiler data file.

[0045] Step 207: The system uses a temporary hashKey (a feature-based hash key) to connect to the SQLite database (a lightweight relational database) to avoid conflicts with existing database connections. After a successful connection, the system queries the `basic_info` table (basic information table) to obtain the `version` field value. This `version` field identifies the version information of the Profiler data; different versions of data may differ in table structure, field semantics, time units, etc. The system verifies the obtained `version` field value against `packageInfo.profilerDataVersions` (a list of supported Profiler data versions). `packageInfo.profilerDataVersions` is a list of supported Profiler data versions. If the `version` field value is not in the supported list or does not match the expected version, error code 5014 is returned, and the data file is rejected for import to avoid data parsing errors due to version inconsistencies.

[0046] Step 208: After the version verification passes, the system copies the DB (database) file to the system directory, naming the file {hashKey}.db, and establishes a formal database connection. Subsequently, the system calls ProfilerDataHandle (performance analysis data processing program) to convert the DB (database) into Trace Event JSON format (data exchange format), and writes the conversion result to the {hashKey}.json file.

[0047] Step 209: The system call processBinFiles processes the bin (binary) file pointed to by the trace_bin_name field in the kernel_function table.

[0048] Step 209: Write the above results into trace JSON (data exchange format).

[0049] Figure 3 A schematic diagram illustrating the conversion process from a heterogeneous database to tracing events according to an embodiment of the present invention is shown; Figure 4 A schematic diagram of an association mapping according to an embodiment of the present invention is shown. The following is in conjunction with... Figure 3 and Figure 4 This describes the workflow of the heterogeneous database to tracking event conversion module 103.

[0050] The heterogeneous database to trace event conversion module (i.e., heterogeneous DB to Trace Event conversion module) is responsible for converting heterogeneous SQLite database (lightweight relational database) data into the standard Trace Event specification format so that it can be parsed and displayed by general trace visualization tools such as Perfetto (performance analysis visualization tool) and Chrome DevTools (browser developer tools).

[0051] The heterogeneous database to tracing event conversion module primarily converts multi-source data into a tracing event specification format through time normalization, pagination scanning, and hierarchical mapping. The multi-source data includes the runtime call table (i.e., runtime_api), the task table, and the kernel function table (kernel_function), as shown in 401.

[0052] Step 301, Time Normalization. The system executes the SQL query `SELECT min(start_ts (start timestamp)) asts FROM runtime_api (runtime interface table)` to obtain the minimum `start_ts` (start timestamp) value of all records in the `runtime_api` table, denoted as `minTime` (minimum baseline time). This `minTime` serves as the global baseline start time, and the timestamps of all subsequent events are calculated relative to this baseline time. If the `runtime_api` table is empty, the system backtracks to query the `task` table or the `kernel_function` table to obtain the minimum time as `minTime`. The normalized time calculation formula for any event is: `trace_ts (normalized trace timestamp) = raw_ts (raw timestamp) - minTime`, with the output unit being microseconds (µs). The nanosecond field is converted proportionally, as shown in step 402. By using a unified baseline time normalization method, cross-table events are ensured to enter a unified time domain, achieving time sequence alignment.

[0053] Step 302, Pagination Scan. The system uses pagination query to read data from the runtime_api table (runtime interface table). The pagination parameter `pageSize` (page size) is the number of records queried and transformed per page, read through system configuration items; a default value is used if not configured. In the current implementation, the default value for `ProfilerDataHandle` (performance analysis data processing program) is 200, and the value in the main process import call is 1000. `pageSize` has upper and lower limits; it is recommended to set `minPageSize` (minimum page size) to 50 and `maxPageSize` (maximum page size) to 5000. If the configuration is invalid or exceeds the limits, the data will be truncated to the valid range. The pagination query SQL statement is: `SELECT ...` FROM runtime_api ORDER BYstart_ts ASC LIMIT $pageSize OFFSET $offset, where offset=(current-1) `pageSize`, `current` is the current page number, initially 1, incrementing by 1 for each page processed. `offset` is the SQL offset. If the `runtime_api` table is empty, the query will fall back to the `task` table for pagination.

[0054] Step 303, Multi-table join query. Based on the list of api_seq (interface sequence numbers) recorded in the current page's runtime_api (runtime interface table), the system performs parallel queries on the related tables: querying the kernel_function table (kernel function table) with the condition WHERE api_seq IN (current page's api_seq list); and querying the task table (task table) with the condition WHERE api_seq IN (current page's api_seq list). Through the api_seq join key, the system establishes the dependency relationship between runtime_api, task, and kernel_function, ensuring that no relationship is lost.

[0055] This invention employs a cross-table primary key association link to achieve complete mapping. The core link is: runtime_api (runtime interface table) -> task (task table) -> kernel_function (kernel function table), retaining the association key (such as api_seq (interface sequence number)) and key identification information. This method ensures that the subordinate relationship between each API and its derived Task and Kernel is traceable. The relationship assembly, i.e., the main association link, includes: a first-level main link, runtime_api.api_seq = task.api_seq; a second-level main link, task.task_uuid (unique task identifier) ​​= kernel_function.task_uuid (and can be combined with api_seq for consistency verification); when merging multiple ranks (distributed node numbers), an additional dimension is added: extend_rank_id (extended node identifier), as shown in 403.

[0056] Step 304, Trace Event Mapping. The system maps the retrieved data into the TraceEvent specification format, as shown in 404: The RUNTIMEAPI (Runtime Interface) data is mapped to a range of events with ph (event phase identifier) ​​of X, representing the duration of the API call. The cat (event category), pid (process identifier), and tid (thread identifier) ​​fields support extend_rank_id (extended node identifier) ​​to distinguish different ranks (distributed node numbers). A range of events includes fields such as name (filename), ph (event phase identifier), ts, dur, pid (process identifier), tid (thread identifier), and cat (event category), where ts is the start timestamp and dur is the duration.

[0057] TASK data is mapped to interval events with ph (event stage identifier) ​​of X and instantaneous events with ph of i, namely TASK_END (task completion). Interval events represent the duration of task execution, while instantaneous events represent the moment the task completes. The combination of interval events and instantaneous events fully expresses the task submission, execution, and completion process.

[0058] KERNELFUNCTION data is mapped to an interval event with ph (event stage identifier) ​​of X, representing the execution duration of the kernel function. Kernel function events contain identification information such as kernel_function_uuid (unique identifier of the kernel function) and kernel_function_name (kernel function name).

[0059] FLOW (execution flow) data (i.e., upstream and downstream dependencies) uses ph (event phase identifier) ​​as s and ph as t to represent execution flow relationships. ph as s indicates the start binding of the flow, and ph as t indicates the end binding of the flow. Different events in the same execution flow are associated through flow_id (execution flow identifier), expressing the causal and dependency relationships between events.

[0060] Trace Events retain the semantics of interval events, instantaneous events, and process events, fully expressing the invocation, execution, and dependency relationships.

[0061] Step 305, Recursive Pagination. The system determines whether the length of the current page's runtimeApiList (runtime interface list) is greater than or equal to pageSize (page size). If it is greater than or equal to pageSize, it means there may be subsequent data, and current (current page number) is incremented by 1. The handle function (pagination processing function) is recursively called to process the next page. If it is less than pageSize, it means all data has been processed, and callback is called to return traceData. The processing flow for each page is: query, map, append trace, release intermediate page objects, and proceed to the next page. The termination condition is that runtimePageCount (runtime interface page record count) is less than pageSize in runtime_api (runtime interface table) pagination mode, or taskPageCount (task page record count) is less than pageSize in task (task table) pagination mode.

[0062] The mapping rules can be parameterized and extended within the existing framework; adding new fields or event categories can add mapping items without disrupting the existing links.

[0063] As shown in 405, the consistency guarantee points of the association mapping include: no loss of relationship: the API-Task-Kernel topology is preserved through api_seq (interface sequence number) and task_uuid (task unique identifier); time comparability: after unifying minTime (minimum base time), they enter the same time axis; multi-source distinguishability: extend_rank_id (extended node identifier) ​​avoids confusion caused by merging multiple ranks (distributed node numbers); complete event semantics: interval, instantaneous, and stream association events jointly express the execution process.

[0064] This invention employs a paging mechanism of "default value + parameterized constraints + recursive paging execution". The paging parameter `pageSize` is read from system configuration items; if not configured, a default value is used. Stability is ensured through upper and lower bound constraints: Missing configuration: revert to the default value; invalid configuration (non-positive integer): revert to the default value; configuration out of bounds: truncate to the valid range according to the upper / lower bound. This invention uses LIMIT / OFFSET paging query and recursive processing: read the current page data; complete the current page Trace Event conversion; output and release the current page's intermediate objects; proceed to the next page until completion. Therefore, memory usage is decoupled from the total data size and is mainly controlled by `pageSize`.

[0065] Compared to a unified conversion after full loading, the paginated conversion mechanism of this invention has the following advantages: peak memory usage is reduced from O(N) related to the total data volume to O(pageSize) related to pageSize. With a pageSize of 200 or 1000, the upper bound of memory usage is controllable, significantly reducing the risk of Out Of Memory (OOM) errors in large data scenarios; it supports conversion and output simultaneously, shortening the time for the first batch of results to become available; and it improves conversion stability and continuous operation capability in large-scale data scenarios. Memory usage is decoupled from the total data size and is mainly controlled by pageSize.

[0066] The heterogeneous NPU performance data processing system proposed in this invention achieves complete mapping through cross-table primary key association links. The core link is that runtime_api (runtime interface table) is associated with task (task table) through api_seq (interface sequence number), and task (task table) is associated with kernel_function (kernel function table) through task_uuid (task unique identifier). The association key and key identification information are retained to ensure that the subordinate relationship between each API and its derived task and kernel is traceable. This achieves lossless and accurate mapping from the complex data model inside the NPU to the flat, time-series Trace Event model.

[0067] The multi-node data merging module is responsible for merging and analyzing the performance data stored in each rank in a distributed training scenario with multiple cards or multiple ranks (distributed node numbers) in CCL (Collective Communication Library).

[0068] Figure 5 A schematic diagram of a multi-node data merging process according to an embodiment of the present invention is shown below. Figure 5 This section explains the workflow of the multi-node data merging module (i.e., the CCL multi-rank data merging module).

[0069] Step 501: Select the CCL (Collection Communication Library) folder.

[0070] Step 502: Scan DB (database) files. The system calls the getDBFiles(path) function to scan all .db files in the specified directory. This function traverses the directory and its subdirectories, identifying all SQLite database (lightweight relational database) files with the .db extension.

[0071] Step 503: Extract the rank (distributed node ID) identifier. The system extracts the rank identifier from each .db filename. For example, if the filename is xxx_1.db, the extracted rankId (node ​​identifier) ​​is 1; if the filename is xxx_2.db, the extracted rankId is 2. The rank identifier in the filename is usually appended to the base filename in the form of underscores followed by numbers.

[0072] Step 504, Initialize Connection. The system establishes an SQLite database (lightweight relational database) connection for each rank (distributed node ID) DB (database) file, and stores the connection object in the tempDbInstance (temporary database instance) [rankId] dictionary, using rankId (node ​​identifier) ​​as the key for management.

[0073] Step 505, Version Verification. The system calls the `checkVersions` function to perform version consistency verification on the DB (database) files of all ranks (distributed node IDs). Specifically, the system queries the `basic_info` table (basic information table) of each rank DB to obtain the `version` field value and checks whether the versions of all ranks are consistent and in the supported list. The merging behavior is comparable and interpretable. If there are inconsistent or unsupported versions, error code 5014 is returned, and merging is rejected. Rejecting merging when versions are inconsistent avoids false conclusions caused by incorrect alignment. In one embodiment of the present invention, in the multi-node data merging module, version verification is used to ensure that the version information of all nodes is consistent. Merging is rejected when the version information is inconsistent. Version information includes field semantics, time unit, and statistical caliber.

[0074] Step 506, Table Structure Copying. The system creates a merged DB instance, `newDbInstance`, as the unified database after the merger. For each source DB (database) of each rank (distributed node ID), the system calls the `copyTables` function to copy the table structure and migrate data: the system traverses the source tables, executing a `PRAGMAtable_info` query to obtain table structure information; an `extend_rank_id` (extended node identifier) ​​`INTEGER` column is added to the target table to identify which rank the data originates from; when inserting each row of data from the source table into the target table, `fromRankId` (source node identifier) ​​is appended as the value of the `extend_rank_id` column. The `extend_rank_id` column distinguishes the data sources of different ranks in the merged data, avoiding confusion during multi-rank merging.

[0075] Step 507, Trace Merging. The system instantiates a ProfilerDataHandle object for each rankId (node ​​identifier), passing in the rankId parameter to distinguish different ranks (distributed node numbers). Each ProfilerDataHandle independently processes the data for its corresponding rank, resulting in multiple sets of traceData.

[0076] Step 508: The system uses the `concat` method to merge multiple sets of `traceData` into a single `traceEvents` array, which is then written to the `{hashKey}.json` file. The merged `traceEvents` array contains event data for all ranks (distributed node IDs). The rank source is distinguished by the `extend_rank_id` field, and the timeline is aligned using a unified baseline time normalization method. This means that all data sources are uniformly converted to the same relative timeline before merging, providing users with a global view of performance data. This method meets the time-series analysis requirements of the scenario described in this application without introducing additional complex calibration algorithms.

[0077] The heterogeneous NPU performance data processing system and method proposed in this invention realizes the unified merging and analysis of performance data of each rank in multi-card or multi-rank scenarios. It distinguishes the data sources of different ranks by using the extend_rank_id column, uses a unified reference time normalization method for time axis alignment, and uses concat to merge them into a single traceEvents array, providing users with a global view of performance data and supporting cross-rank performance comparison and communication analysis.

[0078] The binary tracing parsing module is responsible for parsing the Evamind (instruction-level tracing) instruction-level trace binary bin file and outputting JSON format (data exchange format) data for the front-end visualization tool to load.

[0079] Figure 6 A schematic diagram of a binary tracing and parsing process according to an embodiment of the present invention is shown below. Figure 6 This section explains the workflow of the binary tracing parsing module (i.e., the binary trace parsing module).

[0080] Step 601: Open the bin (binary) file.

[0081] Step 602: Query bin references. The system executes the following SQL query: SELECT kernel_function_uuid (unique identifier of kernel function), kernel_function_name (name of kernel function), trace_bin_name (trace binary file name), api_seq (interface sequence number) FROM kernel_function (kernel function table) WHERE trace_bin_name (trace binary file name) IS NOT NULL AND trace_bin_name != "". This query obtains all kernel function records associated with binary trace files, including the unique identifier of the kernel function, the name of the kernel function, the binary trace file name and the API sequence number.

[0082] Step 603: Locate source files. The system sequentially attempts to locate the binary trace file in the same directory as the DB (database) file. The attempt order is: {trace_bin_name (trace binary file name)}, {trace_bin_name}.json, {trace_bin_name}.bin. The original file name without extension is tried first, followed by the .json extension, and finally the .bin extension.

[0083] Steps 604 and 605: Judge and process file types. If the located file is in .json format, the system directly reads its JSON (data exchange format) content, and writes it to the output file {hashKey (feature identifier hash key)}_{kernel_function_uuid (unique identifier of kernel function)}.json. If the located file is in .bin (binary) format, the system calls gen_trace_json.exe (executable program for generating trace JSON) for parsing, with parameters -bin<binPath (binary file path)>-o<outputPath (output path)>.

[0084] Steps 606 to 609: Parse bin. gen_trace_json.exe performs the following parsing operations on the binary file: Magic number check. The system reads the magic number field in the file header, and checks whether DataHeader (data header structure).dataHeaderMagic is equal to 5a5aa5a5. The magic number is used to identify the file format type. A failed check indicates that the file format is incorrect or the file is corrupted.

[0085] DataHeader parsing. The system parses the DataHeader structure to obtain the following fields: head_len (header length), data_len (data length), inst_cnt (instruction count), cluster_id (cluster identifier), core_id (core identifier), tes0_flag (TES0 flag), tes1_flag (TES1 flag), etc. These fields describe the basic attributes and organizational structure of the binary data.

[0086] EventStr encoding (event string encoding) parsing. The system parses the EventStr encoding to obtain the following fields: type (event type, denoted as e), instance (instance identifier, denoted as i), pc (program counter), and ts (timestamp). EventStr encoding uses a compact binary format to store event information, which is then converted into structured data after parsing.

[0087] Events are organized by core. The system groups and organizes events according to core_id (core identifier), and the event sequence of each core is output independently. The output format is JSON (data exchange format), which includes the core identifier and the corresponding event list for loading and display in the Evamind (instruction-level tracing) core-level view.

[0088] Steps 610 and 611 output JSON and write it to hashKey_kernel_uuid.json.

[0089] In one embodiment of the present invention, the binary tracing and parsing module performs protocol-driven parsing based on a defined protocol structure (such as magic number, header structure, event structure, etc.), rather than a fixed, hard-coded process that cannot be evolved. Adaptation to format upgrades is achieved through version identification and branch parsing: the corresponding parsing logic is selected based on the header version or flag; reserved fields can be passed through or processed according to compatibility strategies; the output layer remains unified to the Trace Event / JSON interface, maintaining the stability of the upper-layer visualization. When the underlying instruction set or format evolves, the upper-layer analysis and display module does not need to be reconstructed synchronously, possessing engineering maintainability and continuous evolution capabilities.

[0090] The heterogeneous NPU performance data processing system proposed in this invention performs magic number verification, DataHeader parsing, and EventStr decoding on Evamind (instruction-level tracing) bin (binary) files, and outputs JSON (data exchange format) for front-end loading. This solves the technical problem of the unvisualability of binary traces, enabling instruction-level trace data to be loaded and displayed by front-end visualization tools.

[0091] The API type-differentiated details view module is responsible for providing differentiated details display logic based on the API type when displaying API details on the front end.

[0092] Figure 7 A schematic diagram of the API details differential routing process according to an embodiment of the present invention is shown below. Figure 7 This explains the workflow of the API type differentiation details view module.

[0093] Step 701: The frontend sends a GET_RUNTIME_API_DETAILS request (request to obtain runtime interface details). The request parameters include the api_name (interface name) field, which identifies the name of the API whose details need to be viewed.

[0094] Step 702: The system looks up the apiNameDetailsMap mapping table (interface name details mapping table) based on api_name (interface name). apiNameDetailsMap is a predefined mapping table that maps API names to corresponding details processing logic. For example, evMalloc (memory allocation interface) maps to malloc (memory allocation) details logic, and evMallocAsync (asynchronous memory allocation interface) maps to mallocAsync details logic (asynchronous memory allocation details logic).

[0095] In steps 703 and 704, if `apiName` exists in `apiNameDetailsMap` (interface name details mapping table), the corresponding details processing logic is invoked. The details processing logic for different API types focuses on different performance dimensions: memory-based APIs (such as `evMalloc`) focus on memory allocation size, allocation address, stream (computation stream) associations, etc.; asynchronous APIs (such as `evMallocAsync`) focus on asynchronous operation status, stream binding, task dependencies, etc.; and stream-based APIs focus on stream identifiers, intra-stream task sequences, inter-stream dependencies, etc.

[0096] Step 705: If `apiName` does not exist in `apiNameDetailsMap` (interface name details mapping table), the default `detailsNorm` logic (default details processing logic) is used. The default logic expands according to the following hierarchy: basic information of `runtime_api` (runtime interface table), associated streams information, associated tasks information, associated kernelFunctionList (kernel function list) information, and associated coreMetaList (kernel metadata list) information. This default logic provides a general details display framework suitable for most API types without special configuration.

[0097] The heterogeneous NPU performance data processing system proposed in this invention routes different APIs to different detail assembly logics through apiNameDetailsMap (interface name detail mapping table), realizing differentiated detail views according to API type. For example, evMalloc (memory allocation interface) is mapped to malloc (memory allocation) detail logic, and evMallocAsync (asynchronous memory allocation interface) is mapped to mallocAsync detail logic (asynchronous memory allocation detail logic), which meets the needs of different API types for differentiated display dimensions.

[0098] This invention employs a routing mechanism from API name to detail processing logic to achieve differentiated detail display (such as memory-based, asynchronous, and streaming types), with the aim of supporting heterogeneous display requirements for different APIs within a unified framework.

[0099] In one embodiment of the present invention, the API name mapping table in the API type differentiation details view module supports configurable extension. When a new API type is added, the extension is completed by adding new processing logic and route registration. When a new API type is added, the main process framework remains unchanged. This approach avoids structural modifications to the core conversion process, reducing maintenance costs. This solution retains the configurable / registration extension interface, supporting the continuous expansion of API details view capabilities with minor modifications, meeting the needs of product evolution and third-party adaptation.

[0100] In one embodiment of the present invention, the heterogeneous NPU performance data processing system further includes: an offset correction module configured to calculate the clock offset by reading the system logs of each node and to introduce a linear compensation term in time normalization.

[0101] Figure 8 A flowchart illustrating a heterogeneous NPU performance data processing method according to an embodiment of the present invention is shown. Figure 8 As shown, the heterogeneous NPU performance data processing system is used in a heterogeneous NPU performance data processing method, which includes the following steps: Step 801: Receive the imported file or folder selected by the user, calculate the hash value based on the file path, file size, modification time, and creation time as a feature identifier, query the system table to perform deduplication detection, and process the files according to their type.

[0102] Step 802: Query the version field in the temporary database table based on the feature identifier, compare it with the system's preset version information, and establish a formal data connection in the system directory after the verification is successful. Step 803: Convert the multi-source data into a trace event specification format through time normalization, pagination scanning, and hierarchical mapping. The multi-source data includes a runtime call table, a task table, and a kernel function table. Step 804: Scan all database files in the directory, extract node identifiers from file names, establish connections to the database of each node and perform version verification, create a merge database instance, traverse the source table to obtain table structure information, add a data source distinction column to the target table, append the corresponding node identifier when inserting each row of data, instantiate the conversion processing program for each node to obtain multiple sets of trace data, and use the merge operation to integrate them into a single trace event array. Step 805: Query the kernel function table for records where the binary filename is not empty. In the same directory as the database, sequentially attempt to locate the binary file and the structured file. For the structured file, directly read its content. For the binary file, call the parser to perform a preset magic number check and header field parsing, decode the event string, and output structured data according to the core event processing logic. Step 806: When the front-end request calls the details, look up the mapping table based on the API name. If a mapping exists, call the corresponding details logic; otherwise, use the default processing logic.

[0103] In one embodiment of the present invention, an electronic device is also provided, comprising a processor, a graphics card, and a memory. The memory is configured to store computer-readable instructions, the graphics card is configured to train the heterogeneous NPU performance data processing method, and the processor is configured to execute the computer-readable instructions. When the processor and / or graphics card executes the computer-readable instructions, the following processing steps are implemented: receiving a user-selected import file or folder; calculating a hash value as a feature identifier based on the file path, file size, modification time, and creation time; querying a system table to perform deduplication detection and processing according to file type; querying the version field in a temporary database table based on the feature identifier and comparing it with the system's preset version information; establishing a formal data connection in the system directory after successful verification; converting multi-source data into a tracking event specification format through time normalization, pagination scanning, and hierarchical mapping; the multi-source data includes a runtime call table, a task table, and a kernel function table; scanning all database files in the directory, extracting node identifiers from filenames, establishing a connection to the database of each node and performing version verification, and creating a merge. The database instance iterates through the source table to obtain table structure information, adds a data source distinction column to the target table, appends a corresponding node identifier when inserting each row of data, instantiates a transformation process for each node to obtain multiple sets of trace data, and uses a merge operation to integrate them into a single trace event array; it queries the kernel function table for records where the trace binary file name is not empty, attempts to locate the binary file and structured file in the same directory as the database in turn, directly reads the content of the structured file, and calls the parser to perform preset magic number verification and header field parsing for the binary file, decodes the event string and outputs structured data according to the core processing organization; and when the front-end requests and calls the details, it looks up the mapping table according to the API name. If a mapping exists, it calls the corresponding details logic; if no mapping exists, it uses the default processing logic.

[0104] The graphics card used can preferably be a model with a GPU computing power higher than 5.0. Since the amount of data to be trained is large, providing a graphics card configuration can significantly improve the training speed.

[0105] The memory includes various media capable of storing computer-readable instructions, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0106] It is understood that, in addition to the memory and processor mentioned above, the computer system described above also includes other hardware and software components not listed in this specification. The specific components can be determined according to the model of the specific data processing equipment in different application scenarios, and will not be listed and described in detail in this specification.

[0107] In one embodiment of the present invention, a computer-readable storage medium is also provided, on which computer-readable instructions are stored. When executed by a processor, the computer-readable instructions perform the following processing steps: receiving an import file or folder selected by the user; calculating a hash value as a feature identifier based on the file path, file size, modification time, and creation time; querying a system table to perform deduplication detection and processing according to file type; querying the version field in a temporary database table based on the feature identifier and comparing it with the system's preset version information; establishing a formal data connection in the system directory after successful verification; converting multi-source data into a tracking event specification format through time normalization, pagination scanning, and hierarchical mapping, wherein the multi-source data includes a runtime call table, a task table, and a kernel function table; scanning all database files in the directory, extracting node identifiers from the filenames, and processing each node... The system establishes a connection to the database and performs version verification, creates a merge database instance, traverses the source tables to obtain table structure information, adds a data source distinction column to the target table, appends the corresponding node identifier when inserting each row of data, instantiates a transformation processor for each node to obtain multiple sets of trace data, and integrates them into a single trace event array using a merge operation; it queries the kernel function table for records where the trace binary file name is not empty, attempts to locate the binary file and structured file in the same directory as the database, directly reads the content of the structured file, calls the parser to perform preset magic number verification and header field parsing for the binary file, decodes the event string and outputs structured data according to the core processing organization; and when the front-end requests and calls the details, it looks up the mapping table based on the API name. If a mapping exists, the corresponding details logic is called; otherwise, the default processing logic is used.

[0108] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined according to the technical solutions of the invention and their equivalents.

Claims

1. A heterogeneous NPU performance data processing system, characterized in that, include: The project import and file type identification module is configured to receive the imported files or folders selected by the user, calculate the hash value based on the file path, file size, modification time, and creation time as a feature identifier, query the system table to perform deduplication detection, and process them according to file type. The single database performance data initialization and version verification module is configured to query the version field in the temporary database table based on feature identifiers, compare it with the system's preset version information, and establish a formal data connection in the system directory after the verification is successful. The heterogeneous database to tracing event conversion module is configured to convert multi-source data into a tracing event specification format through time normalization, pagination scanning, and hierarchical mapping. The multi-source data includes a runtime call table, a task table, and a kernel function table. The multi-node data merging module is configured to scan all database files in the directory, extract node identifiers from filenames, establish connections to the database of each node and perform version verification, create merge database instances, traverse the source tables to obtain table structure information, add a data source distinction column to the target table, append the corresponding node identifier when inserting each row of data, instantiate a conversion processing program for each node to obtain multiple sets of trace data, and use the merging operation to integrate them into a single trace event array. The binary tracing and parsing module is configured to query the kernel function table for records where the binary file name is not empty, attempt to locate the binary file and the structured file in the same directory of the database, directly read the content of the structured file, and call the parser to perform preset magic number verification and header field parsing for the binary file, decode the event string and output structured data according to the core processing of the event; as well as The API type-differentiated details view module is configured to look up the mapping table based on the API name when the details are called during front-end request runtime. If a mapping exists, the corresponding details logic is called; otherwise, the default processing logic is used.

2. The heterogeneous NPU performance data processing system according to claim 1, characterized in that, In the heterogeneous database to tracking event conversion module, time normalization uses a unified reference time normalization method, with the global minimum start time as the reference value. The normalized time of any event is calculated by the difference between the original time and the reference value.

3. The heterogeneous NPU performance data processing system according to claim 1, characterized in that, In the heterogeneous database to tracking event conversion module, pagination parameters are read through system configuration items. If not configured, default values ​​are used, and upper and lower limit constraints are used to ensure processing stability.

4. The heterogeneous NPU performance data processing system according to claim 1, characterized in that, In the multi-node data merging module, version verification is used to ensure that the version information of all nodes is consistent. Merging is rejected when the version information is inconsistent. The version information includes field semantics, time unit and statistical caliber.

5. The heterogeneous NPU performance data processing system according to claim 1, characterized in that, In the binary tracing and parsing module, binary file parsing is based on a defined protocol structure and performs protocol-driven parsing. Adaptation for format upgrades is achieved through version identification and branch parsing.

6. The heterogeneous NPU performance data processing system according to claim 1, characterized in that, In the API type differentiation details view module, the API name mapping table supports configurable extension. When adding a new API type, the extension is completed by adding new processing logic and route registration.

7. The heterogeneous NPU performance data processing system according to claim 1, characterized in that, Also includes: The offset correction module is configured to calculate the clock offset by reading the system logs of each node and introduce a linear compensation term in time normalization.

8. A method for processing heterogeneous NPU performance data, wherein the method uses the heterogeneous NPU performance data processing system according to any one of claims 1-7, characterized in that, Includes the following steps: The system receives the imported file or folder selected by the user, calculates a hash value based on the file path, file size, modification time, and creation time as a feature identifier, queries the system table to perform deduplication detection, and processes the files according to their type. The version field in the temporary database table is queried based on the feature identifier and compared with the system's preset version information. After the verification is successful, a formal data connection is established in the system directory. Multi-source data is converted into a trace event specification format through time normalization, pagination scanning, and hierarchical mapping. The multi-source data includes a runtime call table, a task table, and a kernel function table. Scan all database files in the directory, extract node identifiers from filenames, establish connections to the database of each node and perform version verification, create a merge database instance, traverse the source table to obtain table structure information, add a data source distinction column to the target table, append the corresponding node identifier when inserting each row of data, instantiate the conversion processing program for each node to obtain multiple sets of trace data, and use the merge operation to integrate them into a single trace event array. The kernel function table is queried to track records where the binary file name is not empty. In the same directory of the database, the binary file and the structured file are located sequentially. The content of the structured file is read directly. The parser is called to perform preset magic number verification and header field parsing for the binary file. The event string is decoded and the event is organized and structured data is output according to the processing core. as well as When the front-end requests and calls the details, it looks up the mapping table based on the API name. If a mapping exists, the corresponding details logic is called; otherwise, the default processing logic is used.

9. An electronic device, characterized in that, include: A processor, configured to execute computer-readable instructions; A graphics card configured to train the heterogeneous NPU performance data processing method of claim 8; as well as A memory configured to store computer-readable instructions that, when executed by a processor and / or a graphics card, perform the steps of the heterogeneous NPU performance data processing method according to claim 8.

10. A computer-readable storage medium, characterized in that, It stores computer-readable instructions, which, when executed by a processor, perform the steps of the heterogeneous NPU performance data processing method according to claim 8.

Images