Performance monitoring method for AI server, medium and computer program product

Abstract

The invention provides a performance monitoring method for an AI server, a medium and a computer program product. The method comprises the following steps: in a deep learning framework layer, in response to a received instruction for executing a logic operation, generating an associated ID for identifying the logic operation, analyzing the logic operation into a runtime API call, and submitting the runtime API call and the associated ID to a runtime layer; in the runtime layer, in response to the received runtime API call and the associated ID, generating a corresponding hardware task and a hardware task ID for identifying the hardware task, recording a mapping relationship between the associated ID and the hardware task ID, and submitting the hardware task and the hardware task ID to the hardware equipment layer; the hardware task is executed on the hardware equipment layer, hardware performance data associated with execution of the hardware task are recorded, and the hardware performance data are bound with the hardware task ID of the hardware task associated with the hardware performance data.

Description

Technical Field

[0001] This disclosure relates to a performance monitoring method for artificial intelligence (AI) servers, a computer-readable storage medium, and a computer program product. Background Technology

[0002] As deep learning models become increasingly complex, AI accelerators (such as neural network processing units (NPUs) and graphics processing units (GPUs) have become core hardware for high-performance computing in AI servers. In AI servers, a hierarchical software stack architecture is formed, from the user-side deep learning frameworks (such as PyTorch and TensorFlow), to the intermediate runtime libraries, and finally to the underlying hardware devices. Summary of the Invention

[0003] In one aspect, this application discloses a performance monitoring method for an AI server. The AI ​​server's software stack architecture includes a deep learning framework layer, a runtime layer, and a hardware device layer. The performance monitoring method includes: in the deep learning framework layer, in response to receiving an instruction to execute a logical operation, generating an associated identifier to identify the logical operation, parsing the logical operation into one or more runtime application programming interface (API) calls, and submitting the one or more API calls and the associated identifier to the runtime layer; in the runtime layer, in response to receiving one or more API calls and the associated identifier, generating one or more hardware tasks and one or more hardware task identifiers to identify the one or more hardware tasks, recording the mapping relationship between the associated identifier and the one or more hardware task identifiers, and submitting the one or more hardware tasks and the one or more hardware task identifiers to the hardware device layer; and in the hardware device layer, executing one or more hardware tasks and recording hardware performance data associated with the execution of the one or more hardware tasks, wherein the hardware performance data is bound to the hardware task identifiers of the associated hardware tasks.

[0004] In other respects, this application discloses a computer-readable storage medium storing instructions and a computer program product including instructions. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0005] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be noted that, in accordance with standard practice in the art, the features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the features may be arbitrarily increased or decreased.

[0006] Figure 1 A schematic diagram of the three-layer software stack architecture of the AI ​​server is shown.

[0007] Figure 2 A schematic diagram illustrating the information flow interaction between the deep learning framework layer, runtime layer, and hardware device layer in an AI server according to an embodiment of this application is shown. Figure 3 A schematic diagram of the overall flow of a performance monitoring method for an AI server according to an embodiment of this application is shown.

[0008] Figure 4 This is a schematic diagram of a computing device that can be implemented according to an embodiment of the present invention. Detailed Implementation

[0009] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0010] Furthermore, the various operations will be described as multiple discrete operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations do not need to be performed in the order presented.

[0011] As deep learning models become increasingly complex, AI accelerators (such as NPUs and GPUs) have become core hardware for high-performance computing in AI servers. AI servers typically have a layered software stack architecture, including a deep learning framework layer on the user side, a runtime layer in the middle, and a hardware device layer at the bottom. Developers designing and maintaining AI servers need to analyze and adjust the performance data of each layer; this process is called AI server performance tuning. During performance tuning, developers typically focus on two levels of time consumption information: first, user-level (host-side) time consumption, which reflects the time consumed by upper-layer applications or frameworks calling runtime application programming interfaces (APIs), such as software overhead and queuing latency caused by memory allocation and kernel startup; second, device-level (AI accelerator-side) time consumption, which reflects the actual execution time of instructions by the hardware device, including data transfer time for direct memory access (DMA), arithmetic and logical operation time of the computation kernel, and the scheduling and parallelism of different task engines (such as the computation engine and copy engine) within the hardware device.

[0012] However, this optimization process faces a core obstacle: timeline misalignment caused by asynchronous execution. To improve overall throughput, AI servers commonly employ asynchronous execution. The basic process of asynchronous execution begins with the host calling the runtime API. At this point, the runtime program writes relevant commands (such as kernel startup or data transfer) into the device's (AI accelerator's) "command queue" and returns immediately. The API call time recorded on the host then ends. The device, at a future point in time, retrieves the command from the command queue, parses it, and schedules it to the corresponding internal hardware engine for execution. For example, the DMA engine might handle data transfer, or the computing engine might perform specific kernel calculations. Finally, after the task is completed, the device typically notifies the host through interrupts or host-side polling, thus completing a full asynchronous execution cycle.

[0013] In this asynchronous execution model, the host sends computation tasks or data transfer commands to the command queue and returns immediately, while the actual hardware execution of the commands occurs asynchronously on the device side. This makes the API call duration recorded on the host side unable to accurately correspond to the actual hardware activity on the device side, resulting in a significant timeline disconnect between the two. This disconnect masks the true performance bottlenecks; for example, a seemingly brief call on the host side may cause a long data wait or computational congestion on the device side, making it difficult for developers to perform accurate root cause analysis and system optimization based solely on single timing information.

[0014] In existing technologies, separate tools are typically used to collect host-side logs and device-side hardware counter data. This separate data collection approach has the following problems. First, it creates data fragmentation, making it difficult for developers to accurately correlate a specific API call on the host side (such as a convolution operation) with a series of complex micro-operations actually performed on the device side (such as the execution of multiple computation kernels and multiple memory transfers in between). Second, due to the lack of a unified context identifier throughout the software stack, analysts struggle to effectively pinpoint the root cause of performance bottlenecks when faced with complex asynchronous call flows; that is, they cannot clearly distinguish whether the problem stems from runtime-level scheduling latency and queue contention, or from low computational efficiency or insufficient memory bandwidth at the hardware device level. Furthermore, performance data output by tools at different levels often differs in time base, format, and granularity, which further increases the technical difficulty and workload of subsequent data cleaning, timestamp alignment, and comprehensive analysis, making end-to-end performance profiling inefficient and error-prone.

[0015] In view of the above problems, according to the embodiments of this application, a performance monitoring method for AI servers based on an associated identifier (ID) binding mechanism is proposed. This method achieves end-to-end performance monitoring and joint analysis from the framework layer to the hardware device layer by passing context identifiers between different software layers and adopting a standardized intermediate data format.

[0016] Figure 1 A schematic diagram of the three-layer software stack architecture of the AI ​​server is shown. (For example...) Figure 1 As shown, a typical AI server's software stack generally adopts a three-layer architecture: the top layer is the deep learning framework layer (such as PyTorch and TensorFlow), which provides a user-friendly high-level programming interface for defining, training, and deploying models, and transforming computational tasks into data flow graphs composed of tensor operations; the middle layer is the runtime layer, which provides standard C language APIs to manage hardware resources, schedule asynchronous task queues, handle memory allocation and transfer, and ultimately compile or map the operations issued by the framework layer into instructions executable by the underlying hardware; the bottom layer is the hardware device layer, namely the specific AI accelerators (such as GPUs, NPUs, etc.) and their drivers, which directly execute the low-level operations such as computational kernels and direct memory access (DMA) transfers issued by the runtime layer, and complete the actual parallel computation and data processing through its internal computing units, storage hierarchy, and task engine. These three layers work together to form a complete technical chain from high-level model expression to low-level hardware execution.

[0017] According to embodiments of this application, a performance monitoring method for AI servers is proposed. This method is based on an associated ID binding and mapping mechanism that links the deep learning framework layer, runtime layer, and hardware device layer. The following will combine... Figure 2 and Figure 3 This section will describe the performance monitoring method in detail.

[0018] Figure 2 A schematic diagram illustrating the information flow interaction between the deep learning framework layer, runtime layer, and hardware device layer in an AI server according to an embodiment of this application is shown.

[0019] Before a deep learning framework layer calls its runtime API to execute a specific task, a globally unique correlation ID (correlation_id) can be set throughout the entire software stack architecture via a pre-defined performance analysis interface (such as the tagging API provided by the deep learning framework layer). This correlation ID can be used to identify a logical operation from the user's perspective. In this application, a logical operation refers to a semantically defined computational unit initiated by the user or application within the deep learning framework layer; it is a high-level operation directly called in the code, such as an operator or a convolutional layer. Figure 2As shown, in the deep learning framework layer, in response to receiving an instruction to execute a logical operation, an associated ID identifying that logical operation is generated. The logical operation is then parsed into one or more runtime API calls, and these runtime API calls and the associated ID are submitted to the runtime layer. Furthermore, the deep learning framework layer can record framework layer performance data associated with the executed logical operation; this performance data should be bound to the associated ID identifying the logical operation.

[0020] According to embodiments of this application, setting a globally unique association ID for a logical operation essentially creates a clear logical context. This context declares to the underlying runtime and performance data acquisition system that, from then until the end of this logical context, one or more runtime API calls (which may include kernel startup, memory copying, synchronous waiting, etc.) initiated by the same thread or associated threads logically belong to this user-level logical operation. This association ID, spanning the software stack, becomes the anchor point for subsequent performance data association and aggregation, enabling asynchronous execution events scattered across different levels and time points to be accurately traced back and uniformly attributed to the user-level logical operation that initially triggered them. This solves the core problem of context fragmentation in performance analysis.

[0021] like Figure 2 As shown, upon receiving a runtime API call from the deep learning framework layer, the runtime layer reads the current associated ID, generates one or more corresponding hardware tasks and one or more hardware task IDs (task_id) to identify these hardware tasks, records the mapping relationship between the associated ID and the one or more hardware task IDs, and submits the one or more hardware tasks and their corresponding hardware task IDs to the hardware device layer. According to an embodiment of this application, the runtime layer internally maintains a command queue and generates a hardware task ID to identify the device side of the hardware task before issuing the relevant commands for the hardware task to the hardware device layer. The runtime layer is responsible for recording the mapping relationship between the associated ID and the hardware task ID; that is, recording "which logical operation generated which hardware tasks." Furthermore, the runtime layer can record runtime performance data associated with the execution of the runtime API call, and this runtime performance data should also be bound to the associated ID identifying the logical operation.

[0022] Further as Figure 2As shown, during the task execution phase, when commands issued by the runtime layer (whether synchronous or asynchronous calls) are finally pushed to the hardware device layer for execution, the hardware driver or dedicated performance monitoring unit in the hardware device layer can record fine-grained hardware performance data. This hardware performance data can record the real activity of the physical hardware device, including, for example: the actual data transfer time and throughput of the DMA engine, the precise start and end timestamps and utilization of each computing core executing computing tasks, and the busy / idle status and scheduling sequence of different internal processing engines (such as computing engines, copy engines, and encoding / decoding engines). Simultaneously with data acquisition, the system, for example through a pre-set hardware performance counter or driver interface, binds this collected low-level hardware performance data in real time to the unique task ID previously assigned to the task by the runtime layer and transmitted along with the command. Finally, this set of hardware performance data bound to the task ID can be output for subsequent correlation analysis with host-side logical performance data.

[0023] As described above, based on the identifier passing mechanism that runs through the software stack, an efficient and low-interference performance data association system is constructed according to the embodiments of this application, realizing independent data collection and logical closed-loop at each level. This system fully guarantees the independence of each layer. The deep learning framework layer, runtime layer, and hardware device layer can rely on local context identifiers (such as correlation_id and task_id) to write their respective generated performance data (such as operator calls, API execution time, and hardware counter data) to dedicated logs or buffers in near real-time and parallel. This parallel writing method avoids the blocking caused by cross-layer synchronization, thereby minimizing the additional overhead and interference of performance analysis on system runtime performance.

[0024] When users need to perform performance data analysis, they can use a pre-defined identifier chain to establish correlations between performance data at different layers. For example, upper-layer analysis tools can use the user's `correlation_id` as the query entry point to first locate all relevant API call sequences and their `task_id`s at the runtime layer. Then, using `task_id` as an index, they can trace back to hardware performance data recorded at the hardware device layer regarding specific hardware execution details, such as the complete lifecycle of each computing core and the state changes of the data transport pipeline. Overall, through this "independent collection, correlation analysis" model, seemingly fragmented and asynchronous performance data at different layers can be logically reconstructed into a unified, end-to-end, interpretable execution timeline, accurately locating performance bottlenecks from user code to the underlying hardware.

[0025] According to some embodiments of this application, in response to a user's performance analysis request regarding the execution logic operations of an AI server, framework-level performance data and / or runtime-level performance data associated with the execution logic operations can be displayed together with hardware performance data on a unified visualization interface (e.g., an interactive timeline interface built using ChromeTracing or similar technologies) based on the association ID and hardware task ID. This visualization interface presents the user with a complete and coherent system execution graph. In this graph, the user can not only intuitively see the exact time and call stack of the host-side runtime API call corresponding to a high-level operation (e.g., a convolution operation) initiated in the deep learning framework, but also penetrate down to the hardware layer to observe the performance data of related hardware activities triggered by the API call on the device side on the same timeline. This complete system execution graph, from high-level logic to low-level physical execution, allows users to clearly identify the true location of performance bottlenecks, thus providing a reliable basis for targeted deep performance tuning.

[0026] According to some embodiments of this application, powerful and flexible metadata extension capabilities can be obtained by adopting a general performance data format such as Chrome Trace Format. For example, the args field reserved in the Chrome Trace Format can be used to record and store performance data for each layer. This field is designed as a key-value dictionary, capable of carrying rich contextual information with minimal overhead. Specifically, when generating performance events (e.g., recording performance data related to operator startup or kernel execution), various types of metadata required for performance analysis can be injected into this field. This metadata can span different layers, such as tensor shapes, data types, and operator attributes from the framework layer, memory allocation sizes and addresses from the runtime layer, and specific hardware register states, cache hit rates, or power consumption readings from the hardware device layer. One advantage of this design is its non-intrusive scalability. Specifically, without pre-modifying or redefining the data structure of core events, debugging and profiling information of arbitrary dimensions and granularity can be dynamically and on-demanded added to different events. This allows for the flexible addition of debugging information of arbitrary dimensions without changing the data structure definition, greatly facilitating customized analysis.

[0027] The following will further combine Figure 3 This document describes the overall flow of a performance monitoring method for an AI server according to embodiments of this application.

[0028] Figure 3A schematic flowchart of a performance monitoring method 300 for an AI server according to an embodiment of this application is shown. Method 300 can be performed by a cross-performance data correlation monitoring system or device for an AI server. Figure 3 In the illustrated embodiment, method 300 may include operations 310 to 330. In other embodiments, the method for cross-performance data correlation monitoring for an AI server may include more or fewer or different operations. This disclosure is not limiting in this respect.

[0029] At point 310, in the deep learning framework layer, in response to receiving an instruction to execute a logical operation, an associated identifier identifying the logical operation is generated. The logical operation is then parsed into one or more runtime application programming interface (API) calls, and the one or more runtime API calls and the associated identifier are submitted to the runtime layer. According to some embodiments of this application, the associated identifier is a unique identifier across the entire software stack used to identify the logical operation. Furthermore, in the deep learning framework layer, framework layer performance data associated with the execution of the logical operation can be recorded, and this framework layer performance data is bound to the associated identifier.

[0030] At point 320, in the runtime layer, in response to receiving one or more runtime application programming interface (API) calls and associated identifiers, one or more corresponding hardware tasks and one or more hardware task identifiers for identifying the one or more hardware tasks are generated. The mapping relationship between the associated identifiers and the one or more hardware task identifiers is recorded, and the one or more hardware tasks and one or more hardware task identifiers are submitted to the hardware device layer. According to some embodiments of this application, in the runtime layer, runtime performance data associated with the execution of one or more runtime API calls can be recorded, and this runtime performance data is bound to the associated identifiers.

[0031] At 330, at the hardware device layer, one or more hardware tasks are executed, and hardware performance data associated with the execution of one or more hardware tasks is recorded, which is bound to the hardware task identifier of the associated hardware task.

[0032] According to some embodiments of this application, in response to a user's performance analysis request regarding the execution of logical operations by an AI server, framework layer performance data and hardware performance data associated with the execution of logical operations can be displayed on a unified visualization interface based on association identifiers and hardware task identifiers.

[0033] According to some embodiments of this application, in response to a user's performance analysis request regarding the execution of logical operations by the AI ​​server, framework layer performance data, runtime performance data, and hardware performance data associated with the execution of logical operations can be displayed on a unified visualization interface based on association identifiers and hardware task identifiers.

[0034] According to some embodiments of this application, framework layer performance data, runtime performance data, and / or hardware performance data can be recorded in the args field of the Chrome Trace Format data format.

[0035] Embodiments of this application may include a computer-readable storage medium. The medium stores instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0036] Embodiments of this application may also include a computer program product. This computer program product includes instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0037] Figure 4 This is a schematic diagram of a computing device that can be implemented according to embodiments of the present invention, and can be used to implement the performance monitoring method for AI servers described above. For example... Figure 4 As shown, computing device 400 may include bus 402 or other communication mechanism for transmitting information, and one or more hardware processors 404 coupled to bus 402 for processing information. The one or more hardware processors 404 may include, for example, one or more general-purpose microprocessors.

[0038] like Figure 4 As shown, in some embodiments, computing device 400 may further include main memory 406 coupled to bus 402. Main memory 406 is used to store information and instructions executed by one or more processors 404, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 404. When these instructions are stored in storage media accessible to one or more processors 404, they can cause computing device 400 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 408 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or networking versions thereof.

[0039] like Figure 4As shown, in some embodiments, computing device 400 may further include one or more communication interfaces 410 coupled to bus 402. Communication interface 410 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, communication interface 410 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0040] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0041] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0042] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0043] Specific embodiments further provide a system including a processor and a computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0044] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0045] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0046] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0047] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0048] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0049] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0050] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0051] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0052] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0053] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0054] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A performance monitoring method for an artificial intelligence server, wherein the software stack of the artificial intelligence server includes a deep learning framework layer, a runtime layer, and a hardware device layer, characterized in that, The performance monitoring method includes: In the deep learning framework layer, in response to receiving an instruction to perform a logical operation, an associated identifier is generated to identify the logical operation, the logical operation is parsed into one or more runtime application programming interface (API) calls, and the one or more runtime API calls and the associated identifier are submitted to the runtime layer; In the runtime layer, in response to receiving the one or more runtime application interface calls and the associated identifier, one or more hardware tasks and one or more hardware task identifiers for identifying the one or more hardware tasks are generated; the mapping relationship between the associated identifier and the one or more hardware task identifiers is recorded; and the one or more hardware tasks and the one or more hardware task identifiers are submitted to the hardware device layer; and At the hardware device layer, the one or more hardware tasks are executed, and hardware performance data associated with the execution of the one or more hardware tasks is recorded, which is bound to the hardware task identifier of the associated hardware task.

2. The performance monitoring method according to claim 1, characterized in that, The association identifier is a unique identifier used to identify the logical operation across the entire software stack.

3. The performance monitoring method according to claim 1, characterized in that, The performance monitoring method further includes: recording framework layer performance data associated with the execution of the logical operation in the deep learning framework layer, wherein the framework layer performance data is bound to the associated identifier.

4. The performance monitoring method according to claim 3, characterized in that, The performance monitoring method further includes: recording runtime performance data associated with the execution of the one or more runtime application interface calls at the runtime layer, wherein the runtime performance data is bound to the associated identifier.

5. The performance monitoring method according to claim 3, characterized in that, The performance monitoring method further includes: in response to a request from a user, displaying the framework layer performance data and hardware performance data associated with the execution of the logical operation on a unified visualization interface based on the association identifier and the hardware task identifier.

6. The performance monitoring method according to claim 4, characterized in that, The performance monitoring method further includes: in response to a request from a user, displaying framework layer performance data, runtime performance data, and hardware performance data associated with the execution of the logical operation on a unified visualization interface based on the association identifier and the hardware task identifier.

7. The performance monitoring method according to claim 3, characterized in that, The framework layer performance data and the hardware performance data are recorded in the args field of the Chrome Trace Format data format.

8. The performance monitoring method according to claim 4, characterized in that, The framework layer performance data, the runtime performance data, and the hardware performance data are recorded in the args field of the Chrome Trace Format data format.

9. A computer-readable storage medium storing instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-8.

10. A computer program product, comprising instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-8.

Images