Flow monitoring method in machine learning framework and related product thereof

Abstract

The invention relates to a method for monitoring streams in a machine learning framework and related products thereof. The related products comprise equipment for monitoring the streams in the machine learning framework and a computer readable storage medium. The apparatus may be included in a computing processing device of a combined processing device, which may include one or more data processing devices. The combined processing device can also comprise an interface device and other processing devices. And the computing processing device interacts with other processing devices to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the equipment and the other processing devices and used for storing data of the equipment and the other processing devices. Through the scheme of the invention, the lightweight and effective monitoring of various streams in the machine learning framework can be realized while the efficiency of a business production environment is not influenced.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computers. More specifically, the present disclosure relates to a method for monitoring a flow in a machine learning framework, a device for executing the aforementioned monitoring method, and a computer-readable storage medium. Background Art

[0002] Training or reasoning in machine learning frameworks (such as the PyTorch framework) are usually implemented based on a multi-stream mechanism. In actual training / reasoning and other services, performance optimization scenarios often require analyzing multiple streams in the machine learning framework. However, related technologies require the use of some "heavy" tools to complete stream monitoring. These tools not only generate a lot of extra time overhead during specific use, but also cannot be enabled by default in production environments due to their performance issues and usage methods. They only support small-scale debugging and analysis services and cannot be applied to large-scale business generation environments, greatly affecting the efficiency of business production environments. Summary of the invention

[0003] In view of the technical problems mentioned in the above background technology section, the present disclosure proposes a solution for monitoring flows in a machine learning framework. Using the technical solution of the present disclosure, lightweight and effective monitoring of various flows in a machine learning framework can be achieved without affecting the efficiency of the business production environment.

[0004] In a first aspect, the present disclosure provides a method for monitoring a flow in a machine learning framework, comprising: acquiring a target flow to be monitored in the machine learning framework based on a target interface supported by the machine learning framework; performing instrumentation processing on the target flow to obtain instrumentation information about the target flow; and determining hardware time information of a flow operator in the target flow based on the instrumentation information about the target flow.

[0005] In a second aspect, the present disclosure provides a method for monitoring flows in a machine learning framework, comprising: acquiring a communication flow to be monitored in the machine learning framework based on a target interface supported by the machine learning framework; performing instrumentation processing on the communication flow to obtain instrumentation information about the communication flow; and determining end-to-end time information of the communication flow and / or hardware time information of a communication operator in the communication flow based on the instrumentation information about the communication flow.

[0006] In a third aspect, the present disclosure provides a device for monitoring flows in a machine learning framework, comprising: one or more processors; and a memory, wherein the memory stores computer instructions for monitoring flows in a machine learning framework, and when the computer executable instructions are executed by the one or more processors, the device executes the method described in the first aspect or the second aspect.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing computer instructions for monitoring flows in a machine learning framework. When the computer instructions are executed by one or more processors, the method described in the first aspect or the second aspect is executed.

[0008] Through the solutions provided in the above aspects of the present disclosure, effective monitoring of various flows in the machine learning framework can be achieved. Specifically, the target flow to be monitored can be first obtained through the target interface, and then the plugging information of the target flow can be obtained in combination with the plugging technology, and the hardware time monitoring of the flow operator can be realized through the plugging information. The solution of the present disclosure supports the real-time startup of monitoring of various flows (such as computing flows, communication flows, or other flows that need to be monitored) in the machine learning framework in a variety of business production environments such as model reasoning or training, without suspending the business, thereby ensuring the efficiency of the business production environment. At the same time, in the process of realizing the monitoring of the flow, the additional performance overhead generated by the interface technology and the plugging technology adopted is much less than the performance overhead of the "heavy" tools in the traditional technology, thereby realizing lightweight flow monitoring. In addition, the solution of the present disclosure supports hardware time monitoring of the flow operator, so that the hardware time of the flow operator can be used to effectively identify and analyze the performance bottlenecks of the business scenario in the future, thereby providing a new guiding idea for the identification and analysis of the performance bottlenecks of the business scenario.

[0009] Furthermore, the solution disclosed in the present invention can also implement end-to-end time monitoring of the communication flow and / or hardware time monitoring of the communication operators in the communication flow, thereby meeting the demand for status monitoring of the communication flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0011] Figure 1 is a structural diagram showing a board according to an embodiment of the present disclosure;

[0012] Figure 2 is a structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram showing the internal structure of a computing device according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram showing the internal structure of a processor core according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram showing a data writing process between processor cores of different clusters according to an embodiment of the present disclosure;

[0016] Figure 6 is a structural diagram showing the software and hardware architecture of data flow programming according to an embodiment of the present disclosure;

[0017] Figure 7 is a flowchart schematically illustrating a method for monitoring a flow in a machine learning framework according to an embodiment of the present disclosure;

[0018] Figure 8 A flowchart schematically illustrates a method for monitoring a flow in a machine learning framework according to another embodiment of the present disclosure;

[0019] Fig. 9 A flowchart of a method for monitoring communication flows in a distributed training scenario according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0021] It should be understood that the terms "first", "second", "third", "fourth", etc. in the claims, specifications and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0022] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0023] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0024] As mentioned above, in order to achieve performance optimization of services such as training or reasoning in a machine learning framework, it is usually necessary to monitor and analyze multiple streams in the machine learning framework. As far as the application scenarios of the present disclosure are concerned, for example, in machine learning frameworks such as PyTorch, training or reasoning services are mainly implemented based on a multi-stream mechanism, and the streams in the multi-stream may include communication streams that support communication operations and computing streams that support computing operations. Inside the communication stream is a collection of communication operators that are executed serially, and inside the computing stream is a collection of computing operators that are executed serially.

[0025] In actual training / inference and other businesses, performance optimization scenarios often need to rely on the analysis of multiple streams in the PyTorch framework. For example, related technologies use "heavy" tools such as profiler / nvprof / nsys to monitor multiple streams in the PyTorch framework. However, in the specific use of these tools, they are limited by performance issues and usage methods, and cannot be used in the business production environment, which will cause business suspension, thereby affecting the efficiency of the business production environment and generating a lot of additional performance overhead. Specifically, some "heavy" tools need to register multiple callback functions, which will generate serious CPU time overhead during the callback process and are difficult to optimize. Some tools may also trigger a large number of additional I / O operations, which will also generate a lot of additional CPU time overhead.

[0026] In addition, as business needs increase, further fine-grained analysis may be required to identify the performance bottlenecks of the current training / inference business scenarios (for example, whether it is a communication performance problem or a hardware performance problem). The above-mentioned "heavy" tools are usually used to obtain the end-to-end (E2E) time of the computing flow, which is not sufficient to support fine-grained performance optimization analysis needs.

[0027] In order to overcome the defects of the prior art as discussed here and in the background technology section, the solution disclosed in the present invention proposes that the hardware time information of the stream operators in the target stream to be monitored can be determined based on interface technology and plug-in technology, thereby achieving lightweight and effective monitoring of various streams in the machine learning framework without affecting the efficiency of the business production environment, and supporting fine-grained performance optimization analysis needs based on the various time information monitored.

[0028] The device side in the embodiment of the present disclosure may be an artificial intelligence chip or board, etc. Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 100 according to an embodiment of the present disclosure. Figure 1 As shown, the board 100 includes a chip (or "processing chip") 101, which is a system-on-chip (SoC), or system on chip, integrated with one or more combined processing devices, which are artificial intelligence computing units that support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud intelligence applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 100 of this embodiment is suitable for cloud intelligence applications and has huge off-chip storage, on-chip storage, and a large amount of computing power.

[0029] The chip 101 is connected to the external device 103 through the external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card or a WIFI interface. The data to be processed can be transmitted from the external device 103 to the chip 101 through the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0030] The board 100 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 107 and the chip 101 through a bus and transmits data. The control device 106 in the board 100 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0031] Figure 2 1 is a block diagram showing the structure of the combined processing device 200 in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 200 includes a computing device 201 , an interface device 202 , a processing device 203 and a DRAM 204 .

[0032] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0033] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0034] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0035] DRAM 204 is used to store data to be processed, and is a DDR memory, which is usually 16G or larger in size and is used to store data of the computing device 201 and / or the processing device 203 .

[0036] Figure 3The internal structure diagram of the computing device 201 is shown. The computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system on a chip, which includes multiple clusters, and each cluster includes multiple processor cores, which can be used to execute the tasks issued by the present disclosure. In other words, the computing device 201 is composed of a hierarchy of system on a chip-cluster-processor core.

[0037] At the system-on-chip level, Figure 3 As shown, the computing device 201 includes an external storage controller 301 , a peripheral communication module 302 , an on-chip interconnect module 303 , a synchronization module 304 , and multiple clusters 305 .

[0038] There may be multiple external storage controllers 301, two of which are shown in the figure as an example, and are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2 DRAM 204 in the chip, thereby reading data from outside the chip or writing data. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202 to start the computing device 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (Global Barrier Controller, GBC), which is used to coordinate the work progress of each cluster and ensure the synchronization of information. Multiple clusters 305 are the computing cores of the computing device 201. Four are shown as an example in the figure. With the development of hardware, the computing device 201 disclosed in the present invention can also include 8, 16, 64, or even more clusters 305.

[0039] At the cluster level, Figure 3 As shown, each cluster 305 includes a plurality of processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0040] The figure shows four processor cores 306 as an example, but the present disclosure does not limit the number of processor cores 306. Figure 4 As shown, each processor core 306 includes three major modules: a control module 31 , a computing module 32 and a storage module 33 .

[0041] The control module 31 is used to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the deep learning task, and includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results to the operation module 32 and the storage module 33 as control information.

[0042] The operation module 32 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 422 is responsible for the core calculation of the deep learning algorithm, namely matrix multiplication and convolution.

[0043] The storage module 33 is used to store or move related data, including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. NRAM 431 is used to store input and output data and intermediate results for calculation by the processor core 306; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access between NRAM 431 / WRAM 432 and DRAM 204 through the broadcast bus 309; MVDMA 434 is used to control the memory access between NRAM 431 / WRAM 432 and SRAM 308.

[0044] Back to Figure 3 The storage core 307 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processor cores 306, and to perform communication between the cluster 305 and the DRAM 204, communication between the clusters 305, and communication between the processor cores 306. In other embodiments, the storage core 307 has the capability of scalar operation and is used to perform scalar operation.

[0045] The storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (CDMA) 310, and a global direct memory access module (GDMA) 311. The SRAM 308 plays the role of a high-performance data transfer station. The data reused between different processor cores 306 in the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306, but is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0046] The broadcast bus 309, CDMA 310 and GDMA 311 are respectively used to perform communication between processor cores 306, communication between clusters 305 and data transmission between clusters 305 and DRAM 204. They will be described below.

[0047] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 in the cluster 305. The broadcast bus 309 of this embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a copy of data from SRAM 308 to specific processor cores 306, and broadcast is a communication mode of transmitting a copy of data from SRAM 308 to all processor cores 306, which is a special case of multicast.

[0048] The CDMA 310 is used to control the memory access of the SRAM 308 between different clusters 305 in the same computing device 201 . Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core of another cluster to illustrate the working principle of CDMA310. In this application scenario, the same computing device includes multiple clusters. For the convenience of explanation, only cluster 0 and cluster 1 are shown in the figure. Cluster 0 and cluster 1 each include multiple processor cores. For the convenience of explanation, cluster 0 in the figure only shows processor core 0, and cluster 1 only shows processor core 1. Processor core 0 wants to write data to processor core 1.

[0049] First, processor core 0 sends a unicast write request to write data into the local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and write data W to transfer the data to SRAM 1 of cluster 1. Then the slave sends a write response B in response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0050] Back to Figure 3 , GDMA 311 cooperates with external memory controller 301 to control the access of SRAM 308 of cluster 305 to DRAM 204, or reads data from DRAM 204 to SRAM 308. As can be seen from the above, the communication between DRAM 204 and NRAM 431 or WRAM 432 can be realized through two channels. The first channel is to directly connect DRAM 204 and NRAM 431 or WRAM 432 through IODAM 433; the second channel is to first transmit data between DRAM 804 and SRAM 308 through GDMA 311, and then transmit data between SRAM 308 and NRAM 431 or WRAM 432 through MVDMA 434. Although it seems that the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much greater than that of the first channel, so the communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient through the second channel. The embodiments of the present disclosure can select a data transmission channel according to the hardware conditions thereof.

[0051] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For the convenience of description, this disclosure regards GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions implemented and the technical effects achieved are similar to those of this disclosure, they belong to the protection scope of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, as long as the functions implemented and the technical effects achieved are similar to those of this disclosure, they belong to the protection scope of this disclosure.

[0052] Combination of the above Figure 1-Figure 5The hardware architecture and internal structure of the present disclosure are described in detail. It is understandable that the above description is only exemplary and not restrictive. According to different application scenarios and hardware specifications, those skilled in the art may also make changes to the board (or artificial intelligence device) and its internal structure of the present disclosure, and these changes still fall within the scope of protection of the present disclosure. Figure 1-Figure 5 In addition to the hardware architecture shown, the solution of the present disclosure also relates to a hardware and software architecture, which will be described below.

[0053] Figure 6 The design diagram of the hardware and software architecture in one embodiment of the present disclosure is shown. As shown in the figure, the hardware and software architecture in this embodiment may include an AI processor 601, a driver and operating system 602, a compiler and programming language 603, a library 604, a framework layer 605, and an application layer 606. It is understandable that the hardware and software architecture here can be applied to the artificial intelligence computing system or heterogeneous computing platform of this application.

[0054] Specifically, the AI ​​processor 601 (which may be included in the board described below in conjunction with the accompanying drawings, for example) considers both operation optimization and data handling optimization in hardware design. To this end, it uses a customized operation unit to accelerate operations, and uses on-chip storage to accelerate data handling, thereby obtaining extremely high performance and energy efficiency. In addition, in order to support various algorithm optimizations, the AI ​​processor 601 may have a customized operation unit and instruction set, wherein the instruction set may provide operation instructions (scalar, vector and / or matrix) of different granularities. Further, when considering factors such as algorithm memory access characteristics, hardware cost, verification difficulty, etc., on-chip storage may be used, and data handling may be optimized. In actual operation, the AI ​​processor disclosed in the present invention can achieve a speed that is more than dozens of times faster than that of a mainstream GPU (graphics processing unit).

[0055] The driver and operating system 602 are mainly responsible for implementing the scheduling of tasks on the AI ​​processor 601. The scheduling operation can, for example, implement scheduling according to task priority, communication and synchronization between multiple devices, etc. For the compiled program, it can implement the scheduling and execution of the tasks to be implemented on a specific processor through the operating system and the driver, including but not limited to the following operations: allocating and releasing device memory, implementing data transmission between devices, maintaining task queues, and scheduling tasks according to priority to achieve synchronization and collaboration between multiple devices.

[0056] The compiler and programming language 603 may be a set of assembly languages ​​developed for the instruction set of the AI ​​processor 601. In applications, it may translate deep learning operators developed for the AI ​​processor 601 into processor instruction combinations to facilitate calling the AI ​​processor 601, thereby efficiently using the AI ​​processor 601. In some application scenarios, the compiler may be used to perform the intermediate expression stage of compilation to optimize compilation.

[0057] The library 604 may include a runtime library 614 and a machine learning library 624. In one implementation scenario, the aforementioned library 604 may use the instruction set of the AI ​​processor 601 and partially optimize according to the instruction set of the AI ​​processor 601 to improve the running speed of the operator. The runtime library 614 may be a set of high-performance operator libraries specially developed for the AI ​​processor 601, and it may be used to complete the interaction between the general processor and the artificial intelligence processor. Further, the runtime library 614 may also provide a set of interfaces for artificial intelligence processors. For the machine learning library 624, it may be used to accelerate various machine learning or deep learning algorithms on the artificial intelligence processor. Specifically, the machine learning library 624 may provide a set of efficient, general, flexible and scalable programming interfaces, and the upper-level machine learning applications may directly adopt the programming interfaces of various programming frameworks (such as PyTorch, TensorFlow, Caffe, MXNet, etc.), or may use the interfaces provided by the machine learning library 624 to directly program. In addition, the machine learning library 624 of the present disclosure may facilitate the calling of the hardware platform, and the runtime library 614 may implement some basic commonly used operators, such as various operations such as convolution and pooling.

[0058] The framework layer 605 can add encapsulation of operators developed for AI processors, and mainly encapsulates operators of the runtime library 614. In addition, the framework layer 605 can also modify related task scheduling or memory management. In an application scenario, the framework layer 605 can adopt the architecture of frameworks such as PyTorch and TensorFlow.

[0059] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0060] Figure 7 is a flowchart illustrating a method 700 for monitoring a flow in a machine learning framework according to an embodiment of the present disclosure.

[0061] like Figure 7 As shown, at step S702, the target flow to be monitored in the machine learning framework can be obtained based on the target interface supported by the machine learning framework.

[0062] Among them, the aforementioned machine learning frameworks may include frameworks such as PyTorch, TensorFlow, and Caffe. It should be noted that the description of the types of machine learning frameworks here is only an exemplary description, and the solution disclosed in this disclosure does not limit the specific types of machine learning frameworks.

[0063] The aforementioned target stream may include any stream with monitoring requirements supported by the machine learning framework, such as the computing stream, communication stream, etc. in the multi-stream mechanism supported by the machine learning framework.

[0064] In actual applications, since there may or may not be an interface that exposes the target flow to the outside, it is necessary to call the corresponding target interface to obtain the target flow in combination with the specific situation of the target flow to be monitored. In some embodiments, if the target flow is a computing flow with some externally exposed interfaces, the computing flow can be obtained through the native interface in the machine learning framework. In other embodiments, if the target flow is a communication flow with some externally unexposed interfaces, there is no native interface available for calling in the machine learning framework, and a custom interface needs to be set and called to obtain the communication flow. Whether the target flow is obtained through the native interface of the framework or the custom interface, the demand for obtaining the target flow can be met.

[0065] Next, at step S704, the aforementioned target stream can be instrumented to obtain instrumentation information about the target stream. Different from the traditional use of "heavy" tools to suspend business to monitor the target stream, the solution disclosed in the present invention can transfer the target stream from the back end to the front end to obtain the target stream, and introduce instrumentation technology to perform instrumentation on the acquired target stream. With the execution of the target stream after instrumentation, the instrumentation event can be automatically triggered to put the target stream in a monitoring state, and instrumentation information about the target stream (such as start timestamp, end timestamp, etc.) will be generated.

[0066] Then, at step S706, hardware time information of the stream operator in the target stream may be determined according to the instrumentation information about the target stream.

[0067] As mentioned above, performance optimization of business scenarios often requires monitoring and analysis of flows. Traditional technologies usually only support the acquisition of end-to-end time of computing flows, which may not meet some more fine-grained performance optimization analysis needs. After analyzing the business-side needs and the possible problems of performance bottlenecks in business scenarios, it is found that the hardware time of the flow operator can be used or combined to achieve fine-grained performance bottleneck analysis (such as CPU bottlenecks, computing performance bottlenecks, or communication performance bottlenecks, etc.). To this end, in the scheme disclosed in the present invention, the time information of more target flows can be obtained using the plug-in information. That is, the scheme disclosed in the present invention supports the determination of hardware time information of various flow operators (such as computing operators, communication operators, or other operators, etc.), and the specific hardware time of which flow operator needs to be obtained can be adjusted according to actual needs.

[0068] By using the monitoring method 700 and its multiple embodiments disclosed in the present invention, the target flow to be monitored can be first obtained through the target interface, and then the plugging information of the target flow can be obtained in combination with the plugging technology, and the hardware time monitoring of the operator in the flow can be realized through the plugging information. In a variety of business production environments such as model reasoning or training, the monitoring of various flows (such as computing flows, communication flows, or other flows that need to be monitored) in the machine learning framework can be started in real time without suspending the business, thereby ensuring the efficiency of the business production environment. At the same time, in the process of realizing the monitoring of the flow, the additional performance overhead generated by the interface technology and plugging technology adopted is much less than the performance overhead of the "heavy" tools in the traditional technology, thereby realizing lightweight flow monitoring. In addition, by monitoring the hardware time of the flow operator, the hardware time of the flow operator can be used to effectively identify and analyze the performance bottlenecks of the business scenario in the future, which is conducive to the fine-grained analysis requirements of the performance bottlenecks of the business scenario, thereby providing a new guiding idea for the identification and analysis of the performance bottlenecks of the business scenario.

[0069] Figure 8 FIG. 8 is a flow chart showing a method 800 for monitoring a flow in a machine learning framework according to another embodiment of the present disclosure. It should be noted that the monitoring method 800 can be understood as Figure 7 Therefore, the above combined with the monitoring method 700 Figure 7 The relevant details also apply below.

[0070] In this embodiment, the target flow to be monitored in the machine learning framework can be obtained based on the target interface supported by the machine learning framework. Among them, the machine learning framework includes PyTorch, TensorFlow or other types of frameworks, and the machine learning framework can support a multi-stream mechanism. In some embodiments, the flow in the machine learning framework may include a computational flow and a communication flow. Further, as an example, it can be subdivided according to the way the flow is created in the business code. For example, the flow in the machine learning framework may include a computational flow explicitly created in the business code and a communication flow not explicitly created in the business code. Among them, the communication flow that is not explicitly created usually has no interface exposed to the outside, and is generally created inside the machine learning framework, such as a communication flow executed in the reverse calculation process.

[0071] In this regard, in practical applications, it is necessary to select a suitable target interface for effective acquisition according to the specific situation of the target flow to be monitored. Figure 8 As shown, at step S802, the explicitly created computing flow can be obtained based on the default interface, or the non-explicitly created communication flow can be obtained based on the first custom interface. In some embodiments, the target interface includes a first custom interface or a default interface. Among them, the first custom interface can be understood as an interface that is independently set to achieve the required functions, and the default interface can be understood as a native interface in the machine learning framework. The flows in the machine learning framework may include explicitly created computing flows and non-explicitly created communication flows. In some implementation scenarios, in response to the target flow including an explicitly created computing flow, the explicitly created computing flow is generally exposed to the outside by an interface, and the default interface can be called to obtain the explicitly created computing flow. Therefore, the method of directly calling the default interface can improve the usability of the machine learning framework and reduce user learning costs and interface design costs.

[0072] In other implementation scenarios, in response to the target flow including a non-explicitly created communication flow, the target flow at this time has no interface exposed to the outside, and the user needs to set a first custom interface and call the first custom interface to obtain the non-explicitly created communication flow. Specifically, a communication flow list related to the specified device index can be obtained based on the first custom interface. The communication flow list includes a communication flow information object bound to the specified device index, and each communication flow information object is distinguished based on the communication domain to which its inner communication flow belongs.

[0073] Further, in some embodiments, a global variable can be maintained in the distributed backend of the source code of the machine learning framework, and a communication flow bound to the device index can be created in the process group backend of the machine learning framework, and the communication flow created by the process group backend and its corresponding communication domain ID (i.e., communication domain identifier) ​​can be synchronized to the global variable, so as to synchronize the communication flow bound to the device index created in the process group backend through the global variable, thereby achieving effective maintenance of the communication flow. Among them, the communication flow with the same communication domain identifier can be understood as a communication flow supporting the same communication operation, and the communication flow on the same device or in the same communication flow list can be distinguished by the communication domain identifier. In addition, when distinguishing and identifying the communication flow, it is not limited to the method of using the communication domain ID, and the communication flow can also be identified based on the stream ID (i.e., communication flow identifier) ​​corresponding to the communication flow, for example, the communication flow with the same communication domain identifier on different devices or in different communication flow lists can be distinguished by the communication flow identifier.

[0074] Furthermore, in some embodiments, the global variable may be a nested map class object (e.g., an object of the std::unordered_map class), wherein the map class object includes an outer key representing a device index, an inner key representing an identifier of a communication domain, and an inner value representing a communication flow. It should be understood that the solution disclosed herein does not limit the specific form of the global variable, and it may also be packaged into other data forms according to application requirements.

[0075] When a map class object is maintained in a global variable, in the subsequent process of calling the first custom interface, the map class object bound to the specified device index can be obtained from the global variable according to the specified device index based on the transfer function supported by the machine learning framework. Usually, the communication domain ID in the map class object is in binary form, which is not easy for the user to view intuitively. Therefore, the obtained map class object can also be formatted (for example, the communication domain ID, communication flow, etc. can be converted into a format that is easy for the user to read, etc.) to obtain a communication flow information object and return it to the first custom interface. In some embodiments, the obtained communication flow information objects can be composed of a list and returned to the first custom interface in the form of a list. Therefore, the first custom interface method can support users to obtain any flow that needs to be monitored to meet the user's usage needs.

[0076] It should be noted that the above description of the acquisition details of the target flow is only an exemplary description, and the solution of the present disclosure is not limited to this. For example, the flow in the machine learning framework may not be subdivided, and the user may set a custom interface and call the custom interface to obtain the target flow to be monitored.

[0077] After obtaining the target stream, the target stream can be instrumented to obtain instrumentation information about the target stream. In some embodiments, the instrumentation of the target stream can be achieved by creating an Event event or a second custom interface. Specifically, at step S804, an Event event or a second custom interface supported by a machine learning framework can be created. Among them, the created Event event or the second custom interface supports end-to-end time monitoring of the target stream or hardware time monitoring of the stream operator in the target stream. It should be noted that if the hardware time information of the stream operator needs to be monitored later, then at step S804, an Event event or a custom interface that supports hardware time monitoring of the stream operator in the target stream needs to be created. If the end-to-end time information of the target stream needs to be monitored later, then at step S804, an Event event or a custom interface that supports end-to-end time monitoring of the target stream needs to be created. Of course, if the end-to-end time information of the target stream and the hardware time information of the stream operator need to be monitored later, then at step S804, an Event event or a custom interface that supports end-to-end time monitoring of the target stream and supports hardware time monitoring of the stream operator in the target stream needs to be created. In this scenario, the Event event or custom interface that supports end-to-end time monitoring of the target stream and the Event event or custom interface that supports hardware time monitoring of stream operators in the target stream can be the same Event event or custom interface, or different Event events or custom interfaces can be used, which can be set and adjusted according to actual needs.

[0078] In some embodiments, in order to improve the usability of the machine learning framework, reduce the learning cost and the modification of the business code, the creation of Event events can be achieved by expanding the native Event class in the machine learning framework and adding custom methods in the Event class. That is, the aforementioned Event events can be expanded based on the native Event events supported by the machine learning framework.

[0079] For example, when monitoring the hardware time of a stream operator (such as a computing operator, a communication operator, or other custom operators) in a target stream, a new definition (such as "hardware_time") can be added to the native Event class. The present disclosure exemplarily adopts the following code implementation:

[0080] class Event(torch_mlu._MLUC._MLUEventBase,_EventBase):

[0081] defhardware_time(self,end_event):

[0082] time=super().hardware_time(end_event)

[0083] return time

[0084] The newly created Event (eg, "hardware_time") may include an input parameter (eg, "end_event") and an output parameter, wherein the input parameter "end_event" indicates the event that ends hardware time monitoring, and the output parameter indicates the returned monitored hardware time, in units of us.

[0085] Of course, the solution disclosed in the present invention is not limited to the above-mentioned Event event plugging method when plugging the target stream. As mentioned above, a second custom interface can also be set to plug the target stream. Among them, the above-mentioned Event event and the second custom interface can both include a start time function and an end time function. For example, defhardware_time_start() can be used as the start time function in the second custom interface, and defhardware_time_end() can be used as the end time function in the second custom interface.

[0086] At step S806, the Event event or the second custom interface can be inserted into the code of the target flow to obtain a timestamp associated with the created Event event or the second custom interface. In some embodiments, the start time function and the end time function can be inserted into the corresponding positions in the code of the target flow to obtain the timestamp of the start time function and the timestamp of the end time function.

[0087] As an example, after acquiring the target stream, you can first initialize the Event event or the second custom interface. For example, you can initialize the start event / the start time function of the second custom interface and the end event / the end time function of the second custom interface. Then, place the start event or the start time function of the second custom interface in the code of the target stream to start the end-to-end time or hardware time monitoring of the stream. Then, place the endevent or the end time function of the second custom interface in the code of the target stream to end the end-to-end time or hardware time monitoring of the stream.

[0088] Furthermore, if the end-to-end time of the target stream is monitored, the start time function and the end time function can be installed at the start and end positions of the code of the target stream to obtain the timestamps of the start time function and the end time function. If the hardware time monitoring of the stream operators in the target stream is performed, the start time function and the end time function can also be installed at the start and end positions of the code of the target stream to obtain the timestamps of the start time function and the end time function, or the start time function and the end time function can be inserted at the start and end positions of the code of each operator in the stream to count the timestamps of the start time function and the end time function of the monitored operator.

[0089] Then, at step S808, the end-to-end time information of the target stream and / or the hardware time information of the stream operator in the target stream can be determined according to the difference between the timestamp of the aforementioned end time function and the timestamp of the start time function. For example, when determining the end-to-end time information of the target stream, a runtime library function (such as a CNRT interface, etc.) can be called to calculate according to the timestamp of the start time function and the timestamp of the end time function to obtain the end-to-end time of the target stream. When determining the hardware time information of the stream operator in the target stream, a runtime library function (such as a CNRT interface, etc.) can be called to calculate according to the timestamp of the start time function and the timestamp of the end time function to obtain the hardware time of the stream operator. Alternatively, the timestamp of the start time function and the end time function of each stream operator in the stream can also be obtained, so as to obtain the hardware time of each operator in the stream. Then, the hardware time of the operator in the stream is added to obtain the hardware time, so as to use the hardware time and the hardware time information of the stream operator in the target stream.

[0090] In specific applications, the underlying interface (e.g., cnrtNotifierDuration) in the machine learning framework can be called to obtain the hardware time of the computing operator in the computing flow or the communication operator in the communication flow. In some embodiments, the present disclosure is exemplarily implemented using the following code:

[0091] floatNotifier::hardware_time(constNotifier&other)const{

[0092] TORCH_CHECK(is_created_&&other.isCreated(),

[0093] "Both notifiers must be placed before calculating hardware time.");

[0094] floattime_us = 0;

[0095] TORCH_CNRT_CHECK(cnrtNotifierDuration(notifier_,other.notifier_,&time_us));

[0096] returntime_us;

[0097] }

[0098] The specific implementation principle of the underlying interface cnrtNotifierDuration includes: inserting each stream operator (such as a communication operator or a computing operator) in the target stream to obtain the corresponding start and end timestamps, thereby obtaining the hardware time of each operator in the stream. Then, the driver layer adds the hardware times of the operators in the stream to obtain the hardware time and returns it as the output value.

[0099] Therefore, the solution disclosed in the present invention is not limited to obtaining the end-to-end time of the computing flow through the above monitoring process, but can also accurately perceive the hardware time of the operator on the business flow. That is, the technical solution disclosed in the present invention is not limited to determining the end-to-end time information of the computing flow, but also supports the determination of the hardware time information of the computing operator in the computing flow, the determination of the end-to-end time information of the communication flow, and the determination of the hardware time information of the communication operator in the communication flow. The specific need to obtain the end-to-end time of the flow or the hardware time of various flow operators can be adjusted according to actual needs.

[0100] During the implementation of the above process, only a slight change to the original business code is needed to achieve stable and efficient monitoring of the flow status. At the same time, the additional performance overhead generated by this method is almost negligible compared to the E2E time overhead of the business, and the function of continuously monitoring hardware time can be enabled by default in the production environment. In addition, under the customized API and the related mechanism of the extended Event, compared with the traditional method, the impact on CPU time overhead is almost negligible, and it can be enabled by default and continuously monitored in the business production environment, making it truly "lightweight".

[0101] The disclosed solution can be applied to the end-to-end time monitoring of flows and / or the hardware time monitoring of flow operators in various business scenarios supported by machine learning frameworks. In particular, when it is necessary to monitor the communication flow in the machine learning framework, since the communication flow is located at the back end and has no externally exposed interface, it is impossible to obtain the communication flow using traditional methods, and it is even more impossible to accurately monitor the end-to-end time information of the communication flow and / or the hardware time information of the communication operators in the communication flow.

[0102] In this regard, the present disclosure also proposes a monitoring method for communication flows in a machine learning framework, which specifically requires obtaining the communication flow to be monitored in the machine learning framework based on the target interface supported by the machine learning framework, and then performing instrumentation processing on the communication flow to obtain instrumentation information about the communication flow, and determining the end-to-end time information of the communication flow and / or the hardware time information of the communication operators in the communication flow based on the instrumentation information about the communication flow.

[0103] In some embodiments, the aforementioned target interface includes a first custom interface, and the communication flow to be monitored can be obtained based on the first custom interface. As described above, the first custom interface can be understood as an interface that is independently set to achieve the required function. Since the communication flow has no interface exposed to the outside, the user needs to set the first custom interface and call the first custom interface to obtain the communication flow. For how to use the first custom interface to obtain the communication flow, please refer to the above combined with Figure 8 The relevant details of step S802, such as obtaining a communication flow list related to a specified device index, etc., are not repeated here.

[0104] Further, in some embodiments, after the communication flow to be monitored is obtained, the communication flow can be instrumented. For example, an Event event or a second custom interface supported by a machine learning framework can be created so that the Event event or the second custom interface supports end-to-end time monitoring of the communication flow or hardware time monitoring of the communication operator in the communication flow. Then, the Event event or the second custom interface is inserted into the code of the communication flow to obtain a timestamp associated with the Event event or the second custom interface.

[0105] Furthermore, the above-mentioned Event event can be obtained by expanding the native Event event supported by the machine learning framework, and the Event event and the second custom interface both include a start time function and an end time function. Therefore, the start time function and the end time function can be respectively inserted into the corresponding positions in the code of the communication flow to obtain the timestamp of the start time function and the timestamp of the end time function, and according to the difference between the timestamp of the end time function and the timestamp of the start time function, the end-to-end time information of the communication flow or the hardware time information of the communication operator in the communication flow is determined. The process of using the plug-in information of the communication flow to determine the time information to be monitored in this embodiment can be specifically referred to in conjunction with the above text. Figure 8 The relevant details of steps S804 to S808 are not repeated here.

[0106] Therefore, the solution disclosed in the present invention can realize accurate monitoring of the end-to-end time of the communication flow or the hardware time of the communication operator. And when the end-to-end time of the communication flow and the hardware time of the communication operator are monitored at the same time, the comprehensive monitoring of the communication flow status can be realized.

[0107] Fig. 9 900 is a flow chart showing a method 900 for monitoring a communication flow in a distributed training scenario according to an embodiment of the present disclosure. The monitoring method 900 can be understood as Figure 7 Monitoring methods in 700 or Figure 8 A specific technical implementation of the monitoring method 800. Figure 7 and Figure 8 The relevant details also apply below.

[0108] As mentioned above, the explicitly created computing flow can be directly obtained using the default interface in the machine learning framework, while the non-explicitly created communication flow requires setting the first custom interface to obtain it.

[0109] As an example, a custom interface can be set in the PyTorch framework to obtain a list of communication flows (such as CNCL communication flows) in the communication library bound to a specified device. In order to distinguish different communication flow objects in the list, each communication flow object and its communication domain ID can be packaged into a communication flow information object (such as a StreamInfo class object) as an element in the list.

[0110] In some embodiments, the custom interface can be named “torch.mlu.cncl_stream”, where: torch.mlu.cncl_stream(device:Optional[_device_t]=None)->List[StreamInfo].

[0111] The device parameter (torch.device or int, optional) can be used to set the specified device. For example, the user can set which device to monitor or which processing chip installed on the device to monitor the communication flow. If None (default value), the current device returned by a function (such as current_device()) can be selected.

[0112] The parameter List[StreamInfo] indicates a list consisting of StreamInfo class objects, that is, the elements in the list are StreamInfo class objects, and each StreamInfo class object contains a Stream class object and a str class object. Among them, the Stream class object is also called a communication stream object, such as "cncl_stream(Stream)", and the str class object indicates the unique communication domain ID bound to the communication stream object, such as "clique_id(str)".

[0113] The specific process of obtaining the communication flow using the above custom interface is as follows:

[0114] First, a global variable (e.g., cncl_streams) can be maintained in the distributed backend of the source code (e.g., C++ source code) of the PyTorch framework. The global variable is used to synchronize the communication streams bound to the device index that are created and destroyed in the process group backend. In addition, the global variable can be a nested map class object (e.g., an object of the std::unordered_map class). The outer keyword of the global variable represents the device index, the inner keyword represents the communication domain ID, and the inner value is the communication stream.

[0115] Next, when creating a communication stream bound to a device in the process group backend of the PyTorch framework, a function (such as the updateCnclQueue function) may be called to synchronize the device index, the communication stream, and the communication domain ID to the global variable cncl_streams.

[0116] In addition, the communication flow to be destroyed is also stored in the global variable. In response to the existence of a destruction operation of the communication flow to be destroyed, the information related to the communication flow to be destroyed in the global variable is cleared synchronously. For example, when the process group backend destroys the communication flow bound to the device, a function (such as the clearCnclQueue function) can be called to synchronously clear the relevant information of the communication flow in the global variable.

[0117] Then, the map class object bound to the device index can be obtained from the global variable according to the device index based on the transit function supported by the machine learning framework. For example, the first custom interface above calls the getCnclQueue function of the distributed C++ backend through the transit function such as the Python C extension function THMPModule_getCnclStream_wrap, and obtains the corresponding communication domain ID and the communication stream composition inner std::unordered_map class object according to the device index.

[0118] Finally, the obtained map class object can be format converted and returned to the first custom interface. For example, the Python C extension function can convert the communication domain ID and the communication flow into a corresponding format and return it to the first custom interface.

[0119] With the help of the first custom interface mentioned above, the specific process of monitoring the communication flow status in Python code is as follows: Fig. 9 As shown, including:

[0120] In step S902, the process group is initialized.

[0121] At step S904, a loop is entered. In this embodiment, after the process group is initialized, for example, a distributed network training task may be triggered to start a loop training process.

[0122] At step S906, it is necessary to determine whether the distributed training is the first iteration. The communication flow is different from the calculation flow. Generally, the calculation flow will be assumed to exist on the device. When you need to monitor a calculation flow, you can directly call the interface to switch to the calculation flow for monitoring. The communication flow is created when the program performs a communication operation on the specified device for the first time. If you want to successfully obtain the communication flow bound to the device, you need to perform a communication operation on the device first. Otherwise, calling the interface returns an empty list. In other words, the relevant communication flow cannot be obtained. Therefore, when it is determined that the distributed training is the first iteration, it is necessary to execute step S908 to execute the distributed network training code first. If the distributed training is not the first iteration, it means that the communication flow already exists, and step S910 can be executed.

[0123] At step S910, a custom interface (API) may be used to obtain the communication flow. For example, the communication flow list bound to each process and the device may be obtained through the custom API. The specific acquisition process of the communication flow may refer to the relevant description of the first custom interface above, which will not be repeated here.

[0124] At step S912, the start event and the end event may be set. For example, the setting of the start event and the end event may be completed by optimizing the native Event class of the PyTorch framework or redefining a new second custom interface. The details of the process in this embodiment may be specifically referred to the relevant details in the previous text, and will not be repeated here.

[0125] At step S914, the start event is recorded to obtain a timestamp of the start event.

[0126] At step S916, the distributed network training code is executed to complete this iteration process.

[0127] At step S918, the end event may be recorded to obtain a timestamp of the end event. For example, the start and end events on the communication flow may be recorded before and after the distributed network training code.

[0128] At step S920 , the E2E time of the communication flow and / or the operator hardware time of the communication operator in each generation of training may be obtained.

[0129] At step S922, it is determined whether the loop is finished. If finished, step S924 is executed, and if not finished, the loop is continued.

[0130] At step S924, after the loop ends, the process group can be destroyed.

[0131] Through the above monitoring process, the running status of each generation of distributed network training communication flow and / or communication operator can be accurately perceived. Among them, the above process only requires slight changes to the original distributed network training code to achieve stable and efficient monitoring of the communication flow status. In addition, since only events on the communication flow are recorded, a more fine-grained guidance idea is provided for the stable monitoring of the communication status.

[0132] Based on the above description, those skilled in the art can understand that the present application actually discloses a device, which includes a processor and a memory. Specifically, the memory can store program instructions for monitoring flows in a machine learning framework. When the program instructions are executed by the processor, the present application is combined with Figure 7-Figure 9 In addition, since the solution of the present application can be implemented by computer program instructions, the present application also discloses a computer-readable storage medium or a computer program product, on which a computer program / instruction for monitoring a flow in a machine learning framework is stored, thereby realizing the combination Figure 7-Figure 9 The method steps described.

[0133] The disclosed solution supports real-time start of monitoring of various flows (such as computing flows, communication flows, or other flows that need to be monitored) in a machine learning framework in a variety of business production environments such as model reasoning or training, without suspending the business, thereby ensuring the efficiency of the business production environment. At the same time, in the process of implementing flow monitoring, the additional performance overhead generated by the interface technology and plugging technology used is much smaller than the performance overhead of the "heavy" tools in traditional technologies, thereby achieving lightweight flow monitoring.

[0134] In addition, the solution disclosed in the present invention is not limited to obtaining the end-to-end time of the computing flow, but can also accurately perceive the hardware time of the operator on the business flow. During the implementation of the above process, only a slight change to the original business code is required to achieve stable and efficient monitoring of the flow state. At the same time, the additional performance overhead generated by this method is almost negligible compared to the E2E time overhead of the business, and the function of enabling and continuously monitoring the hardware time in the production environment can be achieved by default. In addition, under the customized API and the related mechanism of the event after the extension is inserted, compared with the traditional method, the impact on the CPU time overhead is almost negligible, and it can be enabled and continuously monitored by default in the business production environment, making it truly "lightweight".

[0135] The above is a detailed description of the scheme of the present disclosure in conjunction with the accompanying drawings. According to different application scenarios, the equipment or device disclosed herein may include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an automatic driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric light, a gas stove, and a range hood; the medical device includes a nuclear magnetic resonance apparatus, a B-ultrasound apparatus, and / or an electrocardiograph. The equipment or device disclosed herein can also be applied to the Internet, the Internet of Things, a data center, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical treatment and other fields.

[0136] Furthermore, the devices or apparatuses disclosed herein can also be used in application scenarios related to artificial intelligence, big data and / or cloud computing, such as the cloud, edge, and terminal. In one or more embodiments, devices or apparatuses with high power consumption according to the disclosed solution can be applied to cloud devices (such as cloud servers), while devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0137] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0138] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the device or apparatus embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0139] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0140] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory ("Read Only Memory", abbreviated as ROM), a random access memory ("Random Access Memory", abbreviated as RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0141] In some other implementation scenarios, the above-mentioned integrated units may also be implemented in the form of hardware, that is, as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the hardware structure of the circuit may include, but is not limited to, physical devices, and the physical devices may include, but are not limited to, devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) may be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which may be, for example, resistive random access memory ("Resistive Random Access Memory", abbreviated as RRAM), dynamic random access memory ("Dynamic Random Access Memory", abbreviated as DRAM), static random access memory ("Static Random Access Memory", abbreviated as SRAM), enhanced dynamic random access memory ("Enhanced Dynamic Random Access Memory", abbreviated as "EDRAM"), high bandwidth memory ("High Bandwidth Memory", abbreviated as "HBM"), hybrid memory cube ("Hybrid Memory Cube", abbreviated as "HMC"), ROM and RAM, etc.

[0142] The foregoing content can be better understood in accordance with the following terms:

[0143] Clause A1. A method for monitoring flows in a machine learning framework, comprising:

[0144] Acquire a target flow to be monitored in the machine learning framework based on a target interface supported by the machine learning framework;

[0145] Performing instrumentation processing on the target flow to obtain instrumentation information about the target flow; and

[0146] According to the instrumentation information about the target flow, end-to-end time information of the target flow and / or hardware time information of the flow operators in the target flow are determined.

[0147] Clause A2. The monitoring method according to clause A1, wherein the target interface includes a first custom interface or a default interface, and obtaining a target flow to be monitored in the machine learning framework based on the target interface supported by the machine learning framework includes:

[0148] In response to the target flow including an explicitly created computing flow, acquiring the explicitly created computing flow based on the default interface; or

[0149] In response to the target flow including a non-explicitly created communication flow, the non-explicitly created communication flow is acquired based on the first custom interface.

[0150] Clause A3. The monitoring method according to clause A2, wherein obtaining the non-explicitly created communication flow based on the first custom interface comprises:

[0151] A communication flow list related to the device index is obtained based on the first custom interface, wherein the communication flow list includes communication flow information objects bound to the device index, and each communication flow information object is distinguished based on the communication domain to which its inner communication flow belongs.

[0152] Clause A4. The monitoring method according to clause A3, wherein obtaining a communication flow list related to the index of the device based on the first custom interface comprises:

[0153] Obtaining a map class object bound to the specified device index from the global variable according to the device index based on a transfer function supported by the machine learning framework; and

[0154] The obtained map class object is format converted to obtain the communication flow information object, and returned to the first custom interface.

[0155] Clause A5. The monitoring method according to Clause A4 is characterized in that the map class object includes an outer keyword representing a device index, an inner keyword representing an identifier of a communication domain, and an inner value representing a communication flow.

[0156] Clause A6. The monitoring method according to clause A4, wherein the global variable also stores the communication flow to be destroyed, and the monitoring method further comprises:

[0157] In response to the existence of a destroy operation on the communication flow to be destroyed, information related to the communication flow to be destroyed in the global variable is synchronously cleared.

[0158] Clause A7. The monitoring method according to any one of Clauses A1 to A6, characterized in that the instrumentation process is performed on the target flow to obtain instrumentation information about the target flow, including:

[0159] Create an Event event or a second custom interface supported by the machine learning framework, so that the Event event or the second custom interface supports hardware time monitoring of stream operators in the target stream; and insert the Event event or the second custom interface into the code of the target stream to obtain a timestamp associated with the Event event or the second custom interface.

[0160] Clause A8. The monitoring method according to Clause A7 is characterized in that the Event event is obtained based on the extension of the native Event event supported by the machine learning framework, the Event event and the second custom interface both include a start time function and an end time function, and inserting the Event event or the second custom interface into the code of the target flow to obtain a timestamp related to the Event event or the second custom interface includes:

[0161] The start time function and the end time function are respectively inserted into corresponding positions in the code of the target stream to obtain a timestamp of the start time function and a timestamp of the end time function.

[0162] Clause A9. The monitoring method according to clause A8, characterized in that the monitoring method further comprises:

[0163] Determining end-to-end time information of the target flow according to the instrumentation information about the target flow;

[0164] Wherein, determining the end-to-end time information of the target flow and / or the hardware time information of the flow operator in the target flow includes:

[0165] The end-to-end time information of the target stream or the hardware time information of the stream operator in the target stream is determined according to the difference between the timestamp of the end time function and the timestamp of the start time function.

[0166] Clause A10. The monitoring method according to Clause A9 is characterized in that the machine learning framework includes a PyTorch framework, and the stream operator in the target stream includes a computing operator or a communication operator.

[0167] Clause A11. A method for monitoring flows in a machine learning framework, characterized in that it includes: acquiring a communication flow to be monitored in the machine learning framework based on a target interface supported by the machine learning framework; performing instrumentation processing on the communication flow to obtain instrumentation information about the communication flow; and determining end-to-end time information of the communication flow and / or hardware time information of communication operators in the communication flow based on the instrumentation information about the communication flow.

[0168] Clause A12. An apparatus for monitoring a flow in a machine learning framework, comprising:

[0169] one or more processors; and

[0170] A memory storing computer instructions for monitoring flows in a machine learning framework, wherein when the computer executable instructions are executed by the one or more processors, the device executes a method as described in any one of clauses A1-A10 or clause A11.

[0171] Item A13. A computer-readable storage medium storing computer instructions for monitoring flows in a machine learning framework, wherein when the computer instructions are executed by one or more processors, a method as described in any one of Items A1-A10 or Item A11 is executed.

[0172] Although the embodiments of the present disclosure are as described above, the contents are only examples used to facilitate understanding of the present disclosure, and are not intended to limit the scope and application scenarios of the present disclosure. Any technician in the technical field described in the present disclosure may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be based on the scope defined in the attached claims.

Claims

1. A method for monitoring flows in a machine learning framework, characterized in that: include: Acquire a target flow to be monitored in the machine learning framework based on a target interface supported by the machine learning framework; Performing instrumentation processing on the target flow to obtain instrumentation information about the target flow; as well as According to the instrumentation information about the target flow, hardware time information of the flow operator in the target flow is determined.

2. The monitoring method according to claim 1, characterized in that: The target interface includes a first custom interface or a default interface, and obtaining a target flow to be monitored in the machine learning framework based on the target interface supported by the machine learning framework includes: In response to the target flow including an explicitly created computing flow, acquiring the explicitly created computing flow based on the default interface; or In response to the target flow including a non-explicitly created communication flow, the non-explicitly created communication flow is acquired based on the first custom interface.

3. The monitoring method according to claim 2, characterized in that: Acquiring the non-explicitly created communication flow based on the first custom interface includes: A communication flow list related to a specified device index is obtained based on the first custom interface, wherein the communication flow list includes communication flow information objects bound to the specified device index, and each communication flow information object is distinguished based on a communication domain to which its inner communication flow belongs.

4. The monitoring method according to claim 3, characterized in that: Acquiring a communication flow list related to a specified device index based on the first custom interface includes: Obtaining a map class object bound to the specified device index from a global variable based on the transfer function supported by the machine learning framework according to the specified device index; and The obtained map class object is format converted to obtain the communication flow information object, and returned to the first custom interface.

5. The monitoring method according to claim 4, characterized in that: The map class object includes an outer key representing a device index, an inner key representing an identifier of a communication domain, and an inner value representing a communication flow.

6. The monitoring method according to claim 4, characterized in that: The global variable also stores a communication flow to be destroyed, and the monitoring method further includes: In response to the existence of a destroy operation on the communication flow to be destroyed, information related to the communication flow to be destroyed in the global variable is synchronously cleared.

7. The monitoring method according to any one of claims 1 to 6, characterized in that: Performing instrumentation processing on the target flow to obtain instrumentation information about the target flow includes: Creating an Event event or a second custom interface supported by the machine learning framework, so that the Event event or the second custom interface supports hardware time monitoring of the stream operator in the target stream; and Insert the Event event or the second custom interface into the code of the target flow to obtain a timestamp related to the Event event or the second custom interface.

8. The monitoring method according to claim 7, characterized in that: The Event event is obtained based on the native Event event supported by the machine learning framework. The Event event and the second custom interface both include a start time function and an end time function. Inserting the Event event or the second custom interface into the code of the target flow to obtain a timestamp related to the Event event or the second custom interface includes: The start time function and the end time function are respectively inserted into corresponding positions in the code of the target stream to obtain a timestamp of the start time function and a timestamp of the end time function.

9. The monitoring method according to claim 8, characterized in that: The monitoring method further comprises: Determining end-to-end time information of the target flow according to the instrumentation information about the target flow; Wherein, determining the end-to-end time information of the target flow and / or the hardware time information of the flow operator in the target flow includes: The end-to-end time information of the target stream or the hardware time information of the stream operator in the target stream is determined according to the difference between the timestamp of the end time function and the timestamp of the start time function.

10. The monitoring method according to claim 9, characterized in that: The machine learning framework includes a PyTorch framework, and the stream operator in the target stream includes a computing operator or a communication operator.

11. A method for monitoring flows in a machine learning framework, characterized in that: include: Acquire a communication flow to be monitored in the machine learning framework based on a target interface supported by the machine learning framework; Performing instrumentation processing on the communication flow to obtain instrumentation information about the communication flow; as well as According to the instrumentation information about the communication flow, end-to-end time information of the communication flow and / or hardware time information of the communication operators in the communication flow are determined.

12. A device for monitoring a flow in a machine learning framework, comprising: one or more processors; as well as A memory storing computer instructions for monitoring flows in a machine learning framework, wherein when the computer executable instructions are executed by the one or more processors, the device executes the method as described in any one of claims 1 to 10 or claim 11.

13. A computer-readable storage medium storing computer instructions for monitoring flows in a machine learning framework, wherein when the computer instructions are executed by one or more processors, the method according to any one of claims 1 to 10 or claim 11 is executed.

Images