Methods and products for evaluating performance of a distributed training system in a single node environment

Abstract

This disclosure discloses a method for evaluating the performance of a distributed training system in a single-node environment, wherein the method can be implemented in a computing device, which may be located in a combined processing unit, which may further include a general interconnect interface and other processing units. The compilation unit interacts with the other processing units to jointly perform user-specified computational operations. The combined processing unit may further include a storage device connected to both the computing device and the other processing units for storing data from the computing device and the other processing units.

Description

Technical Field

[0001] This disclosure relates to the field of testing, and more specifically, to the performance testing of neural networks. Background Technology

[0002] Scalability is a crucial performance metric for distributed training of deep neural network models across multiple nodes. To calculate the scalability of distributed training for a deep neural network, we need to know the time it takes for the network to converge to error in a single-node hardware environment (hereinafter referred to as the first training time T1), and the second training time T2 when the network is trained for the same number of iterations in a multi-node hardware environment using the same data and parameters. By definition, scalability = T1 / (T2*M), where M is the number of nodes in the neural network model. These nodes can be accelerators, processors, accelerator cards, etc., used in the neural network.

[0003] Ideally, if the first training time for a single node is t1, then the second training time for M nodes is T2 = T1 / M, so the system's scalability is 1 in the ideal case. However, in reality, due to system startup time, communication time, etc., the second training time for M nodes cannot be a perfect T2 = T1 / M, but is higher than T2 = T1 / M. Therefore, the closer the system's scalability is to 1, the better its scalability.

[0004] There are several methods for testing scalability. One method is to run the program in a multi-node hardware environment and measure the training time of a specified deep neural network. Another method is to use performance analysis software to capture the network training timeline while the network is training on a single node, and then combine this with the measured performance of the communication library software in a multi-node hardware environment to insert the communication overhead into the timeline, ultimately estimating the training time of the network in a multi-machine, multi-GPU hardware environment.

[0005] All of the above methods require the program to actually run in a multi-node hardware environment. This means that when hardware resources are insufficient (e.g., the required multi-node hardware environment is not available), the scalability of distributed training of deep neural networks cannot be achieved.

[0006] In addition, the second method requires manually finding the location of communication behavior on the timeline, which may lead to prediction accuracy issues and inefficiency. Summary of the Invention

[0007] To at least partially address the technical problems mentioned in the background section, this disclosure provides a method and related products for evaluating the performance of a distributed training system in a single-node environment.

[0008] In one aspect, this disclosure provides a method for evaluating the performance of a distributed training system in a single-node environment, wherein the distributed training system includes multiple nodes, the method comprising: determining the types of operations involved in the distributed training system; determining the time cost of each operation based on the type of operation; and determining the performance of the distributed training system based on the time cost of each operation.

[0009] In another aspect, an electronic device is provided, comprising: one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described above.

[0010] In yet another aspect, a computer-readable storage medium is provided, comprising computer-executable instructions that, when executed by one or more processors, perform the method as described above.

[0011] One advantage of this disclosure is that it enables the testing of the scalability of distributed training of multiple nodes in a deep neural network system using a single node (e.g., a processor). Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0013] Figure 1a and Figure 1b A schematic diagram of a multi-core processor according to one embodiment of the present disclosure is shown;

[0014] Figure 1c A flowchart of a method for evaluating the performance of a distributed training system in a single-node environment according to one embodiment of the present disclosure is shown.

[0015] Figure 2 A schematic diagram of the communication alignment operation is shown;

[0016] Figure 3 A flowchart illustrating the determination of time overhead based on different operation types according to one embodiment of the present disclosure is shown;

[0017] Figure 4a and Figure 4b The diagrams show AllReduce communication in a ring topology and AllReduce communication in a binary tree structure, respectively.

[0018] Figure 5 A schematic diagram illustrating data being divided into slices for transmission according to one embodiment of the present disclosure is shown;

[0019] Figure 6 A comparison diagram of the actual operation process and the simulated operation process according to one embodiment of the present disclosure is shown;

[0020] Figure 7a This diagram illustrates a distributed training system controlled via a kernel under normal conditions.

[0021] Figure 7b A schematic diagram of simulating a distributed training system via a kernel according to the present disclosure is shown;

[0022] Figure 8 A combined processing apparatus is shown; and

[0023] Figure 9 A board is shown. Detailed Implementation

[0024] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0025] In the scheme described in this disclosure, testers can evaluate the overall performance of a distributed training system using a single hardware node, without needing a distributed training system comprising multiple hardware nodes. This allows testers to test the performance of the entire distributed training system using a single node. It should be understood that the single node described herein can be a multi-core processor, which may include, for example, multiple processing cores capable of executing multiple different tasks or operations in parallel.

[0026] Figure 1a and Figure 1b A schematic diagram of a multi-core processor according to one embodiment of the present disclosure is shown, wherein, Figure 1a A schematic diagram of the internal structure of a processor group to which the methods of this disclosure can be applied is shown.

[0027] Artificial intelligence (AI) chips accelerate data processing capabilities and reduce memory access latency. Employing a multi-core processor architecture, AI chips support up to 16 cores of parallel computing and incorporate on-chip or internal memory cores to accelerate data retrieval, thus resolving the memory access bottleneck between the processor cores and DDR (external memory). This provides users with enhanced computing power in scenarios such as deep learning and network computing.

[0028] The AI ​​chip has 16 processor cores for performing computational tasks. Four processor cores form a processor group, resulting in four processor groups in total. Each processor group contains one memory core. The memory core is primarily used for data exchange between shared memory units within the processor group and the processor cores, as well as for data exchange between processor groups. When both the memory core and the processor core access DDR simultaneously, arbitration via a multiplexer ensures that only one bus accesses DDR at a time.

[0029] Figure 1b A schematic diagram of the structure of an artificial intelligence processor to which the method of this disclosure can be applied is shown. Figure 1b As shown, the AI ​​chip's DDR uses a Non-Uniform Memory Access (NUMA) architecture. Each processor group can access different DDR channels through NOC0, but the latency of accessing different DDR channels varies. Each processor group corresponds to a DDR channel with the lowest access latency, while accessing other channels has a relatively longer latency. Figure 1b As shown in the processor group and DDR architecture diagram, processor groups 0, 1, 2, and 3 have the lowest latency when accessing their respective DDR0, DDR1, DDR2, and DDR3 memory channels. In other words, each processor core accesses the DDR channel with the lowest memory access latency within its respective processor group.

[0030] Since the memory access bandwidth within the processor group is higher than the access bandwidth between the processor core and DDR, AI chips can improve data throughput by using the processor group to internally access shared memory units, thereby reducing the direct access of the processor core to DDR.

[0031] When 4-core parallel computing is required, the memory core can broadcast data from the shared memory to all 4 processor cores in the processor group simultaneously via NOC1 for computation. Compared to all processor cores reading data via DDR, this reduces memory access latency and optimizes computational performance.

[0032] As computing demands increase, 16 processor cores may need to handle multiple computing tasks simultaneously. Direct access to DDR by the processor cores inevitably causes data access latency, leading to slow computing speeds and other issues. AI chips, through data exchange between the processor groups, avoid direct communication between the 16 processor cores and DDR, thereby reducing data access latency.

[0033] Figure 1a and Figure 1b Exemplary nodes (multi-core processors) that can be used in this disclosure are shown, but this does not constitute any limitation on the technical solutions of this disclosure. Those skilled in the art can implement the technical solutions of this disclosure using any suitable single-core or multi-core processor.

[0034] Figure 1c A flowchart of a method for evaluating the performance of a distributed training system in a single-node environment according to one embodiment of the present disclosure is shown, wherein the distributed training system includes multiple nodes, the method comprising: in operation S110, determining the types of operations involved in the distributed training system; in operation S120, determining the time cost of each operation based on the type of operation; and in operation S130, determining the performance of the distributed training system based on the time cost of each operation.

[0035] A distributed training system can include multiple nodes, which can form various topologies or systems through hardware or software methods. In each topology, the order and frequency of communication between nodes may differ, resulting in varying time overhead.

[0036] In this distributed training system, operations are broadly categorized into two types: computational operations and non-computational operations. Non-computational operations are also referred to as multi-node operations in this application. Computational operations occur within a node and can be executed by a single node; however, non-computational operations do not necessarily occur within a single node but may occur between multiple nodes. Alternatively, non-computational operations may be waiting operations; therefore, non-computational operations are also referred to as multi-node operations in this paper. It is important to understand that the term "operation" as used herein does not necessarily refer to an action; any possible process or procedure can be called an operation. For example, the process of a node waiting for another node to initiate communication, although no "action" occurs, can still be called an "operation."

[0037] Computational operations refer to the process by which nodes perform calculations on data. They do not involve data transmission or communication between nodes, nor do they involve initiation or alignment operations during communication. Computational operations incur corresponding computational overhead, which is the time cost of performing calculations on the data allocated to a single node during data-parallel distributed training. These overheads include, but are not limited to: the time spent on forward operator computation, loss function calculation, and gradient backpropagation during network training; the overhead of data reading, data preprocessing, API calls, and computation on the framework side; and the time spent on inference verification during training.

[0038] Non-computational operations are mostly related to multiple nodes and refer to various operations involved in communication between multiple nodes, excluding computational operations. These operations include, but are not limited to, communication operations and communication alignment operations. Communication operations include communication initiation operations and data transmission operations, etc. Communication initiation operations are an inherent process of inter-node communication, that is, the process from communication call alignment to the start of communication in this distributed training system. Such operations also incur certain time overhead, which is referred to as communication initiation time overhead in this application.

[0039] Data transmission operations refer to the process of data transmission between multiple nodes. In distributed training, data transmission between different nodes incurs corresponding time overhead. This time overhead is a major component of distributed training, and it is specifically related to factors such as topology, number of nodes, data size, and transmission bandwidth, which will be explained in detail later.

[0040] The time overhead caused by communication alignment operations refers to the time cost incurred when multiple communication nodes invoke communication operations at different times. This factor should also be taken into account when considering the time overhead of non-computational operations.

[0041] Figure 2 A schematic diagram of a communication alignment operation is shown. According to one embodiment of this disclosure, the communication alignment time overhead is the difference between the first communication start time of the first node initiating the communication operation and the second communication start time of the last node initiating the communication operation in the distributed training system.

[0042] like Figure 2 As shown, assume the distributed training system includes four nodes: node 1, node 2, node 3, and node 4. Each node performs computation and communication operations, with communication operations occurring after the corresponding computation operations. Specifically, node 1 performs computation operation 1 and communication operation 1, node 2 performs computation operation 2 and communication operation 2, node 3 performs computation operation 3 and communication operation 3, and node 4 performs computation operation 4 and communication operation 4.

[0043] like Figure 2 As shown, communication operation 1 of node 1 is called at time t1, therefore node 1 is the first of the four nodes to call the communication operation; node 3 is the second to call communication operation 3; after node 3, node 4 is the third to call communication operation 4; then, node 2 calls communication operation 2 last at time t2. According to the embodiment of this disclosure, the communication alignment time overhead is t2-t1.

[0044] Next, after determining the type of each operation, the time cost of that operation can be determined based on its type. Since each operation has different types and characteristics, their time costs also vary.

[0045] According to one embodiment of this disclosure, the types of operations include: computational operations and multi-node operations, and wherein determining the time cost of each operation according to the type of operation includes: instructing the computational operation to run in a single node to determine the computational time cost of the computational operation in response to the type of operation being a computational operation; and determining the communication time cost and communication alignment time cost for communication by simulating multi-node operations in a distributed training system in response to the type of operation being a multi-node operation.

[0046] Figure 3 A flowchart illustrating the determination of time overhead based on different operation types according to one embodiment of this disclosure is shown.

[0047] like Figure 3 As shown, after training begins, the type of each operation is first determined to identify whether it is a computational operation or a multi-node operation; in other words, whether the operation involves multi-node communication. If the operation is determined to be a computational operation, it can be actually run on a single node, thus directly obtaining the time cost of the computational operation. If the operation is determined to be a multi-node operation, its time cost can be determined by simulating multi-node operations in a distributed training system, i.e., determining the communication alignment time cost and the communication time cost. Next, it can be determined whether all operations involved in the training have been completed. If they have been completed, the training ends; if there are still unfinished operations, the unfinished operations are judged in the next round. According to one embodiment of this disclosure, operations involving calling communication primitive interfaces of a communication library can be considered as multi-node operations; other operations that can be completed within a single node are considered as computational operations. Such a judgment criterion is merely an example and not a limitation on the technical solution of this disclosure. It should be understood that there is no necessary order in which the time cost is determined in this document. Figure 3 The flowchart in the image is merely an example.

[0048] The types of operations have been introduced above, which can be broadly divided into computational operations and non-computational operations (or multi-node operations). According to the embodiments of this disclosure, as described above, for computational operations, a single node can be used to actually run the corresponding computation process, and the time cost of the computational operation is determined by detecting the time consumed by this computation process. Since computational operations are detected by actually running the corresponding node, the detection of computational operations is relatively accurate.

[0049] According to this implementation, the data involved in the computation can be loaded into the single node for actual execution, so as to obtain the corresponding computational overhead. According to one embodiment of this disclosure, the single-time computation overhead for the single-time computation data volume N / n is calculated by dividing the data volume N of a single iteration of the distributed training system by the number of nodes n. The computational overhead for the single-time computation data volume N / n is then determined based on the single-time computation overhead and the number of iterations of the distributed training system.

[0050] In a distributed training system, training or other computations on data may involve multiple iterations, each processing a certain amount of data. For example, assuming a distributed training system has n nodes and the data volume in a single iteration is N, then theoretically each node handles N / n of the data. According to the embodiments of this disclosure, to more accurately obtain the corresponding computational overhead, it is preferable to form a training set of size N, and then split this training set into n parts for each individual node to actually run, thereby making it as similar as possible to the real operating environment. In contrast, it is also feasible to directly use data of size N / n, but compared to the preferred implementation of splitting the training set of size N into n parts, the latter is closer to the actual operating environment. Therefore, in this disclosure, to more accurately obtain various time costs, it is necessary to simulate the real operating environment as closely as possible.

[0051] After obtaining the time cost of a single computation, it is necessary to further determine the computational time cost of the computation operation based on the number of iterations of the distributed training system. Those skilled in the art can determine the corresponding number of iterations based on the total amount of data involved in the computation and the amount of data involved in each computation.

[0052] However, if the operation type is a multi-node operation, the training system cannot be run because there is only a single hardware node. Therefore, it is not possible to determine the actual time cost of the multi-node operation by actually running the multi-node operation of the training system, as is the case when the computation operation is actually run on a single node.

[0053] Therefore, when the type of operation is a multi-node operation, its time cost can be determined by simulating multi-node operations in a distributed training system, that is, the communication time cost and the communication alignment time cost can be determined.

[0054] According to one embodiment of this disclosure, determining the communication alignment time overhead by simulating multi-node operations in a distributed training system includes using a random number from a specific probability distribution as the communication alignment time. Statistical analysis shows that the communication alignment time of distributed training systems with different topologies exhibits a certain probability distribution, rather than being fixed. More specifically, this probability distribution is a Gaussian distribution. Based on actual testing, a random number can be randomly selected between 0-2ms using a Gaussian distribution as the communication alignment time overhead.

[0055] The communication time overhead is described below. According to one embodiment of this disclosure, the communication time overhead is a function of the following factors: communication startup time overhead, number of nodes, inherent time overhead of cross-node communication, data size, and transmission bandwidth.

[0056] Communication startup time overhead is a statistically significant constant, corresponding to the overhead of startup, control, and other actions of the communication system. It still exists when the communication data is close to zero and there is only one communication node. For ease of description, this paper refers to communication startup time overhead as "overhead".

[0057] The number of nodes is a significant factor in communication time overhead; the more nodes there are, the greater the communication time overhead may be. For ease of description, this article will denote the number of nodes as node_num.

[0058] The inherent time overhead of cross-node communication refers to the system control overhead incurred when communication data is transmitted between nodes. It exists even when the communication data is close to zero and is a quantifiable quantity. For ease of description, this paper refers to the inherent time overhead of cross-node communication as latency.

[0059] Data size refers to the amount of data involved in data transmission. For ease of description, this article will refer to data size as data_size.

[0060] Transmission bandwidth is related to the communication medium used. For ease of description, transmission bandwidth will be referred to as bandwidth in this article.

[0061] According to one embodiment of this disclosure, when the distributed training system performs broadcast communication through a ring topology, the function is: communication time overhead = overhead + (node_num-1)*(latency + data_size / bandwidth).

[0062] In a ring topology, data is relayed. Therefore, apart from the fixed overhead, the various time costs of the entire ring network are proportional to node_num-1, because the entire ring communication can be broken down into node_num-1 data transmissions. For example, when there are 4 nodes, it takes 3 data transmissions from the first node to the last node to ensure that each node can process the data.

[0063] According to one embodiment of this disclosure, when the distributed training system performs AllReduce communication through a ring topology, the function is: communication time overhead = overhead + 2 * (node_num - 1) * (latency + data_size / (node_num * bandwidth)).

[0064] According to one embodiment of this disclosure, when the distributed training system performs AllReduce communication through a binary tree topology, the function is: communication time overhead = overhead + 2 * (log2(node_num)) * (latency + data_size / bandwidth).

[0065] In the embodiments of this disclosure, various topologies and communication primitives of the distributed training system can be modeled to determine the communication model. Then, based on the established communication model, various data are sampled and statistically analyzed to obtain more accurate values ​​of the above factors, thereby facilitating the analysis of the time cost of the distributed training system for each topology.

[0066] Figure 4a and Figure 4b The diagrams show AllReduce communication in a ring topology and AllReduce communication in a binary tree structure, respectively.

[0067] like Figure 4aAs shown, assume the data involved in the computation in the four communication nodes are as follows: in node 0, data 1, 2, 3, and 4 are processed; in node 1, data 5, 6, 7, and 8 are processed; in node 2, data 9, 10, 11, and 12 are processed; and in node 3, data 13, 14, 15, and 16 are processed. The final computation result for each node includes the computation results of all data (the sum of the corresponding data, e.g., 1+5+9+13, 2+6+10+14, 3+7+11+15, 4+8+12+18). In the ring topology, AllReduce includes a Scatter-Reduce phase and an All-Gather phase. For the sake of simplicity, the entire process of AllReduce will not be described in detail in this paper.

[0068] like Figure 4b As shown, assume the data involved in the computation in the seven communication nodes are as follows: in node 0, operations are performed on data 1, 2, 3, and 4; in node 1, operations are performed on data 5, 6, 7, and 8; in node 2, operations are performed on data 9, 10, 11, and 12; and in node 3, operations are performed on data 13, 14, 15, and 16. Nodes 4, 5, and 6 are initially empty. The final computation result includes the computation results of all data in each node (the sum of the corresponding data, for example, 1+5+9+13, 2+6+10+14, 3+7+11+15, 4+8+12+18). In the binary tree topology, AllReduce includes a Reduce phase and a Broadcast phase. For the sake of simplicity, the entire process of AllReduce will not be described in detail in this paper.

[0069] The aforementioned time overhead can be determined using α-β models or LogP models, etc. This paper only exemplifies the communication time overhead of AllReduce communication in ring and binary tree topologies, but this disclosure is not limited to the aforementioned topologies and communication primitives. Many other types of topologies are possible, including but not limited to butterfly, recursive halving and doubling, segmented ring, hierarchical RingAllReduce, 2D Torus, 2D-mesh, 3D Torus, etc. The communication time overhead of all these topologies can be obtained using corresponding models.

[0070] The foregoing described a model of communication time overhead. According to one embodiment of this disclosure, the communication time overhead can also be a function of the following factors: communication startup time overhead, number of data slices, number of nodes, inherent time overhead of cross-node communication, data size, and transmission bandwidth.

[0071] The factors such as communication startup time overhead, number of nodes, inherent time overhead of cross-node communication, data size, and transmission bandwidth have already been introduced above, so they will not be repeated here. Next, we will introduce the factor "number of data slices" in detail.

[0072] Figure 5 A schematic diagram is shown illustrating how data is divided into slices for transmission according to one embodiment of the present disclosure.

[0073] like Figure 5 As shown, assuming there are three nodes, namely node 0, node 1, and node 2, the data is divided into eight data slices 0-7. For ease of description, the slice stored in node 0 is represented as "data slice 0n"; the slice stored in node 1 is represented as "data slice 1n"; and the slice stored in node 2 is represented as "data slice 2n", where n is 0 to 7. It should be understood that using different numbers for the slices stored in each node is merely for ease of description and does not imply that the data has necessarily undergone other transformation processes.

[0074] like Figure 5 As shown, assume that the data in node 0 is divided into eight slices, namely data slice 00-data slice 07, while nodes 1 and 2 are empty. In the first round of data transmission, data slice 00 is transmitted to node 1 and stored as data slice 10, while no data transmission occurs between node 1 and node 2. Next, in the second round of data transmission, data slice 01 in node 0 is transmitted to node 1, and simultaneously, data slice 10 in node 1 is transmitted to node 2 and stored as data slice 20. This continues in multiple rounds, with data slices being transmitted from node 0 to node 1 in each round, and the data slices transmitted from node 0 to node 1 in the previous round being further transmitted to node 2. This continues until the eighth round of data transmission, where the final data slice 07 is transmitted from node 0 to node 1 and stored as data slice 17, while data slice 16 in node 1 is transmitted to node 2 and stored as data slice 26. In the final round of data transmission, no data transmission occurs between node 0 and node 1, and data slice 17 in node 1 is transmitted to node 2 and stored as data slice 27.

[0075] Figure 5This merely illustrates one example of transmitting data in slices. The specific transmission order of these slices may differ depending on the topology and communication primitives (e.g., broadcast, AllReduce, etc.), resulting in varying overall time overhead. For example, for... Figure 5 In some cases, dividing the data into eight data slices and dividing the data into four data slices result in different time costs. Therefore, the number of data slices is also a factor affecting the overall time cost.

[0076] Based on the above description, factors affecting time overhead can be identified, including but not limited to the computation time overhead of a single node, as well as the communication startup time overhead involving multiple nodes, the number of nodes, the inherent time overhead of cross-node communication, data size, and transmission bandwidth. Regarding the computation time overhead of a single node, as mentioned above, it can be obtained directly by actually performing the corresponding computation operation on that single node. However, according to one embodiment of this disclosure, the time overhead of multi-node operations in a distributed training system can be simulated using a hibernation program.

[0077] In the embodiments of this disclosure, since multi-node operations cannot be actually run by a single node, a sleep kernel can be used to simulate the multi-node operation by putting the corresponding communication resources into a dormant state. This method includes, but is not limited to, using a sleep kernel to simulate the corresponding time occupancy on the device queue. A sleep kernel occupies hardware computing resources but does not perform actual calculations; it simply idles the kernel program for a specified time.

[0078] It should be understood that, according to one embodiment of this disclosure, when running the hibernation program, the hibernation program occupies the corresponding resources of a single node, rather than all resources, to simulate multi-node operation in a distributed training system.

[0079] like Figure 1a and Figure 1b As shown, each node may contain multiple processing cores. Each processing core may participate in computational operations or non-computational operations. Therefore, in the actual simulation process, it is also necessary to simulate the actual resource usage in reality.

[0080] Figure 6 A comparison diagram of the actual operation process and the simulated operation process according to one embodiment of the present disclosure is shown.

[0081] like Figure 6As shown, in actual communication, it is assumed that a node has four processing cores C0-C3, and that the first convolution operation Conv0 requires three processing cores C0, C1 and C2 to participate in the computation operation, while processing core C3 remains idle. In this case, in simulated communication, since Conv0 is a computation operation, it can actually run on a single machine. Therefore, the behavior of occupancy of processing cores is consistent with that in actual communication.

[0082] Next, after the calculation is complete, the AllReduce communication process begins. In actual communication, only core C0 participates, while cores C1, C2, and C3 are idle. Meanwhile, in simulated communication, the hibernation program puts core C0 into a hibernation state (occupied), while cores C1, C2, and C3 remain idle. It's important to understand that this hibernation state means that core C0 is occupied and cannot be called by other programs.

[0083] Next, the second convolution operation Conv2 is performed. In actual communication, the processing kernel C0 is still occupied by AllReduce communication, so processing kernels C1 and C2 will be called. In simulated communication, the same situation should be simulated, that is, the sleeping program still occupies the processing kernel C0, keeping it in a sleeping state, while when running the second convolution operation, only the processing kernels C1 and C2 participate in the calculation operation.

[0084] In this embodiment, the single-node computation operation and the asynchronous call behavior of multi-node operation during actual distributed training can be simulated, thereby improving the accuracy of the simulation.

[0085] Furthermore, according to one embodiment of this disclosure, the computational operations and multi-node operations can be set up in an operation queue on a single node so as to simulate the performance of the distributed training system based on the time cost of each operation by running the operation queue.

[0086] In traditional distributed training systems, the kernel program can control the operation sequence of each node and the entire distributed training system. Figure 7a This diagram illustrates a typical scenario where a distributed training system is controlled via a kernel. Figure 7b A schematic diagram illustrating a distributed training system simulated via a kernel according to this disclosure is shown.

[0087] like Figure 7a As shown, in a conventional system, one can first programmatically build the kernel to configure multiple nodes. Once the required kernel is built, as... Figure 7aAs shown, a kernel can be sent to one or more nodes, allowing multiple nodes to form a corresponding topology based on the kernel and perform computation and communication based on that topology. It's important to understand that when a kernel is sent to a node, that node can further forward the kernel to other nodes, so that each node can operate under the control of the kernel.

[0088] In the technical solution disclosed herein, such as Figure 7b As shown, a node's operation queue, such as computational operations and non-computational operations, can be set in the kernel, and this kernel can be sent to the single node to simulate, for example... Figure 7a The multi-node diagram is shown. For ease of understanding, in... Figure 7b Solid lines represent actual hardware nodes, while dashed lines represent other nodes to be simulated. By executing this kernel, a single node can perform corresponding computational and non-computational operations according to the instructions of the operation queue. That is, if the queue reaches a computational operation, the node can actually perform that computational operation; if the queue reaches a non-computational operation, the node can consume a specific amount of time through a sleep procedure (the Sleep Kernel can be a subroutine in the overall kernel or a separate program).

[0089] By adding the operation queue to the program and sending the program to the single node mentioned above, the single node can automatically execute the kernel, thereby automatically obtaining the time cost of each operation, and thus making it easier to determine the scalability of the system based on these time costs.

[0090] This disclosure also provides an electronic device, including: one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the electronic device to perform the method described above.

[0091] This disclosure also provides a computer-readable storage medium including computer-executable instructions that, when executed by one or more processors, perform the method described above.

[0092] The technical solution disclosed herein can predict the training time of a deep neural network in a multi-node hardware environment using only a single hardware node, and further predict the corresponding scalability.

[0093] Furthermore, the inventors discovered several time overheads that are not easily noticeable when simulating deep neural networks, and through careful design of the technical solution disclosed herein, enabled accurate and sufficient prediction of the corresponding time overheads.

[0094] The technical solution disclosed herein can be applied to the field of artificial intelligence, and can be implemented as or in an artificial intelligence chip. This chip can exist independently or be included in a computing device.

[0095] Figure 8 A combined processing apparatus 800 is shown, comprising the aforementioned computing device 802, a general interconnect interface 804, and other processing devices 806. The computing device according to this disclosure interacts with the other processing devices to jointly perform user-specified operations. Figure 8 This is a schematic diagram of the combined processing device.

[0096] Other processing devices include one or more processor types such as central processing unit (CPU), graphics processing unit (GPU), and neural network processor. There is no limit to the number of processors included in other processing devices. These other processing devices serve as interfaces between the machine learning computing device and external data and control, including data transfer and basic control such as starting and stopping the machine learning computing device. Other processing devices can also collaborate with the machine learning computing device to complete computational tasks.

[0097] A general interconnect interface is used to transfer data and control commands between a computing device (including, for example, a machine learning computing device) and other processing devices. The computing device can obtain required input data from other processing devices and write it to on-chip storage; it can obtain control commands from other processing devices and write them to on-chip control caches; and it can read data from the computing device's storage modules and transmit it to other processing devices.

[0098] Optionally, the structure may further include a storage device 808, which is connected to both the computing device and the other processing device. The storage device is used to store data in the computing device and the other processing device, and is particularly suitable for data that cannot be fully stored in the internal storage of the computing device or other processing device, requiring computation.

[0099] This combined processing device can serve as a System-on-a-Chip (SoC) for devices such as mobile phones, robots, drones, and video surveillance equipment, effectively reducing the core area of ​​the control unit, increasing processing speed, and lowering overall power consumption. In this case, the universal interconnect interface of the combined processing device connects to certain components of the device, such as cameras, monitors, mice, keyboards, network cards, and Wi-Fi interfaces.

[0100] In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip.

[0101] In some embodiments, this disclosure also discloses a circuit board that includes the above-described chip packaging structure. See also... Figure 9 The present invention provides an exemplary board, which, in addition to the chip 902, may also include other supporting components, including but not limited to: a storage device 904, an interface device 906, and a controller 908.

[0102] The storage device is connected to the chip within the chip package structure via a bus for storing data. The storage device may include multiple sets of storage cells 910. Each set of storage cells is connected to the chip via a bus. It is understood that each set of storage cells may be DDR SDRAM (Double Data Rate SDRAM).

[0103] DDR can double the speed of SDRAM without increasing the clock frequency. DDR allows data to be read on both the rising and falling edges of the clock pulse. DDR is twice as fast as standard SDRAM. In one embodiment, the storage device may include four groups of storage cells. Each group of storage cells may include multiple DDR4 chips. In one embodiment, the chip may internally include four 72-bit DDR4 controllers, of which 64 bits are used for data transmission and 8 bits are used for ECC verification. In one embodiment, each group of storage cells includes multiple Double Data Rate Synchronous Dynamic Random Access Memory (DRAM) units connected in parallel. DDR can transmit data twice per clock cycle. A controller for controlling DDR is provided in the chip for controlling data transmission and data storage in each storage cell.

[0104] The interface device is electrically connected to the chip within the chip package structure. The interface device is used to enable data transmission between the chip and an external device 912 (e.g., a server or computer). For example, in one embodiment, the interface device can be a standard PCIe interface. For instance, data to be processed is transferred from the server to the chip via a standard PCIe interface, realizing data transfer. In another embodiment, the interface device can also be other interfaces; this disclosure does not limit the specific form of these other interfaces, as long as the interface unit can perform the switching function. Furthermore, the calculation results of the chip are still transmitted back to the external device (e.g., the server) by the interface device.

[0105] The controller is electrically connected to the chip. The controller monitors the state of the chip. Specifically, the chip and the controller can be electrically connected via an SPI interface. The controller may include a microcontroller (MCU). The chip may include multiple processing chips, multiple processing cores, or multiple processing circuits, capable of driving multiple loads. Therefore, the chip can operate in different states, such as high load and low load. The controller can regulate the operating states of multiple processing chips, multiple processing cores, and / or multiple processing circuits within the chip.

[0106] In some embodiments, this disclosure also discloses an electronic device or apparatus that includes the aforementioned board.

[0107] Electronic devices or apparatuses include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, dashcams, navigators, sensors, cameras, servers, cloud servers, cameras, camcorders, projectors, watches, headphones, mobile storage, wearable devices, vehicles, home appliances, and / or medical devices.

[0108] The means of transportation include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical equipment includes MRI scanners, ultrasound scanners and / or electrocardiographs.

[0109] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0110] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0111] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, optical, acoustic, magnetic, or other forms.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Furthermore, the functional units in the various embodiments disclosed herein can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0114] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, when the technical solution disclosed herein can be embodied in the form of a software product, this computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0115] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A method for evaluating the performance of a distributed training system in a single-node environment, wherein the distributed training system comprises multiple nodes, the method comprising: Determine the types of operations involved in the distributed training system; The time cost of each operation is determined based on the type of operation. as well as The performance of the distributed training system is determined based on the time cost of each operation. The types of operations include computational operations and multi-node operations, and determining the time cost of each operation based on its type includes: In response to the operation being a computational operation, the computational operation is instructed to be run on a single node to determine the computational time cost of the computational operation; In response to the operation being a multi-node operation, the communication time overhead and communication alignment time overhead are determined by simulating multi-node operations in a distributed training system.

2. The method according to claim 1, wherein, The single computation time cost for the single computation data volume N / n is calculated by dividing the data volume N of a single iteration of the distributed training system by the number of multiple nodes n. The computation time cost of the computation operation is determined based on the single computation time cost and the number of iterations of the distributed training system.

3. The method according to claim 1, wherein, The communication alignment time overhead is the difference between the first communication start time of the first node that calls the communication operation and the second communication start time of the last node that calls the communication operation in the distributed training system. The communication alignment time overhead was determined by simulating multi-node operations in a distributed training system, including using random numbers from a Gaussian distribution as the communication alignment time.

4. The method according to claim 3, wherein, The communication alignment time overhead is a random number within a specific time range, which is 0-2ms.

5. The method according to claim 1, wherein, The communication time overhead is a function of the following factors: communication startup time overhead, number of nodes, inherent time overhead of cross-node communication, data size, and transmission bandwidth.

6. The method according to claim 5, wherein, When the distributed training system uses a ring topology for broadcast communication, the function is: Communication time overhead = Communication startup time overhead + (Number of nodes - 1) (Inherent time overhead + data size / transmission bandwidth).

7. The method according to claim 5, wherein, When the distributed training system uses a ring topology for AllReduce communication, the function is: Communication time overhead = Communication startup time overhead + 2 (Number of nodes - 1) (Inherent time overhead + data size / (number of nodes)) Transmission bandwidth).

8. The method according to claim 5, wherein, When the distributed training system uses a binary tree topology for AllReduce communication, the function is: Communication time overhead = Communication startup time overhead + 2 (log2(number of nodes)) (Inherent time overhead + data size / transmission bandwidth).

9. The method according to claim 1, wherein, The communication time overhead is a function of the following factors: communication startup time overhead, number of data slices, number of nodes, inherent time overhead of cross-node communication, data size, and transmission bandwidth.

10. The method according to claim 1, wherein, The time overhead of multi-node operations in a distributed training system is simulated by using a hibernation procedure.

11. The method according to claim 10, wherein, By using a hibernation program to occupy the corresponding resources of a single node, the operation of multiple nodes in a distributed training system can be simulated.

12. The method according to any one of claims 1-11, wherein, The computational operations and multi-node operations are set up in an operation queue on a single node so that the performance of the distributed training system can be simulated by running the operation queue, based on the time cost of each operation.

13. An electronic device, comprising: One or more processors; as well as A memory storing computer-executable instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-12.

14. A computer-readable storage medium comprising computer-executable instructions that, when executed by one or more processors, perform the method as described in any one of claims 1-12.

Images