Training analysis method, device, storage medium and electronic device

Abstract

The present disclosure relates to a training analysis method, device, storage medium and electronic device to achieve fine-grained communication operator tracking and align tracking timestamps from different computing units to improve the analysis accuracy of distributed training. The method includes: determining the global dependency graph of a deep neural network; determining the time offset of multiple processors relative to a reference processor with the goal of minimizing the time alignment function and the constraint that the operation dependencies between nodes in the global dependency graph before and after time alignment remain unchanged; simulating the distributed training process of the deep neural network offline based on the global dependency graph and the time offset, and determining the distributed training analysis results of the deep neural network based on the results of the offline simulation.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular, to a training analysis method, device, storage medium, and electronic device. Background Art

[0002] Distributed training for large datasets has been widely used in machine learning models, such as deep neural networks (DNNs), to support a variety of intelligent technology-driven applications. Compared with single-node training, using multiple processors to perform distributed training on the same machine learning model can reduce the training time cost.

[0003] In the related art, the analysis of distributed training is mainly based on the bandwidth and tensor size to estimate the communication time of synchronizing tensors between processors. Among them, tensors are data generated during distributed training, such as gradients generated during distributed training. This coarse-grained estimation method treats the synchronization of a tensor as a black box, without distinguishing between communication queuing time and actual communication transmission time, and thus cannot accurately analyze the specific training details of distributed training. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a training analysis method for analyzing a distributed training process of a deep neural network, wherein the deep neural network runs on multiple devices, the method comprising:

[0006] Determine a global dependency graph of the deep neural network, wherein the global dependency graph is used to characterize the dependency relationship between computing nodes in the deep neural network running on each processor of the device and the communication relationship between communication nodes between the plurality of processors for synchronizing tensors generated by the computing nodes, and each two processors in the global dependency graph include a plurality of communication nodes;

[0007] With the goal of minimizing a time alignment function and with the constraint that the operation dependencies between the nodes in the global dependency graph remain unchanged before and after the time alignment, determine the time offsets of the multiple processors relative to a reference processor, wherein the time alignment function is used to characterize the time difference of the processors receiving tensors from the same processor and the time offset difference between processors in the same device, and the reference processor is any processor included in the multiple devices;

[0008] According to the global dependency graph and the time offset, the distributed training process of the deep neural network is simulated offline, and according to the result of the offline simulation, the distributed training analysis result of the deep neural network is determined.

[0009] In a second aspect, the present disclosure provides a training analysis apparatus for analyzing a distributed training process of a deep neural network, wherein the deep neural network runs on multiple devices, the apparatus comprising:

[0010] A first determination module is configured to determine a global dependency graph of the deep neural network, wherein the global dependency graph is configured to characterize dependency relationships between computing nodes in the deep neural network running on a processor of each of the devices and communication relationships between communication nodes between the plurality of processors for synchronizing tensors generated by the computing nodes, and each two processors in the global dependency graph include a plurality of communication nodes;

[0011] A second determination module is used to determine the time offsets of the plurality of processors relative to a reference processor with the goal of minimizing a time alignment function and with the constraint that the operation dependencies between the nodes in the global dependency graph remain unchanged before and after the time alignment, wherein the time alignment function is used to characterize the time difference of the processors receiving tensors from the same processor and the time offset difference between processors in the same device, and the reference processor is any processor included in the plurality of devices;

[0012] A training simulation module is used to simulate the distributed training process of the deep neural network offline according to the global dependency graph and the time offset, and determine the distributed training analysis result of the deep neural network according to the result of the offline simulation.

[0013] In a third aspect, the present disclosure provides a non-temporary computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0014] In a fourth aspect, the present disclosure provides an electronic device, including:

[0015] a storage device having a computer program stored thereon;

[0016] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect.

[0017] Through the above technical solution, the distributed training process of the deep neural network can be simulated and analyzed offline based on the global dependency graph and time offset. Among them, the global dependency graph includes multiple communication nodes between each two processors. Compared with the related technology that treats the tensor synchronization between processors as a black box, it is equivalent to splitting a tensor synchronization process into multiple communication sub-processes, so that the communication queuing time and the actual communication transmission time can be distinguished, and a more accurate distributed training time can be obtained, thereby improving the offline analysis accuracy of distributed training. On the other hand, the time offset is obtained by minimizing the time alignment function, and the time alignment function represents the time difference between the processor receiving the tensor from the same processor and the time offset difference between the processors in the same device. Therefore, the timestamps of offline simulation training on different processors can be aligned offline, providing a more accurate communication transmission time for the synchronization of each tensor, and further improving the analysis accuracy of distributed training.

[0018] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0020] Figure 1 is a flow chart of a training analysis method according to an exemplary embodiment of the present disclosure;

[0021] Figure 2 is a schematic diagram of a global dependency graph in a training analysis method according to an exemplary embodiment of the present disclosure;

[0022] Figure 3 is a schematic diagram of a key path in a training analysis method according to an exemplary embodiment of the present disclosure;

[0023] Figure 4 is a process schematic diagram of a training analysis method according to an exemplary embodiment of the present disclosure;

[0024] Figure 5 is a block diagram of a training analysis device according to an exemplary embodiment of the present disclosure;

[0025] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should also be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] All actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0031] Distributed training for large datasets has been widely used in machine learning models, such as deep neural networks (DNNs), to support a variety of intelligent technology-driven applications. Compared with single-node training, using multiple devices to perform distributed training on the same machine learning model can reduce the training time cost.

[0032] In the related technology, the analysis of deep neural network jobs (i.e., the training process of deep neural networks) mainly uses an analyzer to collect the trajectory of each operation in the job (mainly the execution time of the operation), and analyzes the dependencies between the operations to obtain the dependency graph corresponding to the job. The simulator then uses the results collected by the analyzer to simulate the operation of the job and obtain a performance evaluation of the job.

[0033] The inventors have found that in a distributed training scenario, the relevant technology can collect the trace of each operation in the job through hardware analysis tools, built-in analyzers of the machine learning framework, and analyzers in the communication library, and analyze the dependencies between the operations to obtain the dependency graph corresponding to the job. Among them, the hardware analysis tool can collect the start or end time of the processor core, GPU (Graphics Processing Unit) memory usage and other hardware information. In addition, the hardware analysis tool can also collect relevant analysis information when the job is running. However, this type of hardware analysis tool relies on hardware and does not capture the dependencies between operators in deep neural networks, which makes it challenging to parse kernel-level traces.

[0034] For machine learning frameworks, built-in profilers are provided, such as TensorFlow, PyTorch, and MXNet. These profilers collect trace information for operators in deep neural networks, including the execution time and memory consumption of each operator. TensorFlow and MXNet profilers also collect coarse-grained communication trace information for their distributed APIs (Application Programming Interface), including the start and end times of communication operators. However, these tools do not exclude the communication queuing time in the communication library, so the actual communication transmission time cannot be obtained.

[0035] For the analyzer in the communication library, two communication schemes (i.e., parameter synchronization schemes) are widely used in distributed training in related technologies: (1) AllReduce, that is, the processors participating in the training in distributed training (also called workers, usually a GPU) are connected in a tree or ring topology, and use the corresponding communication method to synchronize the aggregated gradients, and then update the training parameters; (2) Parameter server (PS) architecture, that is, the processors participating in the training push the local gradients generated by themselves to the PS, and then extract the aggregated gradients from the PS. Among them, the AllReduce communication scheme can treat the entire synchronization task of the tensor as a single operation, and collect the start time and duration of such synchronization tasks on a single processor. The PS communication scheme allows tensor partitioning, and its analyzer can collect the time spent on pulling tensors from the PS and the time spent on uploading tensors to the PS. However, neither of these two communication schemes supports the tracking of computing operators.

[0036] Alternatively, related techniques can monitor the gradient noise scale by inserting monitoring operations into the dependency graph and using a collective communication layer to aggregate the gradient noise scale between processors. However, this approach also does not track the execution time of operators and the dependencies between operators.

[0037] In addition, the inventors also found that the analyzer of the related art does not support timestamp alignment between the processor and the parameter server. Even through clock synchronization tools, such as NTP (Network Time Protocol) or other more precise tools, there is millisecond or sub-millisecond clock drift between the processor and the parameter server. Moreover, the analysis tools in the related art can only capture the start time of the RECV operation (ie, the tensor receiving operation), but cannot obtain the exact time when the receiving data starts. Without accurate timestamp alignment, the errors in the communication timestamps will accumulate, thereby increasing the error in the end-to-end performance estimation.

[0038] In summary, in the distributed training scenario, the analyzer in the related art has the problem of not being able to support operator tracking and timestamp alignment. In addition, the analyzer in the related art mainly estimates the communication time of synchronizing tensors between processors based on bandwidth and tensor size. This coarse-grained estimation method treats the synchronization of a tensor as a black box, without distinguishing between communication queuing time and actual communication transmission time, making it impossible for the simulator to accurately simulate the running time of distributed training jobs, and thus unable to accurately evaluate the performance of distributed training jobs.

[0039] In view of this, the present disclosure provides a training analysis method to provide a general distributed training analysis method, distinguish between communication queuing time and actual communication transmission time, realize fine-grained communication operator tracking, and align tracking timestamps from different processors, so as to provide a more accurate global time for distributed deep neural network training, so that the simulator can more accurately simulate the running time of distributed training jobs, and then perform a more accurate performance evaluation of distributed training jobs.

[0040] Figure 1 FIG. 1 is a flowchart of a training analysis method according to an exemplary embodiment of the present disclosure. Figure 1 The training analysis method can be used to analyze the distributed training process of a deep neural network running on multiple devices, including the following steps:

[0041] Step 101, determine a global dependency graph of a deep neural network. The global dependency graph is used to characterize the dependency relationship between computing nodes in a deep neural network running on a processor of each device and the communication relationship between communication nodes for synchronizing tensors generated by computing nodes between multiple processors, and each two processors in the global dependency graph include multiple communication nodes.

[0042] Step 102, with the goal of minimizing the time alignment function, and with the constraint that the operation dependencies between the nodes in the global dependency graph before and after the time alignment remain unchanged, determine the time offsets of multiple processors relative to the reference processor. The time alignment function is used to characterize the time difference of the processor receiving the tensor from the same processor and the time offset difference between the processors in the same device, and the reference processor is any processor included in the multiple devices.

[0043] Step 103, based on the global dependency graph and the time offset, offline simulate the distributed training process of the deep neural network, and determine the distributed training analysis result of the deep neural network based on the result of the offline simulation.

[0044] First of all, it should be understood that the implementation scenario of the training analysis method in the embodiment of the present disclosure can be to obtain the performance data of the online training (such as the time overhead of each computing node and communication node, etc.) after the deep neural network running on multiple processors is trained online. Then, an offline simulation is performed based on the performance data to obtain the distributed training analysis results of the deep neural network, and to achieve offline optimization of the deep neural network. Among them, the deep neural network can be a deep neural network that can realize different application functions such as text recognition, speech recognition, and image recognition. It should be understood that if the deep neural network is a deep neural network for realizing the image recognition function, the computing operations and communication operations mentioned in the present disclosure are all operations for image data. Of course, if the deep neural network is a deep neural network for realizing other functions, the computing operations and communication operations mentioned in the present disclosure are all operations for corresponding types of data.

[0045] It should also be understood that the processor in the embodiment of the present disclosure may be a processor with a hardware structure, such as a GPU. Accordingly, the processor of the device may be all GPUs included in the device. Alternatively, the processor in the embodiment of the present disclosure may be a process for performing a corresponding computing function. Accordingly, the processor of the device may be a process for running a deep neural network, which is not limited in the embodiment of the present disclosure.

[0046] In a possible manner, step 101 may be: determining a local dependency graph of a deep neural network running on a processor of each device, the local dependency graph being used to characterize the dependency relationship between computing nodes in the deep neural network, and the local dependency graph including a first virtual operation identifier for indicating the start of tensor synchronization and the end of tensor synchronization. Then, determining a communication topology of multiple processors, the communication topology being used to characterize the communication relationship between communication nodes for synchronizing tensors generated by computing nodes between multiple processors, and the communication topology including a second virtual operation identifier for indicating the start of tensor synchronization and the end of tensor synchronization. Finally, the local dependency graph and the communication topology are associated through the first virtual operation identifier and the second virtual operation identifier having the same identification information to obtain a global dependency graph.

[0047] That is to say, the establishment of the global dependency graph can be divided into two parts: the local dependency graph and the communication topology, such as Figure 2 As shown, the local dependency graph corresponds to the computational part in distributed training. Figure 2 Each circle in the local dependency graph represents a computing node, corresponding to a layer of network structure in the deep neural network. The local dependency graph can be established according to the network structure of the deep neural network, or can be established by referring to the method of generating dependency graphs in machine learning frameworks such as TensorFlow and MxNet, which will not be repeated here. In the embodiment of the present disclosure, in order to establish an association between the local dependency graph and the communication topology, the local dependency graph may include a first virtual operation identifier for indicating the start and end of tensor synchronization. For example, referring to Figure 2 , in the local dependency graph, a first virtual operation identifier A1 is used to indicate the start of tensor synchronization and a first virtual operation identifier B1 is used to indicate the end of tensor synchronization. It should be understood that, Figure 2 Virtual communication operations are represented by dashed triangles.

[0048] Continue to refer to Figure 2 , the communication topology is used to describe how tensors are transferred between processors on a device, Figure 2 The nodes in represent the processors of the device (i.e., the computing units involved in training in distributed training, such as GPUs or processing processes, also called workers). Figure 2 The dashed triangle in the figure represents a virtual communication operation. Figure 2 The solid triangle in represents a communication node, and one communication node corresponds to one communication operation (a sending operation or a receiving operation).

[0049] In the disclosed embodiment, the communication topology mainly includes two types of communication operations: 1) the producer sends a certain tensor to other processors; 2) the consumer receives a certain tensor from other processors. A unique transaction ID can be used to mark each transmission of a tensor between any two processors, and the producer is connected to the corresponding consumer with the same transaction ID. Figure 2 A triangle in the communication topology represents a communication operation (a sending operation or a receiving operation).

[0050] In the PS architecture, each tensor upload operation and tensor pull operation is regarded as a pair of send and receive operations between the processor and the parameter server. Accordingly, the transaction ID can be generated by the sender or receiver IP address, the tensor name, and the operation type identifier, where the operation type identifier is used to characterize that the operation corresponding to the transaction ID is a tensor upload operation or a tensor pull operation. In the AllReduce architecture, a pair of send and receive operations can be used to represent the transmission of the tensor to the next processor (i.e., worker) on the AllReduce ring. Accordingly, the transaction ID can be generated by the block ID of the tensor partition and the step ID in the AllReduce ring. Among them, the block ID of the tensor partition is used to identify the tensor corresponding to each operation, and the step ID is used to identify which two processors in the AllReduce ring each operation is for. The relevant content of the AllReduce ring can refer to the relevant technology, which will not be repeated here.

[0051] In the embodiment of the present disclosure, in order to associate the local dependency graph with the communication topology, the communication topology of each tensor may include a pair of input or output virtual operations marked with the tensor name, indicating the start or end of the tensor transmission, that is, the second virtual operation identifier for indicating the start of tensor synchronization and the end of tensor synchronization. For example, the second virtual operation identifier A2 indicating the start of tensor synchronization and the second virtual operation identifier A2 for indicating the end of tensor synchronization Figure 2 , used to identify B2 in the communication topology.

[0052] Afterwards, the local dependency graph and the communication topology are associated through the first virtual operation identifier and the second virtual operation identifier having the same identification information, and a global dependency graph can be obtained. For example, continuing with the above example, the first virtual operation identifier A1 and the second virtual operation identifier A2 have the same identification information "A", so there is a connection relationship between the first virtual operation identifier A1 and the second virtual operation identifier A2. At the same time, the first virtual operation identifier B1 and the second virtual operation identifier B2 have the same identification information "B", so there is a connection relationship between the first virtual operation identifier B1 and the second virtual operation identifier B2. In this way, the local dependency graph and the communication topology can be associated to obtain a global dependency graph.

[0053] Through the above method, the construction of the global dependency graph is decoupled into two parts: the local dependency graph and the communication topology, and the local dependency graph and the communication topology are associated through virtual operation identifiers with the same identification information. When the processors participating in distributed training remain unchanged, when the local dependency graph varies due to different machine learning frameworks, there is no need to modify the communication topology, and it can support various machine learning frameworks and communication libraries, reducing the cost of communication modification.

[0054] In addition, refer to Figure 2 , each two processors in the global dependency graph include multiple communication nodes. Compared with the related technology that regards tensor synchronization between processors as a black box, it is equivalent to splitting a tensor synchronization process into multiple communication sub-processes, so that the communication queuing time and the actual communication transmission time can be distinguished, and a more accurate distributed training time can be obtained, which improves the accuracy of offline analysis of distributed training.

[0055] After obtaining the global dependency graph, the time offset of multiple processors relative to the reference processor can be determined through the time alignment function, and the start time of the RECV operation (i.e., tensor receiving operation) can be corrected offline to obtain a more accurate communication duration for each tensor.

[0056] For example, let W and P represent the set of processors (i.e., workers) and PSs in distributed training, respectively. It should be understood that for the AllReduce architecture, P is empty. and They represent the start and end timestamps of the operation op on the node i∈P∪ measured in the initial trajectory, and denote the adjusted start and end timestamps respectively. Wherein, node i is a processor or PS. Assume that node 0 is the reference node for other nodes, that is, all other nodes are aligned to the time axis of node 0. Accordingly, the time offset θ of node i can be expressed as the difference between the measured time of node i and node 0. i ,Right now Based on this, a time alignment function can be established in the following way to align the timestamps between nodes.

[0057] First, the operations (i.e., RECV operations) of the same node receiving the same tensor from the same sender under different training iterations should have similar execution times, and these receiving operations can be called belonging to the same receiving operation series. For a pair of tensor sending operations and tensor receiving operations from node i to node j, the start time of the tensor sending operation can be Adjust the start time of tensor receive operations Therefore, the time for receiving the tensor in the time alignment function can be determined by: determining the end time of the first processor performing the tensor receiving operation, and determining the latest time between the start time of the first processor performing the tensor receiving operation and the start time of the second processor performing the tensor sending operation, wherein the end time, the start time of the first processor performing the tensor receiving operation, and the start time of the second processor performing the tensor sending operation are all represented by the time offset of the corresponding processor relative to the reference processor, and the first processor and the second processor are different. Then, the time difference between the end time of the first processor performing the tensor receiving operation and the latest time is determined as the time when the first processor receives the tensor from the second processor.

[0058] For example, the time to receive a tensor, that is, the execution time of the tensor receiving operation, can be obtained from becomes

[0059] It should be understood that in actual applications, there is a tensor receiving operation that starts before a tensor sending operation. In this case, if the execution time of the tensor receiving operation is determined by the start time of the tensor receiving operation, the determined execution time will be longer than the actual execution time. Therefore, the embodiment of the present disclosure reduces the time determination error in the above manner, thereby improving the accuracy of subsequent calculations.

[0060] In the disclosed embodiment, the first goal is to minimize the variance of the execution time of the tensor receiving operation in the same receiving operation series by adjusting the time offset θ on the time axis:

[0061]

[0062] in, represents the set of operations corresponding to all receiving operation series, f recv represents a series of receive operations, j represents the node that performs the tensor receive operation, and i represents the node that performs the tensor send operation.

[0063] Second, since processors on the same device share the same physical clock, processors on the same device should have the same time offset. Let M be the set of all devices, g m is the node set corresponding to the processor on device m. The time offset difference between processors on the same device is calculated according to the following formula:

[0064]

[0065] Therefore, the time offset of the processors included in the multiple devices relative to the reference processor, that is, the time offset θ of the time alignment between the distributed nodes, can be determined by solving the following optimization problem:

[0066]

[0067] stθ0=0

[0068]

[0069] Where a1 and a2 are weights for measuring two objectives, and a1, a2 ≥ 0, and E is the set of dependencies between operations. It is used to ensure that the operation dependencies between nodes in the global dependency graph remain unchanged before and after time alignment. That is, assuming that operation o2 on node j in the global dependency graph depends on operation o1 on node i in the global dependency graph, the time after the former adjustment is shall not be earlier than the adjusted time of the latter The optimization problem can be solved using optimization tools in related technologies, such as CVXPY, etc., which is not limited in the embodiments of the present disclosure.

[0070] After obtaining the time offset, the distributed training process of the deep neural network can be simulated and analyzed offline based on the time offset and the global dependency graph.

[0071] In a possible manner, according to the global dependency graph and the time offset, the distributed training process of the deep neural network can be simulated offline, which can be: first, each processor corresponds to an operation queue, and the sending operation of each tensor in the global dependency graph corresponds to an operation queue, and the receiving operation of each tensor corresponds to an operation queue, the operation queue is used to store the operations to be executed, and the operation queue has a queue time represented by the time offset, and the queue time is the end time of the last operation performed by the processor or tensor communication process corresponding to the operation queue. Then, determine the target operation queue with the earliest queue time, execute the head operation stored in the target operation queue, and update the queue time of the target operation queue based on the end time of the execution of the head operation, and return to the step of determining the target operation queue with the earliest queue time, until all operations in the local dependency graph are executed, and the latest queue time is determined in the queue times corresponding to multiple operation queues as the iteration time of the global dependency graph. Accordingly, according to the results of the offline simulation, determining the distributed training analysis results of the deep neural network can be: determining the distributed training analysis results of the deep neural network according to the iteration time.

[0072] When the present disclosure is implemented, the execution of the global dependency graph can be simulated based on the Kahn topological sorting algorithm. It should be understood that the simulation process in the related art uses a global ready queue to store all currently executable operations. In the embodiment of the present disclosure, for distributed training jobs, each processor (i.e., worker), PS, and communication process can be regarded as a virtual device, and a ready queue and queue time are maintained for each virtual device. Among them, the communication process is a tensor communication process corresponding to a tensor sending operation or a tensor receiving operation, and the queue time is the end time of the last operation executed on the virtual device. Once an operation is ready (i.e., all dependent operations are completed), it is queued into the queue of the corresponding virtual device. Then, the virtual device with the smallest queue time can be continuously selected and an operation can be taken out from the head of its device queue (i.e., the head of the queue) for execution, and the corresponding queue time can be updated with the execution time of the operation. After all operations in the global dependency graph have run to completion, the maximum queue time can be used as the iteration time of the global dependency graph to evaluate its performance.

[0073] It should be understood that a global dependency graph may have multiple feasible topological sortings, but the embodiment of the present disclosure has performed multiple iterative training on the deep neural network before simulation (for example, an average of 10 iterative trainings), that is, the trajectory information in the global dependency graph is obtained through multiple training iterations, and the embodiment of the present disclosure uses a first-in-first-out (FIFO) queue, so that the most likely topological sorting can be quickly generated, thereby improving the efficiency of offline analysis.

[0074] In actual applications, after using the simulator to simulate the operation of a distributed training job and obtain a performance evaluation of the job, the job can also be optimized, such as changing the operations in the dependency graph, such as replacing several operations with one operation, or changing the time information of some operations.

[0075] The inventors carefully studied and found that for this kind of optimization, the relevant technologies mainly include communication optimization methods and computing optimization methods. Among them, the communication optimization methods include: 1) fusing the communication operations corresponding to multiple small tensors to reduce the communication overhead in gradient synchronization; 2) dividing a large tensor into multiple tensors for communication, so as to perform tensor upload operations and tensor pull operations in parallel in the PS architecture; 3) reducing the gradient size through various compression methods; 4) defining a tensor transmission scheduling algorithm to better perform computing operations and communication operations in parallel. The computing optimization methods include: 1) fusing multiple computing operators into an overall operator to reduce operator scheduling overhead; 2) using 16-bit floating point numbers instead of 32-bit floating point numbers as the digital format for most computing operations to reduce memory consumption and faster computing.

[0076] For communication optimization, in actual applications, the granularity of tensor fusion can be set by setting environment variables, that is, tensors are fused into new tensors of specified size for transmission, or all tensors within a specific time interval are fused. However, no matter which method is used, it depends on manually specifying these environment variables and cannot be searched automatically. Although the relevant technology proposes an automatic search method based on Bayesian search, this search is an online search process, which takes up the time of the actual training process and cannot be automatically searched offline.

[0077] In addition, the inventors also found that there is a conflict between the effects of different optimization methods. For example, the DL compiler uses operator fusion technology to reduce GPU memory access, but its default solution may fuse all back-propagation operators, which will delay the communication operations of tensors corresponding to these direction propagation operators. Although the calculation time will be shorter after the fusion of operators, the end-to-end training time may increase due to the less overlap between calculation and communication. In addition, some memory optimization techniques also interact with calculation and communication, but the recalculation of intermediate results sacrifices training speed to reduce memory usage and may delay the communication of tensors.

[0078] Moreover, there is a very large space of combination strategies between different optimization methods. Specifically, in the actual optimization process, the optimizer needs to search for the best optimization technology for each operator and how to implement the combination of these optimization technologies. However, the global dependency graph corresponding to the distributed training of deep neural networks contains a huge number of operators, so it is very time-consuming and laborious to manually find the best optimization strategy.

[0079] Therefore, the present disclosure proposes an optimization search framework for distributed training tasks, which can add custom optimization techniques and automatically search for the optimal optimization solution.

[0080] In a possible manner, a global execution graph can be first generated according to a global dependency graph, and a critical path can be determined according to the global execution graph, and the global execution graph is used to indicate the order of execution of operations between nodes with dependencies in the global dependency graph. Then, with the goal of minimizing the operation execution time of nodes on the critical path, an optimization strategy for the global dependency graph is determined, and the optimization strategy is executed on the global dependency graph to obtain an optimized dependency graph, wherein the nodes on the critical path include communication group nodes and computing nodes in a deep neural network, and the communication group nodes include multiple communication nodes. Accordingly, according to the global dependency graph and the time offset, the distributed training process of offline simulation of the deep neural network can be: according to the optimized dependency graph and the time offset, the distributed training process of the deep neural network is offline simulated.

[0081] It should be understood that in the disclosed embodiment, given a global dependency graph G of a distributed DNN training job and a set of optimization strategies S, the goal is to identify bottlenecks in the global dependency graph through offline simulation and generate a subset S of optimization strategies that minimize the training time per iteration (referred to as iteration time) * ∈S:

[0082]

[0083] Where G′=f(G,S ′ ) is the application S ′ The global dependency graph modified after the optimization strategy.

[0084] In the disclosed embodiment, the key path of the global execution graph is continuously and iteratively checked and optimized, thereby determining the corresponding optimization strategy for each node on the key path. The key path C may include a series of calculation and communication operations: C = [p0, p1, ..., p i ,q i ,q i+1 ,…,q |C|-1 ], where p0, p1, …, p i is a computational operation, q i ,q i+1 ,…,q |C|-1 is a communication operation, and |C| is the total number of nodes on the critical path, that is, the length of the critical path. To facilitate the analysis of the relationship between communication and computation, the fine-grained communication operations (e.g., a set of corresponding tensor send operations and tensor receive operations) used to transmit tensors n in the global execution graph can be abstracted into a communication operation group q n . Figure 3 The critical path (illustrated by the node path enclosed by the dashed line) and each pair of operations p are depicted. n and q n The correspondence between them, n = 0, 1, ..., |C|-1. Since the generation of tensors depends on computational operations, the critical path can start from a computational operation sequence and end with a communication operation sequence. On the one hand, each computational operation p on the critical path n (n=1,...,i), may correspond to a communication operation group q n , but this communication operation group is not on the critical path because the computation operation at this stage is the bottleneck. On the other hand, each communication operation q on the critical path n (n=i,...,|C|-1) corresponds to a computation operation p that is not on the critical path n , because communication is the bottleneck in this stage.

[0085] In each iteration, we can check from p0 to q |c|-1On the critical path, for each operation on the critical path, determine the corresponding optimization strategy. n To indicate that when p is checked n / q n The optimization strategy in the embodiment of the present disclosure may include the following situations: 1) d n =opfs, indicating the fusion of two computing operations p n-1 and p n ; 2)d n =tsfs, indicating the fusion computing operation p n-1 and p n Generate the communication operation q corresponding to the tensor n-1 and q n ; 3)d n =opfs_tsfs, indicating simultaneous fusion of computing operations p n-1 and p n And the corresponding communication operation q n-1 and q n ; 4)d n = null, no fusion. It should be understood that d0 = null. When tensor partitioning is enabled, an optimal number of partitions k can also be determined for tensors on the critical path. n Unlike tensor fusion and operator fusion, tensor partitioning does not delay the start time of tensor communication, but only changes the duration of tensor communication, so that each d n Possible values ​​to find the optimal k n , and then select the best d n Refer to Table 1, let T n To execute from the global dependency graph to p n / q n The time required for the execution to end, that is The goal of optimization is to find the best decision D = [d0, d1, ..., d |C|-1 ] and K=[k0,k1,...,k |C|-1 ] to minimize the execution time of each node on the critical path, that is:

[0086] Table 1

[0087]

[0088] The specific method for determining the above-mentioned optimization strategy is described below.

[0089] In a possible manner, determining an optimization strategy for a global dependency graph may be: for each computing node in a critical path, when the operation execution time of a previous communication group node of a communication group node corresponding to the computing node is less than or equal to a first fusion time difference, determining an operator fusion strategy as an optimization strategy for the global dependency graph, the operator fusion strategy being used to fuse the computing operations of the computing node and the previous computing node, the first fusion time difference being the time difference between the total computing time of the computing node and the previous computing node and the fusion computing time of the computing node and the previous computing node.

[0090] The computing node is used to perform computing operations, the communication group node is used to perform communication operations, and the computing time of the computing node is the computing operation p performed by the computing node. n The execution time of The operation execution time of the previous communication group node of the communication group node is the communication operation q n-1 The execution time of The fusion computing time is the execution time of the fusion computing operation after the computing operations of the computing node and the previous computing node of the computing node are fused. It can be obtained by collecting the trajectory of the fusion operation offline, and the embodiments of the present disclosure are not limited to this.

[0091] In the embodiment of the present disclosure, if Then the T realized by operator fusion is n Less than no fusion, that is, T n (d n =opfs) <T n (d n =null), fusion p n-1 and p n Better than no fusion. On the contrary, if Then the T realized by operator fusion is n Greater than or equal to no fusion, that is, T n (d n =opfs)≥T n (d n =null), fusion p n-1 and p n Will result in worse performance. Among them, opfs_time(p n-1 ,p n ) is the fusion p n-1 and p n The execution time of the post-computation operation (i.e., the fusion calculation time).

[0092] It should be understood that the inventors have also found that, when considering operator fusion and tensor fusion simultaneously, if fusing two computational operations is better than not fusing them, then the fusion of their corresponding communication operations (if any) will not sacrifice the overlap between computational communications.

[0093] Therefore, in a possible manner, after determining the operator fusion strategy as the optimization strategy for the global dependency graph, the tensor fusion strategy can also be determined as the optimization strategy for the global dependency graph, and the tensor fusion strategy is used to fuse the communication operations of the communication group node and the communication group node corresponding to the previous computing node. Accordingly, the optimization strategy includes the operator fusion strategy and the tensor fusion strategy.

[0094] In other possible ways, determining the optimization strategy for the global dependency graph can also be: for each communication group node in the critical path, determining the first optimal partition number of the tensor corresponding to the fusion communication operation in the case of fusion of the communication operation of the communication group node and the previous communication group node, and determining the second optimal partition number of the tensor corresponding to the communication group node. When the operation end time of the previous communication group node of the communication group node is greater than the second fusion time difference, the tensor fusion strategy is determined as the optimization strategy for the global dependency graph, the tensor fusion strategy is used to fuse the communication operation of the communication group node and the previous communication group node, the second fusion time difference is the time difference between the calculation end time of the calculation node corresponding to the communication group node plus the first tensor synchronization time and the second tensor synchronization time, the first tensor synchronization time is the tensor synchronization time after the tensor corresponding to the fusion communication operation is partitioned according to the first optimal partition number, and the second tensor synchronization time is the tensor synchronization time after the tensor corresponding to the communication group node is partitioned according to the second optimal partition number.

[0095] For example, the optimal number of partitions of the corresponding tensor can be obtained through grid search. The grid search method can refer to related technologies and will not be described in detail here.

[0096] For example, the communication group node includes multiple communication nodes for performing multiple communication operations. The previous communication group node of the communication group node is a communication group node located before the communication group node on the critical path. The fused communication operation is an operation that fuses the first communication operation corresponding to the communication group node and the second communication operation corresponding to the previous communication group node of the communication group node. The tensor corresponding to the fused communication operation is a tensor obtained by fusing the tensor corresponding to the first communication operation and the tensor corresponding to the second communication operation. The operation end time of the previous communication group node of the communication group node is the communication operation q n-1 The end time of The calculation end time of the computing node corresponding to the communication group node is the calculation operation p performed by the computing node. n The end time of

[0097] Therefore, in the embodiment of the present disclosure, if Then the communication operation group q corresponding to the fused tensor n-1 and q nThe obtained T n Less than no fusion, that is Fusion n-1 and q n Better than no fusion. On the contrary, if Then the communication operation group q corresponding to the fused tensor n-1 and q n The obtained T n Greater than or equal to no fusion, that is Fusionq n-1 and q n The performance is worse than that of non-fusion. For the Converged Communications Operation Group n The corresponding tensor and the previous communication operation group q n-1 In the case of the corresponding tensor, the optimal number of partitions of the fused tensor, that is, the first optimal number of partitions, The tensor synchronization time after the fused tensor is partitioned according to the first optimal partition number, that is, the first tensor synchronization time, For communication operation group q n The optimal number of partitions of the corresponding tensor, that is, the second optimal number of partitions, To group the communication operations n The tensor synchronization time after the corresponding tensor is partitioned according to the second optimal number of partitions, that is, the second tensor synchronization time.

[0098] Similarly, the inventors have also found that when considering operator fusion and tensor fusion at the same time, if fusing two communication operations is better than not fusing them, the fusion of their corresponding computing operations will not sacrifice the overlap between computing communications.

[0099] Therefore, in a possible manner, after determining the tensor fusion strategy as the optimization strategy for the global dependency graph, the operator fusion strategy can also be determined as the optimization strategy for the global dependency graph, and the operator fusion strategy is used to fuse the computing operations of the computing node corresponding to the communication group node and the computing node corresponding to the previous communication group node. Accordingly, the optimization strategy includes the operator fusion strategy and the tensor fusion strategy.

[0100] In other possible ways, after determining the tensor fusion strategy as the optimization strategy for the global dependency graph, the operator fusion strategy and the tensor partition strategy can be determined as the optimization strategy for the global dependency graph, the operator fusion strategy is used to fuse the computing operations of the computing nodes corresponding to the communication group nodes and the computing nodes corresponding to the previous communication group nodes, and the tensor partition strategy is used to partition and synchronize the corresponding tensors according to the first optimal partition number and the second optimal partition number. Accordingly, the optimization strategy includes the operator fusion strategy, the tensor fusion strategy and the tensor partition strategy.

[0101] Therefore, the global dependency graph can be optimized by combining the optimization strategies of operator fusion, tensor fusion and tensor partitioning, further improving the optimization effect of the global dependency graph.

[0102] It should be understood that exhaustively exploring a large-scale optimization strategy space is very time-consuming. Therefore, the disclosed embodiments also provide a method for accelerating the automatic search for optimization strategies.

[0103] In a possible manner, a coarsened view may be first determined by: determining a target computing node that does not correspond to a communication node in a global dependency graph, and fusing the computing operations corresponding to the target computing node and the next computing node of the target computing node, and / or determining a target communication node associated with the same computing node in a global dependency graph, and fusing the communication operations corresponding to the target communication node. Accordingly, according to the global dependency graph and the time offset, offline simulation of the distributed training process of the deep neural network may be: according to the coarsened view and the time offset, offline simulation of the distributed training process of the deep neural network.

[0104] It should be understood that if a computing node does not correspond to a communication node, then the computing node and the next computing node must meet the following conditions: Therefore, the computing operations of the computing node and the next computing node can be directly merged without searching for optimization strategies, so as to reduce the search time and improve the optimization efficiency. Similarly, for multiple communication group nodes associated with the same computing node, the multiple communication group nodes and the computing node meet the following conditions: Therefore, the communication operations can be directly integrated without searching for optimization strategies, so as to reduce the search time and improve the optimization efficiency.

[0105] That is to say, in the embodiment of the present disclosure, the computing operations that do not generate tensors (i.e., the computing operations of the target computing nodes that do not correspond to the communication nodes) but the adjacent next computing operations that generate tensors (i.e., the computing operations corresponding to the next computing node of the target computing node) can be put into a group, and the communication operations connected to the same computing operation can be put into a group. Then, the computing operations or communication operations in the same group can be directly fused to obtain a coarsened view. Finally, based on the coarsened view and the time offset, the distributed training process of the deep neural network can be simulated offline, and the optimization strategy can be searched based on the coarsened view to reduce the search time and improve the optimization efficiency.

[0106] In other possible ways, the first tensor synchronization time mentioned above can be determined as follows: determine a first communication operation related to a tensor corresponding to a synchronous fusion communication operation, generate a first communication subgraph based on the first communication operation, the first optimal number of partitions, and the tensor corresponding to the fusion communication operation, and determine the first tensor synchronization time based on the first communication subgraph. The second tensor synchronization time mentioned above can be determined as follows: determine a second communication operation related to a tensor corresponding to a synchronous communication group node, generate a second communication subgraph based on the second communication operation, the second optimal number of partitions, and the tensor corresponding to the communication group node, and determine the second tensor synchronization time based on the second communication subgraph.

[0107] It should be understood that in order to avoid frequent simulations of the entire global dependency graph during policy search, especially to estimate t sync (s, k), the embodiment of the present disclosure provides a partial simulation method. For example, given the current global dependency graph, identify all communication operations S related to the current tensor p , and generate a communication subgraph, which contains S p The operations in and the edges between these operations are then simulated to obtain the tensor synchronization time.

[0108] In other possible ways, executing the optimization strategy on the global dependency graph to obtain the optimized dependency graph may be: synchronously executing the optimization strategy on the local dependency graph corresponding to each processor in the global dependency graph to obtain the optimized dependency graph corresponding to each processor. Accordingly, according to the optimized dependency graph and the time offset, offline simulation of the distributed training process of the deep neural network may be: according to the optimized dependency graph corresponding to each processor and the time offset, offline simulation of the distributed training process of the deep neural network.

[0109] It should be understood that for distributed training in data parallel mode, multiple processors train the same deep neural network. Therefore, in the training analysis stage, each analysis will be performed on one processor. After the analysis of this processor is completed, a similar analysis will be performed on the next processor. Since the training is for the same deep neural network, the local dependency graphs corresponding to different processors in the global dependency graph are the same. If the optimization strategy is searched again for each analysis, it will undoubtedly increase unnecessary search time. Therefore, the embodiment of the present disclosure proposes a synchronous execution method to accelerate the search for optimization strategies, that is, after determining the optimization strategy for the global dependency graph, the local dependency graph corresponding to each processor in the global dependency graph is synchronously executed with the optimization strategy for the computing node, and the optimization dependency graph corresponding to each processor is obtained. Finally, according to the optimization dependency graph corresponding to each processor and the time offset, the offline simulation training process can be performed to complete the offline simulation and optimization of distributed training.

[0110] The training analysis method provided by the present disclosure is described below through another exemplary embodiment.

[0111] Reference Figure 4 After the online distributed training job is performed through the corresponding machine learning framework and communication framework, the trace of each operation in the job is collected through the analyzer (mainly the execution time of the operation), and the dependencies between the operations are analyzed to obtain the global dependency graph corresponding to the job, and the global dependency graph is time-aligned to obtain a global timeline view. Among them, the machine learning framework includes TensorFlow, MxNet, etc., and the communication framework includes BytePS, Horowod, etc., which are not limited in the embodiments of this disclosure. Then, the results collected by the analyzer are used to simulate the operation of the job using a simulator to obtain a performance evaluation of the job, and finally, the job is optimized based on the performance evaluation results.

[0112] like Figure 4 As shown, the optimizer may include a scheduler and a policy registration module, and the policy registration module is used to receive various registered optimization policies. Figure 4 The recalculation strategy and gradient accumulation strategy shown can refer to the relevant technology and will not be described here. The scheduler includes an evaluation module and a candidate module. The evaluation module is used to determine the iteration time and memory usage of the global dependency graph, and the candidate module is used to determine the operation corresponding to the critical path in the global dependency graph. Figure 4 , the candidate module sends the operation information corresponding to the critical path to the policy registration module through step ①, and the policy registration module can return the registered policy optimization scheme to the scheduler in response to the received operation information through step ②. The scheduler determines the target policy optimization scheme from the registered policy optimization schemes through the evaluation result of the evaluation module, and calls the target policy optimization scheme from the policy registration module through step ③, and executes the target optimization scheme on the global dependency graph to obtain the optimized dependency graph. Then, through step ④, the simulator is triggered to re-simulate offline based on the optimized dependency graph, and then the result of the offline simulation is returned through step ⑤ to re-determine the optimization strategy. By analogy, the global dependency graph can be updated and the critical path can be updated during each analysis process, and then the search can be repeated on the new critical path until the iteration time of the global dependency graph reaches the preset time threshold or the preset analysis stop condition is met.

[0113] It should be understood that the optimizer analyzes bottlenecks in the global dependency graph and optimizes them in an iterative manner. First, the optimizer simulates the operation of the initial global dependency graph through a simulator to evaluate the iteration time and memory usage of the global dependency graph. If the memory usage exceeds the memory threshold, the memory optimization technology is called to reduce the memory usage. The memory optimization technology can refer to the relevant technology and will not be repeated here.

[0114] Then, the optimizer continues to optimize for throughput to minimize the time span of the critical path in the global execution graph. Given a critical path C = [p0, p1, ..., p i ,q i ,q i+1 ,…,q |C|-1 ], the optimizer first checks the computation operations p in order n (n=1,...,i), because the performance of this critical path is limited by the calculation, the optimizer first evaluates whether to apply the operator fusion strategy to the calculation operation p according to the judgment conditions mentioned above. n-1 and p n If yes, then call the operator fusion strategy to fuse p n-1 and p n , and call the tensor fusion strategy to fuse the communication operation corresponding to the calculation operation q n-1 and q n Then, the optimal number of partitions is calculated and applied to the tensor q n (If tensor fusion is performed, q n represents the fused tensor).

[0115] Next, the optimizer checks the communication operations q on the critical path in turn. n (n=i,...,|C|-1). In this segment, the performance bottleneck is communication. The optimizer first calculates the optimal number of partitions k for the fused tensor and the unfused tensor respectively. * , that is, determine the first optimal number of partitions and the second optimal number of partitions. And evaluate whether to apply the tensor fusion strategy to the communication operation q according to the above-mentioned judgment conditions n-1 and q n If yes, then the tensor fusion strategy is called to fuse the communication operation q n-1 and q n , and at the same time call the operator fusion strategy to fuse the corresponding calculation operation p n-1 and p n Finally, the optimal number of partitions k is applied to the tensor corresponding to the fusion communication operation. * After applying the optimization, the global dependency graph is updated. The optimizer updates the critical path using the simulator and then repeats the search on the new critical path until the iteration time of the global dependency graph reaches a preset time threshold or the preset analysis stop condition is met.

[0116] Through the above method, a global dependency graph can be built for distributed machine learning training tasks, specific operations in the communication process can be analyzed and their trajectory information can be collected. At the same time, based on the dependency relationship between communication operations and the distribution of processors, the timestamps of the trajectories on different processors can be aligned offline. In addition, a search framework can be provided for the optimization of distributed machine learning training tasks, and custom optimization techniques can be added in a shared way to automatically search for the optimal optimization solution. In addition, for operator fusion, tensor fusion, and tensor partitioning, the optimal operator fusion, tensor fusion, and partitioning strategies for distributed machine learning training tasks can be searched based on a heuristic search algorithm that includes corresponding judgment conditions. In addition, the strategy search time can be shortened and the strategy optimization time can be improved through technologies such as coarsened views, partial simulations, and symmetry (i.e., parallel execution).

[0117] After testing, in the distributed training scenario of TensorFlow and MXNet training frameworks, using PS and AllReduce as communication libraries, and using RDMA (Remote Direct Memory Access) and TCP (Transmission Control Protocol) as transmission protocols, the training analysis method provided by the present disclosure can accurately simulate the distributed training of deep neural networks compared to the related art methods, because it analyzes the tensor communication process and aligns the timestamps of different nodes, with an average error of less than 5% (10 times better than Daydream). In addition, the optimizer in the training analysis method provided by the present disclosure can effectively explore good optimization strategies, which can increase the training throughput by 3.48 times compared to the related art methods, especially for large-scale distributed training tasks. In addition, the training analysis method provided by the present disclosure can reduce the strategy search time by 2 orders of magnitude compared to the related art methods, that is, it can shorten the strategy search time and improve the strategy optimization time.

[0118] Based on the same concept, the present disclosure also provides a training analysis device, which can be part or all of an electronic device through software, hardware, or a combination of both. Figure 5 , the apparatus is used to analyze the distributed training process of a deep neural network, the deep neural network runs on multiple devices, and the training analysis apparatus 500 includes:

[0119] A first determining module 501 is used to determine a global dependency graph of the deep neural network, wherein the global dependency graph is used to characterize the dependency relationship between computing nodes in the deep neural network running on each processor of the device and the communication relationship between communication nodes between the multiple processors for synchronizing tensors generated by the computing nodes, and each two processors in the global dependency graph include multiple communication nodes;

[0120] A second determination module 502 is used to determine the time offsets of the plurality of processors relative to a reference processor with the goal of minimizing a time alignment function and with the constraint that the operation dependencies between the nodes in the global dependency graph remain unchanged before and after the time alignment, wherein the time alignment function is used to characterize the time difference of the processors receiving tensors from the same processor and the time offset difference between processors in the same device, and the reference processor is any processor included in the plurality of devices;

[0121] The training simulation module 503 is used to simulate the distributed training process of the deep neural network offline according to the global dependency graph and the time offset, and determine the distributed training analysis result of the deep neural network according to the result of the offline simulation.

[0122] Optionally, the first determining module 501 is used to:

[0123] Determine a local dependency graph of a deep neural network running on a processor of each of the devices, wherein the local dependency graph is used to characterize dependency relationships between computing nodes in the deep neural network, and the local dependency graph includes a first virtual operation identifier for indicating a tensor synchronization start and a tensor synchronization end;

[0124] Determine a communication topology of the plurality of processors, the communication topology being used to characterize a communication relationship between communication nodes of the plurality of processors for synchronizing tensors generated by the computing nodes, and the communication topology comprising a second virtual operation identifier for indicating a tensor synchronization start and a tensor synchronization end;

[0125] The local dependency graph and the communication topology are associated with each other through the first virtual operation identifier and the second virtual operation identifier having the same identification information, so as to obtain a global dependency graph.

[0126] Optionally, in the time alignment function, the time of receiving the tensor is determined by the following modules:

[0127] a third determination module, determining an end time of the first processor performing a tensor receiving operation, and determining a latest time between a start time of the first processor performing the tensor receiving operation and a start time of the second processor performing a tensor sending operation, wherein the end time, the start time of the first processor performing the tensor receiving operation, and the start time of the second processor performing the tensor sending operation are all represented by a time offset of the corresponding processor relative to the reference processor, and the first processor and the second processor are different;

[0128] The fourth determination module is used to determine the time difference between the end time when the first processor performs the tensor receiving operation and the latest time as the time when the first processor receives the tensor from the second processor.

[0129] Optionally, the device 500 further includes:

[0130] a fifth determination module, configured to generate a global execution graph according to the global dependency graph, and determine a critical path according to the global execution graph, wherein the global execution graph is used to indicate an execution order of operations between nodes having dependency relationships in the global dependency graph;

[0131] An optimization module, configured to determine an optimization strategy for the global dependency graph with the goal of minimizing the operation execution time of the nodes on the critical path, and execute the optimization strategy on the global dependency graph to obtain an optimized dependency graph, wherein the nodes on the critical path include communication group nodes and computing nodes in the deep neural network, and the communication group nodes include a plurality of the communication nodes;

[0132] The training simulation module 503 is used for:

[0133] The distributed training process of the deep neural network is simulated offline according to the optimized dependency graph and the time offset.

[0134] Optionally, the optimization module is used to:

[0135] For each computing node in the critical path, when the operation execution time of the previous communication group node of the communication group node corresponding to the computing node is less than or equal to the first fusion time difference, the operator fusion strategy is determined as the optimization strategy for the global dependency graph, and the operator fusion strategy is used to fuse the computing operations of the computing node and the previous computing node. The first fusion time difference is the time difference between the total computing time of the computing node and the previous computing node and the fusion computing time of the computing node and the previous computing node.

[0136] Optionally, the device 500 further includes:

[0137] A sixth determination module, configured to determine a tensor fusion strategy as the optimization strategy for the global dependency graph after determining the operator fusion strategy as the optimization strategy for the global dependency graph, wherein the tensor fusion strategy is used to fuse the communication operations of the communication group node and the communication group node corresponding to the previous computing node;

[0138] Correspondingly, the optimization strategy includes the operator fusion strategy and the tensor fusion strategy.

[0139] Optionally, the optimization module is used to:

[0140] For each communication group node in the critical path, determining a first optimal number of partitions of a tensor corresponding to a fused communication operation when the communication operations of the communication group node and the previous communication group node are fused, and determining a second optimal number of partitions of a tensor corresponding to the communication group node;

[0141] When the operation end time of the previous communication group node of the communication group node is greater than the second fusion time difference, the tensor fusion strategy is determined as the optimization strategy for the global dependency graph, and the tensor fusion strategy is used to fuse the communication operations of the communication group node and the previous communication group node. The second fusion time difference is the time difference between the calculation end time of the computing node corresponding to the communication group node plus the first tensor synchronization time and the second tensor synchronization time. The first tensor synchronization time is the tensor synchronization time after the tensor corresponding to the fusion communication operation is partitioned according to the first optimal partition number, and the second tensor synchronization time is the tensor synchronization time after the tensor corresponding to the communication group node is partitioned according to the second optimal partition number.

[0142] Optionally, the device 500 further includes:

[0143] A seventh determination module, configured to determine an operator fusion strategy as the optimization strategy for the global dependency graph after determining the tensor fusion strategy as the optimization strategy for the global dependency graph, wherein the operator fusion strategy is used to fuse the computing operations of the computing node corresponding to the communication group node and the computing node corresponding to the previous communication group node;

[0144] Correspondingly, the optimization strategy includes the operator fusion strategy and the tensor fusion strategy.

[0145] Optionally, the device 500 further includes:

[0146] an eighth determination module, configured to determine an operator fusion strategy and a tensor partitioning strategy as the optimization strategy for the global dependency graph after determining the tensor fusion strategy as the optimization strategy for the global dependency graph, wherein the operator fusion strategy is used to fuse the computing operations of the computing node corresponding to the communication group node and the computing node corresponding to the previous communication group node, and the tensor partitioning strategy is used to partition and synchronize the corresponding tensors according to the first optimal number of partitions and the second optimal number of partitions;

[0147] Correspondingly, the optimization strategy includes the operator fusion strategy, the tensor fusion strategy and the tensor partitioning strategy.

[0148] Optionally, the first tensor synchronization time is determined by a first subgraph module, and the first subgraph module is used to: determine a first communication operation related to synchronizing a tensor corresponding to the fusion communication operation, generate a first communication subgraph based on the first communication operation, the first optimal number of partitions and the tensor corresponding to the fusion communication operation, and determine the first tensor synchronization time based on the first communication subgraph;

[0149] The second tensor synchronization time is determined by a second subgraph module, which is used to: determine a second communication operation related to synchronizing the tensor corresponding to the communication group node, generate a second communication subgraph, the second optimal number of partitions and the tensor corresponding to the communication group node based on the second communication operation, and determine the second tensor synchronization time based on the second communication subgraph.

[0150] Optionally, the optimization module is used to:

[0151] Synchronously execute the optimization strategy on the local dependency graph corresponding to each processor in the global dependency graph to obtain an optimized dependency graph corresponding to each processor;

[0152] The offline simulation of the distributed training process of the deep neural network according to the optimized dependency graph and the time offset includes:

[0153] According to the optimized dependency graph and the time offset corresponding to each processor, the distributed training process of the deep neural network is simulated offline.

[0154] Optionally, the device 500 further includes:

[0155] A coarsening module is used to determine a coarsening view in the following manner: determining a target computing node that does not correspond to a communication node in the global dependency graph, and fusing the computing operations corresponding to the target computing node and a computing node subsequent to the target computing node, and / or determining a target communication node associated with the same computing node in the global dependency graph, and fusing the communication operations corresponding to the target communication node;

[0156] The training simulation module 503 is used for:

[0157] The distributed training process of the deep neural network is simulated offline according to the coarsened view and the time offset.

[0158] Optionally, the training simulation module 503 is used to:

[0159] Corresponding each of the processors to an operation queue, corresponding each sending operation of each tensor in the global dependency graph to an operation queue, and corresponding each receiving operation of each tensor to an operation queue, wherein the operation queue is used to store operations to be executed, and the operation queue has a queue time represented by the time offset, and the queue time is the end time of the last operation performed by the processor or tensor communication process corresponding to the operation queue;

[0160] Determine the target operation queue with the earliest queue time, execute the head operation stored in the target operation queue, and update the queue time of the target operation queue based on the execution end time of the head operation, and return to the step of determining the target operation queue with the earliest queue time until all operations in the global dependency graph are executed, and determine the latest queue time among the queue times corresponding to the multiple operation queues as the iteration time of the global dependency graph;

[0161] According to the iteration time, a distributed training analysis result of the deep neural network is determined.

[0162] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0163] Based on the same concept, the present disclosure also provides a non-temporary computer-readable medium on which a computer program is stored. When the program is executed by a processing device, the steps of any of the above-mentioned training analysis methods are implemented.

[0164] Based on the same concept, the present disclosure also provides an electronic device, including:

[0165] a storage device having a computer program stored thereon;

[0166] A processing device is used to execute the computer program in the storage device to implement the steps of any of the above-mentioned training analysis methods.

[0167] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0168] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0169] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0170] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0171] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0172] In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be used for communication, and can be interconnected with any form or medium of digital data communication (e.g., communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0173] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0174] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines a global dependency graph of the deep neural network, wherein the global dependency graph is used to characterize the dependency relationship between computing nodes in the deep neural network running on the processor of each device and the communication relationship between communication nodes used to synchronize tensors generated by the computing nodes between multiple processors, and the global dependency graph includes multiple communication nodes between every two processors; determines the time offsets of the multiple processors relative to a reference processor with the goal of minimizing a time alignment function and the constraint that the operation dependencies between the nodes in the global dependency graph remain unchanged before and after time alignment, wherein the time alignment function is used to characterize the time difference between the processors receiving tensors from the same processor and the time offset difference between processors in the same device, wherein the reference processor is any processor included in the multiple devices; offline simulates the distributed training process of the deep neural network based on the global dependency graph and the time offset, and determines the distributed training analysis results of the deep neural network based on the results of the offline simulation.

[0175] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0176] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a module does not, in some cases, limit the module itself.

[0178] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0179] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0181] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0182] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A training analysis method, characterized in that: A distributed training process for analyzing a deep neural network, wherein the deep neural network is a deep neural network for implementing text recognition, speech recognition or image recognition functions, and the deep neural network runs on multiple devices, the method comprising: Determine a local dependency graph of the deep neural network running on a processor of each of the devices, wherein the local dependency graph is used to characterize the dependency relationship between computing nodes in the deep neural network, and the local dependency graph includes a first virtual operation identifier for indicating the start and end of tensor synchronization, wherein one computing node corresponds to a layer of network structure in the deep neural network; Determine a communication topology of the plurality of processors, the communication topology being used to characterize a communication relationship between communication nodes of the plurality of processors for synchronizing tensors generated by the computing nodes, and the communication topology comprising a second virtual operation identifier for indicating a tensor synchronization start and a tensor synchronization end; By using the first virtual operation identifier and the second virtual operation identifier having the same identification information, the local dependency graph and the communication topology are associated to obtain a global dependency graph, wherein the global dependency graph is used to characterize the dependency relationship between the computing nodes in the deep neural network running on the processor of each of the devices and the communication relationship between the communication nodes between the multiple processors for synchronizing the tensors generated by the computing nodes, and each two processors in the global dependency graph include multiple communication nodes; With the goal of minimizing a time alignment function and with the constraint that the operation dependencies between the nodes in the global dependency graph remain unchanged before and after the time alignment, determine the time offsets of the multiple processors relative to a reference processor, wherein the time alignment function is used to characterize the time difference of the processors receiving tensors from the same processor and the time offset difference between processors in the same device, and the reference processor is any processor included in the multiple devices; According to the global dependency graph and the time offset, the distributed training process of the deep neural network is simulated offline, and according to the result of the offline simulation, the distributed training analysis result of the deep neural network is determined.

2. The method according to claim 1, characterized in that In the time alignment function, the time of receiving the tensor is determined as follows: Determine an end time of a tensor receiving operation performed by a first processor, and determine a latest time between a start time of the first processor performing the tensor receiving operation and a start time of the second processor performing the tensor sending operation, wherein the end time, the start time of the first processor performing the tensor receiving operation, and the start time of the second processor performing the tensor sending operation are all represented by time offsets of corresponding processors relative to the reference processor, and the first processor and the second processor are different; The time difference between the end time when the first processor performs the tensor receiving operation and the latest time is determined as the time when the first processor receives the tensor from the second processor.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Generate a global execution graph according to the global dependency graph, and determine a critical path according to the global execution graph, wherein the global execution graph is used to indicate an execution order of operations between nodes having dependency relationships in the global dependency graph; With the goal of minimizing the operation execution time of the nodes on the critical path, determining an optimization strategy for the global dependency graph, and executing the optimization strategy on the global dependency graph to obtain an optimized dependency graph, wherein the nodes on the critical path include communication group nodes and computing nodes in the deep neural network, and the communication group nodes include a plurality of the communication nodes; The offline simulation of the distributed training process of the deep neural network according to the global dependency graph and the time offset includes: The distributed training process of the deep neural network is simulated offline according to the optimized dependency graph and the time offset.

4. The method according to claim 3, characterized in that: The determining of an optimization strategy for the global dependency graph includes: For each computing node in the critical path, when the operation execution time of the previous communication group node of the communication group node corresponding to the computing node is less than or equal to the first fusion time difference, the operator fusion strategy is determined as the optimization strategy for the global dependency graph, and the operator fusion strategy is used to fuse the computing operations of the computing node and the previous computing node. The first fusion time difference is the time difference between the total computing time of the computing node and the previous computing node and the fusion computing time of the computing node and the previous computing node.

5. The method according to claim 4, characterized in that The method further comprises: After determining the operator fusion strategy as the optimization strategy for the global dependency graph, determining the tensor fusion strategy as the optimization strategy for the global dependency graph, the tensor fusion strategy is used to fuse the communication operations of the communication group node and the communication group node corresponding to the previous computing node; Correspondingly, the optimization strategy includes the operator fusion strategy and the tensor fusion strategy.

6. The method according to claim 3, characterized in that The determining of an optimization strategy for the global dependency graph includes: For each communication group node in the critical path, determining a first optimal number of partitions of a tensor corresponding to a fused communication operation when the communication operation of the communication group node and a previous communication group node are fused, and determining a second optimal number of partitions of a tensor corresponding to the communication group node; When the operation end time of the previous communication group node of the communication group node is greater than the second fusion time difference, the tensor fusion strategy is determined as the optimization strategy for the global dependency graph, and the tensor fusion strategy is used to fuse the communication operations of the communication group node and the previous communication group node. The second fusion time difference is the time difference between the calculation end time of the computing node corresponding to the communication group node plus the first tensor synchronization time and the second tensor synchronization time. The first tensor synchronization time is the tensor synchronization time after the tensor corresponding to the fusion communication operation is partitioned according to the first optimal partition number, and the second tensor synchronization time is the tensor synchronization time after the tensor corresponding to the communication group node is partitioned according to the second optimal partition number.

7. The method according to claim 6, characterized in that The method further comprises: After determining the tensor fusion strategy as the optimization strategy for the global dependency graph, determining an operator fusion strategy as the optimization strategy for the global dependency graph, the operator fusion strategy being used to fuse the computing operations of the computing node corresponding to the communication group node and the computing node corresponding to the previous communication group node; Correspondingly, the optimization strategy includes the operator fusion strategy and the tensor fusion strategy.

8. The method according to claim 6, characterized in that The method further comprises: After the tensor fusion strategy is determined as the optimization strategy for the global dependency graph, an operator fusion strategy and a tensor partitioning strategy are determined as the optimization strategy for the global dependency graph, the operator fusion strategy is used to fuse the computing operations of the computing nodes corresponding to the communication group node and the computing nodes corresponding to the previous communication group node, and the tensor partitioning strategy is used to partition and synchronize the corresponding tensors according to the first optimal number of partitions and the second optimal number of partitions; Correspondingly, the optimization strategy includes the operator fusion strategy, the tensor fusion strategy and the tensor partitioning strategy.

9. The method according to claim 6, characterized in that The first tensor synchronization time is determined by: determining a first communication operation related to synchronizing a tensor corresponding to the fused communication operation, generating a first communication subgraph based on the first communication operation, the first optimal number of partitions, and the tensor corresponding to the fused communication operation, and determining a first tensor synchronization time based on the first communication subgraph; The second tensor synchronization time is determined in the following manner: determining a second communication operation related to synchronizing the tensor corresponding to the communication group node, generating a second communication subgraph based on the second communication operation, the second optimal number of partitions and the tensor corresponding to the communication group node, and determining the second tensor synchronization time based on the second communication subgraph.

10. The method according to claim 3, characterized in that The step of executing the optimization strategy on the global dependency graph to obtain an optimized dependency graph includes: Synchronously execute the optimization strategy on the local dependency graph corresponding to each processor in the global dependency graph to obtain an optimized dependency graph corresponding to each processor; The offline simulation of the distributed training process of the deep neural network according to the optimized dependency graph and the time offset includes: According to the optimized dependency graph corresponding to each processor and the time offset, the distributed training process of the deep neural network is simulated offline.

11. The method according to claim 1 or 2, characterized in that: The method further comprises: The coarsened view is determined by: determining a target computing node that does not correspond to a communication node in the global dependency graph, and fusing computing operations corresponding to the target computing node and a computing node subsequent to the target computing node, and / or determining a target communication node associated with the same computing node in the global dependency graph, and fusing communication operations corresponding to the target communication node; The offline simulation of the distributed training process of the deep neural network according to the global dependency graph and the time offset includes: The distributed training process of the deep neural network is simulated offline according to the coarsened view and the time offset.

12. The method according to claim 1 or 2, characterized in that: The offline simulation of the distributed training process of the deep neural network according to the global dependency graph and the time offset includes: Corresponding each of the processors to an operation queue, corresponding each sending operation of each tensor in the global dependency graph to an operation queue, and corresponding each receiving operation of each tensor to an operation queue, wherein the operation queue is used to store operations to be executed, and the operation queue has a queue time represented by the time offset, and the queue time is the end time of the last operation performed by the processor or tensor communication process corresponding to the operation queue; Determine the target operation queue with the earliest queue time, execute the head operation stored in the target operation queue, and update the queue time of the target operation queue based on the execution end time of the head operation, and return to the step of determining the target operation queue with the earliest queue time until all operations in the global dependency graph are executed, and determine the latest queue time among the queue times corresponding to the multiple operation queues as the iteration time of the global dependency graph; Determining the distributed training analysis result of the deep neural network according to the result of the offline simulation includes: According to the iteration time, a distributed training analysis result of the deep neural network is determined.

13. A training analysis device, characterized in that: The apparatus is used to analyze the distributed training process of a deep neural network, wherein the deep neural network is a deep neural network for realizing text recognition, speech recognition or image recognition functions, and the deep neural network runs on multiple devices, and the apparatus comprises: A first determination module is used to determine a local dependency graph of the deep neural network running on the processor of each of the devices, the local dependency graph is used to characterize the dependency relationship between the computing nodes in the deep neural network, and the local dependency graph includes a first virtual operation identifier for indicating the start and end of tensor synchronization, wherein one computing node corresponds to a layer of network structure in the deep neural network; determine a communication topology of multiple processors, the communication topology is used to characterize the communication relationship between the communication nodes for synchronizing the tensors generated by the computing nodes between the multiple processors, and the communication topology includes a second virtual operation identifier for indicating the start and end of tensor synchronization; through the first virtual operation identifier and the second virtual operation identifier having the same identification information, the local dependency graph and the communication topology are associated to obtain a global dependency graph, the global dependency graph is used to characterize the dependency relationship between the computing nodes in the deep neural network running on the processor of each of the devices and the communication relationship between the communication nodes for synchronizing the tensors generated by the computing nodes between the multiple processors, and each two processors in the global dependency graph include multiple communication nodes; A second determination module is used to determine the time offsets of the plurality of processors relative to a reference processor with the goal of minimizing a time alignment function and with the constraint that the operation dependencies between the nodes in the global dependency graph remain unchanged before and after the time alignment, wherein the time alignment function is used to characterize the time difference of the processors receiving tensors from the same processor and the time offset difference between processors in the same device, and the reference processor is any processor included in the plurality of devices; A training simulation module is used to simulate the distributed training process of the deep neural network offline according to the global dependency graph and the time offset, and determine the distributed training analysis result of the deep neural network according to the result of the offline simulation.

14. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 12 are implemented.

15. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 12.

Images