Distributed simulation system, simulation method, electronic equipment and storage medium

Abstract

The invention discloses a distributed simulation system, a simulation method, electronic equipment and a computer readable storage medium. The distributed simulation system comprises a database and a plurality of simulation nodes, wherein the database is configured to store performance data when multiple tasks are operated in a real distributed system; each simulation node comprises a scheduler, a distributor and a task simulator, and the scheduler is configured to obtain at least one simulation task; the distributor is configured to distribute virtual computing resources for each simulation task; and the task simulator is configured to run at least one simulation task according to a predetermined distributed strategy based on the virtual computing resources allocated for each simulation task and the performance data corresponding to each simulation task acquired from the database to obtain a running result.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a distributed simulation system, a simulation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, artificial intelligence technology is increasingly being used to solve practical problems. However, the problems faced by large-scale training in traditional single-machine environments are becoming increasingly prominent, especially in complex tasks, where it takes a lot of time to complete model training. In order to solve this problem, distributed systems can be used to implement large-scale training, and algorithm performance can be optimized in a targeted manner to improve model training efficiency. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a distributed simulation system, including: a database and multiple simulation nodes, the database is configured to store performance data when running multiple tasks in a real distributed system; each simulation node includes a scheduler, a distributor and a task simulator, the scheduler is configured to obtain at least one simulation task; the distributor is configured to allocate virtual computing resources to each simulation task; the task simulator is configured to run at least one simulation task according to a predetermined distributed strategy based on the virtual computing resources allocated to each simulation task and the performance data corresponding to each simulation task obtained from the database, to obtain an operation result.

[0004] For example, in a distributed simulation system provided by at least one example of the above-mentioned embodiments of the present disclosure, the real distributed system includes multiple computing nodes; the multiple tasks include multiple communication tasks and multiple computing tasks, the multiple communication tasks include tasks for transmitting multiple different data amounts within the computing nodes and tasks for transmitting multiple different data amounts between different computing nodes, the multiple computing tasks include computing tasks corresponding to multiple operators respectively; the performance data corresponding to each of the communication tasks includes the communication bandwidth and the time required for communication when running the communication task, and the performance data corresponding to each of the computing tasks includes the video memory consumed by the computing task and the time required for execution.

[0005] For example, in the distributed simulation system provided by at least one example of the above-mentioned embodiments of the present disclosure, the at least one simulation task includes at least one simulation computing task and at least one communication simulation task; the task simulator is configured to: for each of the simulation computing tasks, search the database for performance data corresponding to the operator and data volume of the simulation computing task as the performance data corresponding to the simulation computing task, and determine the running result of the simulation computing task based on the performance data, wherein the running result of the simulation computing task includes the video memory consumed by the simulation computing task and the time required for execution; for each of the communication simulation tasks, search the database for performance data corresponding to the communication range and data volume of the communication simulation task as the performance data corresponding to the communication simulation task, and determine the running result of the communication simulation task based on the performance data, wherein the running result of the communication simulation task includes the communication bandwidth of the communication simulation task and the time required for communication.

[0006] For example, in a distributed simulation system provided by at least one example of the above-mentioned embodiments of the present disclosure, the task simulator includes a computing task queue and a communication task queue; the task simulator is configured to place the at least one simulation task into one of the computing task queue and the communication task queue in a predetermined order, and execute the tasks in the computing task queue and the communication task queue in sequence.

[0007] For example, in the distributed simulation system provided by at least one example of the above embodiments of the present disclosure, the computing task queue includes a first computing task, and the communication task queue includes a first communication task and a second communication task; there is no dependency relationship between the first computing task and the first communication task, and the first computing task and the first communication task are executed in parallel; the second communication task depends on the first computing task, and the second communication task waits for the first computing task to be completed before starting to execute.

[0008] For example, in a distributed simulation system provided by at least one example of the above-mentioned embodiments of the present disclosure, the predetermined order is determined based on the distributed strategy, and the distributed strategy includes one of data parallelism, pipeline parallelism and tensor parallelism.

[0009] For example, in the distributed simulation system provided by at least one example of the above-mentioned embodiments of the present disclosure, it also includes: a result collector, configured to collect the running results of each of the simulation tasks in the multiple simulation nodes, and integrate the collected running results to obtain simulation results, wherein the simulation results include the total running time and video memory change data during the running process.

[0010] For example, in a distributed simulation system provided by at least one example of the above embodiments of the present disclosure, the allocator includes a tensor allocator and a video memory emulator, the tensor allocator is configured to allocate or release tensors; the video memory emulator is configured to allocate or release video memory.

[0011] At least one embodiment of the present disclosure provides a simulation method for a distributed simulation system, wherein the distributed simulation system includes multiple simulation nodes, and the method includes: each of the simulation nodes obtains at least one simulation task and allocates computing resources to each of the simulation tasks; each of the simulation nodes obtains performance data corresponding to its own simulation task from a database, and based on the computing resources allocated to each of the simulation tasks and the performance data corresponding to each of the simulation tasks, runs the at least one simulation task according to a predetermined distributed strategy to obtain an running result, wherein the database stores performance data when running multiple tasks in a real distributed system.

[0012] At least one embodiment of the present disclosure provides an electronic device, comprising a processor; a memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the processor to implement the simulation method provided by any embodiment of the present disclosure.

[0013] At least one embodiment of the present disclosure provides a computer-readable storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the simulation method provided by any embodiment of the present disclosure can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0015] Figure 1 A schematic diagram of distributed training based on a data parallel strategy is shown;

[0016] Figure 2 A schematic diagram of distributed training based on a pipeline model parallel strategy is shown;

[0017] Figure 3 A schematic diagram showing the parallel processing of multiple batches of data in pipelines is shown;

[0018] Figure 4 A schematic diagram showing forward propagation and reverse calculation of multiple batches of data;

[0019] Figure 5 A schematic diagram of distributed training based on a tensor model parallel strategy is shown;

[0020] Figure 6 A schematic diagram of an interleaved1f1b parallel strategy is shown;

[0021] Figure 7 A schematic diagram of forward propagation and reverse calculation of a zero bubble parallel strategy is shown;

[0022] Figure 8 A schematic diagram of a zero bubble parallel strategy is shown;

[0023] Fig. 9 A schematic diagram of an asynchronous loading strategy is shown;

[0024] Fig.10 A schematic diagram of a distributed simulation system provided by at least one embodiment of the present disclosure is shown;

[0025] Fig.11 A schematic diagram of a simulation node provided by at least one embodiment of the present disclosure is shown;

[0026] Fig.12 A schematic diagram showing a task execution timeline provided by at least one embodiment of the present disclosure;

[0027] Fig.13 A flowchart of a simulation method provided by at least one embodiment of the present disclosure is shown;

[0028] Fig.14 A schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure is shown;

[0029] Fig.15 A schematic block diagram showing another electronic device provided by at least one embodiment of the present disclosure; and

[0030] Fig.16 A schematic diagram of a computer-readable storage medium provided by at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0033] A distributed system can include multiple computing nodes, such as multiple GPUs (graphics processing units). When the training model is large, due to the GPU memory limitation, the model layers, operators, data, etc. in the model can be split and the multiple parts obtained by splitting can be placed on different GPU cards. In addition, technologies such as recalculation can be used to reduce the memory requirements of a single GPU card to perform distributed training on the model.

[0034] Figure 1 A schematic diagram of distributed training based on a data parallel strategy is shown.

[0035] like Figure 1 As shown, in some embodiments, distributed training can be performed in a data parallel manner. Data parallelism refers to splitting the input data and replicating the model on each GPU card, that is, each GPU card has a complete model, and different GPU cards receive and process different data.

[0036] Figure 2 A schematic diagram of distributed training based on a pipeline model parallel strategy is shown.

[0037] like Figure 2 As shown, in some embodiments, distributed training can be performed in a pipeline model parallel manner. Pipeline model parallelism refers to splitting the model into multiple layers, and running different layers of the model on different GPU cards. For example, the model is split into 3 GPU cards, and 3 training stages are implemented respectively.

[0038] Figure 3 A schematic diagram showing the pipeline parallelization of multiple batches of data.

[0039] like Figure 3 As shown in Figure 1, by splitting the global batch into smaller micro batches, the pipeline parallelization is performed on multiple micro batches of data. i,j represents the forward propagation of the jth micro-batch in the i-th stage. In the forward process, the calculation of the later stage depends on the calculation of the previous stage. For example, F 1,0 Need to wait until F 0,0 After the forward propagation of each microbatch is completed, the backword calculation is performed. i,j represents the reverse computation of the jth micro-batch at the i-th stage, B 0,0 Need to wait until B 1,0 The calculation starts only after the calculation is completed. During the training process, because the reverse calculation requires the activation value obtained by the forward calculation, the activation value obtained by the forward calculation cannot be released immediately and needs to wait until the reverse calculation is completed. Among them, some layers of the neural network include activation functions, and the calculation value of the activation function is the activation value. The activation value of each layer can be used as the input of the next layer, and the activation value of the last layer (output layer) can be used as the forward calculation result of the neural network. Figure 3 In the figure, Bubble represents empty bubbles, and the blank part is regarded as idling time, called bubbles. The larger the proportion of bubbles, the lower the training efficiency.

[0040] For example, in some examples, the forward propagation of multiple micro-batches is completed before the backward propagation process of multiple micro-batches is started. However, this requires caching the activation values ​​of the forward propagation of multiple micro-batches at the same time, which requires more video memory.

[0041] For example, in order to reduce video memory, a 1F1B strategy can be adopted. After performing forward propagation of one microbatch, the corresponding reverse calculation is performed immediately, rather than waiting until the forward propagation of all microbatches is completed before performing the reverse calculation. In this way, the activation values ​​calculated and cached in the forward propagation stage can be quickly processed in the backward propagation stage, thereby releasing the video memory they occupy, thereby reducing the use of video memory.

[0042] Figure 4 A schematic diagram showing forward propagation and reverse calculation based on the 1F1B strategy for multiple batches of data.

[0043] like Figure 4As shown, device0 to device 3 can represent 4 computing nodes (such as GPUs), and the numbers 0 to 4 in the box represent 5 micro-batches of data. Micro-batch 0 passes through device 0 to device 3 in turn to perform forward propagation. After device 3 completes the forward propagation (Forward) of micro-batch 0, it immediately performs reverse calculation (Backword) on batch 0 without waiting for other batches to complete forward propagation. In addition, before reverse calculation, for example, recomputation can be performed. For each micro-batch, during forward calculation, except for a small number of variables that must be stored, other intermediate variables can be deleted. During reverse calculation, the forward operator can be calculated (Recomputation) first to obtain the required intermediate results, and then the reverse operator can be run normally. Figure 4 In the figure, Bubble means bubble. Critical Path means critical path, which is used to indicate the execution path of micro-batch.

[0044] Figure 5 A schematic diagram of distributed training based on a tensor model parallel strategy is shown.

[0045] like Figure 5 As shown, in some implementations, distributed training can be performed using tensor model parallelism. Tensor model parallelism refers to splitting each operator in the model and placing part of the operator's weight on different GPU cards, which can reduce the video memory on each card.

[0046] For example, the execution of a parallel pipeline can be divided into three stages: warmup stage, 1F1B steady-state stage, and cooldown stage. In order to reduce the bubble in the warmup and cooldown stages, the interleaved1f1b parallel strategy can be adopted. The warmup stage, 1F1B steady-state stage, and cooldown stage are the three stages that are entered in sequence as the program proceeds. In the warmup stage, only forward propagation is executed. In the 1F1B steady-state stage, forward propagation and reverse calculation are interspersed. In the cooldown stage, the finishing work of reverse calculation is performed.

[0047] Figure 6 A schematic diagram showing an interleaved1f1b parallel strategy.

[0048] like Figure 6As shown in the figure, interleaved1f1b splits the model into multiple chunks, for example, it splits into 2 chunks (chunk0 and chunk1). For example, each layer of the model is divided into 2 chunks, that is, each device performs two calculation parts in sequence when executing the calculation of each layer of the model, instead of performing the calculation of the entire layer at once. For each micro-batch in the box (indicated by a number, such as the number 1), the calculation in each layer is divided into two parts (chunk0 and chunk1). The dark blue box in the forward propagation represents chunk0, and the light blue box represents chunk1. Similarly, the dark green box in the reverse calculation represents chunk0, and the light green box represents chunk1. For 1F1B, each GPU is responsible for several consecutive layers. For example, the model has 16 layers in total, GPU1 is responsible for layers 1 to 4, GPU2 is responsible for layers 5 to 8, GPU3 is responsible for layers 9 to 12, and GPU4 is responsible for layers 13 to 16. For interleaved1f1b, each GPU can be responsible for multiple small segments of continuous layers, for example, GPU1 is responsible for layers 1, 2, 9, and 10, GPU2 is responsible for layers 3, 4, 11, and 12, GPU3 is responsible for layers 5, 6, 13, and 14, and GPU4 is responsible for layers 7, 8, 15, and 16. For interleaved1f1b, because during the warm up phase, only chunk0 and chunk1 of the forward propagation of one micro-batch (such as micro-batch 1) need to be executed to enter the steady state phase, at this time, other micro-batches (such as micro-batches 2-4) may only need to execute chunk0 of the forward propagation, without waiting for other micro-batches (such as micro-batches 2-4) to complete the chunk1 of the forward propagation, so the warm up phase is shorter, and the cool down phase is similar.

[0049] Figure 7 A schematic diagram showing forward propagation and reverse computation of a zero bubble parallel strategy.

[0050] like Figure 7 As shown in the figure, in order to further reduce the cavitation in the warmup and cooldown phases, a zero bubble parallel strategy can be used. In the zero bubble parallel strategy, the backword process is split into bpa (gradient for input activation) and bpw (gradient for weight). Because the backpropagation of subsequent operators only depends on bpa, all bpw can be executed later. Figure 7 In the figure, F represents forward propagation.

[0051] Figure 8A schematic diagram of a zero bubble parallel strategy is shown.

[0052] like Figure 8 As shown, the reverse calculation process of each micro-batch is split into a bpa part and a bpw part. For each micro-batch, the bpa part of the reverse calculation is started immediately after the forward propagation is executed. The bpw part can be delayed and the bpw part can be interspersed in the bubbles, so that the forward and reverse directions overlap with each other, thereby eliminating the bubbles in the warmup and cooldown stages. For example, after executing the forward propagation F of micro-batch 1, the B(bpa) part of the reverse propagation of micro-batch 1 can be executed immediately, and the W(bpw) part of the reverse propagation of micro-batch 1 can be interspersed in other positions to eliminate bubbles. Figure 8 Part (a) shows the running status when the zero bubble parallel strategy is not adopted, and part (b) shows the running status after the zero bubble parallel strategy is adopted. It can be seen that after the zero bubble parallel strategy is adopted, the bubble part is reduced. Figure 8 In , Optimizer step means updating parameters according to the gradient.

[0053] Fig. 9 A schematic diagram of an asynchronous loading (asynchronous offload) strategy is shown.

[0054] like Fig. 9 As shown, since the activation value needs to be saved for reverse use, and high-bandwidth video memory is usually expensive and scarce, it often leads to the problem of video memory explosion. Therefore, the activation value can be loaded (offloaded) to the CPU and pre-fetched in advance when appropriate. This reduces the video memory by utilizing memory without causing performance degradation. Fig. 9The distributed training system shown includes at least one computing node node 0 to node n (n is an integer greater than or equal to 1), an analysis unit and a loading unit (offloader). Each computing node includes at least one CPU memory and at least one GPU card. The analysis unit can obtain the model structure of the target model to be trained and the parallel strategy of the target model in the distributed training system, and obtain the memory simulation result, the device memory simulation result and the runtime information according to the model structure and the parallel strategy. The loading unit (offloader) includes a strategy subunit (offload strategy) and an operation subunit (offload operator), the strategy subunit (offload strategy) is used to generate a loading (or moving) strategy for target data according to the memory simulation result, the device memory simulation result and the runtime information, and the operation subunit (offload operator) is used to send a transfer instruction and a pre-fetch instruction corresponding to the target data to the distributed training system based on the loading (or moving) strategy, wherein the transfer instruction is used to move the target data from at least one GPU card to at least one CPU memory in each computing node, and the pre-fetch instruction is used to move the target data from at least one CPU memory to at least one GPU card in each computing node. The analysis unit may include a runtime analyzer, a memory simulation unit, and a memory simulation unit. The runtime analyzer is configured to obtain runtime information, the memory simulation unit is configured to generate CPU memory simulation results, and the video memory simulation unit is configured to generate GPU card memory simulation results.

[0055] In addition, in some embodiments, a hybrid parallel approach can be used. Since the use of a single distributed algorithm often cannot simultaneously meet the graphics memory limitations and achieve high efficiency, in actual scenarios, pipeline model parallelism, tensor model parallelism, data parallelism, asynchronous offload, etc. can be used in combination.

[0056] When a distributed system performs model training based on any distributed strategy, a large amount of resources is required, usually a cluster of thousands or tens of thousands of cards. If a large amount of debugging is required on the cluster for each distributed strategy developed, this occupation of resources is unrealistic. Therefore, how to use fewer resources to implement model training for various distributed strategies is a technical problem that needs to be solved.

[0057] At least one embodiment of the present disclosure provides a distributed simulation system, a simulation method, an electronic device, and a computer-readable storage medium. The distributed simulation system includes: a database and multiple simulation nodes, the database is configured to store performance data when running multiple tasks in a real distributed system; each simulation node includes a scheduler, a distributor, and a task simulator, the scheduler is configured to obtain at least one simulation task; the distributor is configured to allocate virtual computing resources to each simulation task; the task simulator is configured to run at least one simulation task according to a predetermined distributed strategy based on the virtual computing resources allocated to each simulation task and the performance data corresponding to each simulation task obtained from the database, and obtain an operation result.

[0058] The distributed simulation system stores the performance data (such as the performance of operators in different shapes and the performance of various communication operators between different machines) obtained by running the model in a real distributed system in a database. Each simulation node extracts the performance data from the database, simulates the execution process of various real distributed strategies, and then obtains simulation results such as running time and video memory information. In this way, there is no need to execute a large number of real computing and communication processes in a real distributed system, so there is no need to occupy a large amount of resources, and fewer resources can be used to implement model training of various distributed strategies.

[0059] Fig.10 A schematic diagram of a distributed simulation system provided by at least one embodiment of the present disclosure is shown.

[0060] like Fig.10 As shown, the distributed simulation system includes a database 1010 and multiple simulation nodes 1020. The database 1010 is configured to store performance data when running various tasks in a real distributed system.

[0061] For example, you can run the model training process in a real distributed system and obtain various performance data during the running process.

[0062] For example, a real distributed system may include multiple computing nodes, such as multiple GPUs. Multiple tasks may include multiple communication tasks and multiple computing tasks. Multiple communication tasks include tasks for transmitting multiple different amounts of data within a computing node and tasks for transmitting multiple different amounts of data between different computing nodes. Multiple computing tasks include computing tasks corresponding to multiple operators. The performance data corresponding to each communication task includes the communication bandwidth and the time required for communication when running the communication task. The performance data corresponding to each computing task includes the video memory consumed by the computing task and the time required for execution.

[0063] For example, a distributed system can perform computing tasks, intra-node communication tasks, and inter-node communication tasks during model training. Computing tasks may include computing nodes performing forward propagation, reverse computing, and other tasks involving data computing. The attributes of each computing task may include operator type and data size. The video memory information and execution time of multiple computing tasks with different operator types and different data sizes can be obtained as the performance data of the computing task. Intra-node communication tasks refer to tasks that perform data transmission within a computing node. The communication bandwidth and transmission time when transmitting various data sizes within the node can be obtained as the performance data of intra-node communication tasks, for example, the communication bandwidth and transmission time when transmitting data of 1KB, 10KB, and 100KB sizes within the computing node. Inter-node communication tasks refer to tasks that perform data transmission between two computing nodes. The communication bandwidth and transmission time when transmitting various data sizes between nodes can be obtained as the performance data of inter-node communication tasks.

[0064] For example, the simulation node can be implemented as a thread, and the database can be implemented as a data table. Each simulation node 1020 can obtain the required performance data from the database 1010. Multiple simulation nodes can be used to simulate multiple computing nodes (such as GPUs). For example, for data parallel strategies, the input data is split into multiple parts, and each simulation node can simulate the training process of the entire model using one of the data. For the pipeline model parallel strategy, the model is split into multiple layers, and multiple simulation nodes can be used to execute simulation tasks of different layers in the model, that is, each simulation node can simulate the training process of one layer.

[0065] Fig.11 A schematic diagram of a simulation node provided by at least one embodiment of the present disclosure is shown.

[0066] like Fig.11 As shown, each simulation node 1020 includes a scheduler 1110, a distributor 1120 and a task simulator 1130. The scheduler 1110 is configured to obtain at least one simulation task, and the distributor 1120 is configured to allocate virtual computing resources to each simulation task. The task simulator 1130 is configured to run the at least one simulation task according to a predetermined distributed strategy based on the virtual computing resources allocated to each simulation task and the performance data corresponding to each simulation task obtained from the database to obtain the running result.

[0067] For example, the scheduler 1110 may receive distributed information, which may be pre-stored information or obtained from other devices, and the distributed information may include, for example, model information, data parallel parameters, tensor model parallel parameters, pipeline model parallel parameters, etc., and generate multiple simulation tasks for the current node in chronological order according to the distributed information. For example, for Figure 3 The 1f1b distributed strategy shown in FIG. 1 treats device 0 to device 3 as four simulation nodes. The pipeline sequence of the distributed strategy includes, for example, node device 0 generates and executes computing task F 0,0 , then, node device0 generates and executes the transmission of F to node device 1 0,0 The inter-node communication task of the calculation result, and then the node device 1 generates and executes the calculation task F 1,0 , and during the period when node device 0 generates and executes the inter-node communication task or when node device 1 generates and executes the computing task F 1,0 During this period, node device 0 can generate and execute computing task F 0,1 , and so on, in this way, the type and time of the task generated by each simulation node can be determined according to the distributed strategy, that is, the type of task generated by each simulation node at what time can be determined according to the distributed strategy. Among them, the type of task can include computing tasks and communication tasks. For computing tasks, the task type can be further classified according to the type of the corresponding operator.

[0068] For example, the scheduler 1110 may send the generated simulation tasks to the task simulator 1130 in sequence. After receiving each simulation task, the task simulator 1130 notifies the allocator 1120 to allocate virtual computing resources to each simulation task. The computing resources include, for example, virtual video memory.

[0069] For example, the allocator 1120 includes a tensor allocator 1121 and a graphics memory simulator 1122. The tensor allocator 1121 is configured to allocate or release tensors, and the graphics memory simulator 1122 is configured to allocate or release graphics memory. For example, after receiving a simulation task, the task simulator 1130 notifies the tensor allocator 1121 to allocate tensors for the simulation task for constructing the task. The tensors allocated by the tensor allocator 1121 to a simulation task can be used as input data of the simulation task. The task simulator 1130 can search for corresponding performance data (such as the required calculation time and communication bandwidth, etc.) from the database according to the size of the input data (tensor) of the simulation task and the corresponding operator type. When creating a new tensor for a simulation task, the tensor distributor 1121 needs to obtain the corresponding video memory from the video memory simulator 1122. Therefore, the tensor distributor 1121 can notify the video memory simulator 1122 to allocate video memory. The video memory allocation process is to simulate the change of video memory. The video memory simulator 1122 can allocate virtual video memory for the simulation task according to the performance data corresponding to each simulation task (such as video memory consumption). For example, the video memory size corresponding to the real task with the same task type and data volume as the current simulation task can be found from the database, and the virtual video memory of the same size is allocated to the current simulation task. Among them, the task type can include a computing task or a communication task. For computing tasks, the task type can be further classified according to the type of the corresponding operator. For example, when the task simulator 1130 completes a simulation task, the tensor of the simulation task can be released to the tensor distributor 1121, and the tensor distributor 1121 can release the corresponding video memory to the video memory simulator 1122.

[0070] For example, after allocating virtual computing resources to the simulation task, the task simulator 1130 can put the simulation task into a virtual stream to execute each simulation task in sequence, wherein the virtual stream can be a task queue. The execution of the simulation task here does not mean actually executing a calculation or communication operation, but rather finding the target performance data corresponding to the real task of the same type and data volume as the simulation task from the database, and determining the running result of the simulation task based on the target performance data.

[0071] For example, at least one simulation task generated by the scheduler may include at least one simulation computing task and at least one communication simulation task. The task simulator is configured to: for each simulation computing task, search the database for performance data corresponding to the operator and data volume of the simulation computing task as the performance data corresponding to the simulation computing task, and determine the running result of the simulation computing task based on the performance data, wherein the running result of the simulation computing task includes the video memory consumed by the simulation computing task and the time required for execution; for each communication simulation task, search the database for performance data corresponding to the communication range and data volume of the communication simulation task as the performance data corresponding to the communication simulation task, and determine the running result of the communication simulation task based on the performance data, wherein the running result of the communication simulation task includes the communication bandwidth of the communication simulation task and the time required for communication.

[0072] For example, for each simulated computing task, the task simulator can find the same or similar real computing task from the database according to the operator type and computing data volume of the simulated computing task, and use the video memory consumed and the execution time required for the real computing task as the running result of the simulated computing task. For each simulated communication task, the task simulator can find the same or similar real communication task from the database according to the communication range (within a node or between nodes) and data volume of the simulated communication task, and use the communication bandwidth and execution time required for the real communication task as the running result of the simulated communication task.

[0073] For example, a task simulator may include a computational task queue (e.g. Fig.11 The virtual computing flow shown) and the communication task queue (such as Fig.11 The task simulator is configured to place at least one simulation task into one of a computing task queue and a communication task queue in a predetermined order, and execute the tasks in the computing task queue and the communication task queue in sequence. For example, the predetermined order is determined based on a distributed strategy, and the distributed strategy includes one of data parallelism, pipeline parallelism, and tensor parallelism, wherein pipeline parallelism may include 1f1b parallelism, interleaved1f1b parallelism, zerobubble parallelism, etc.

[0074] For example, the computing task queue includes a first computing task (such as Fig.11 The computing task 1 shown in FIG. 1 ) includes a first communication task (such as Fig.11 The communication task 2 shown) and the second communication task (such as Fig.11There is no dependency relationship between the first computing task (such as computing task 1) and the first communication task (such as communication task 2), and the first computing task and the first communication task are executed in parallel. The second communication task (such as communication task 3) depends on the first computing task (such as computing task 1), and the second communication task waits for the first computing task to be executed before starting to execute.

[0075] For example, there are dependencies between some communication tasks and computation tasks in the same node, e.g. Figure 3 As shown, the calculation task F 0,0 After the execution is completed, the calculation task F can be executed. 0,0 The communication task of transmitting the execution result to the next node. For two tasks with dependencies, it is necessary to wait for the previous task to be completed before starting the next task. For two tasks without dependencies, they can be executed in parallel.

[0076] For example, there may be dependencies between computing tasks on different nodes. For example, after device0 completes computing task F 0,0 After that, device1 can start to execute computing task F 1,0 Therefore, an event can be inserted into the communication task queue to indicate that the relevant computing task of the previous node has been completed.

[0077] For example, in some embodiments, the distributed simulation system may further include a result collector, which is configured to collect the running results of each simulation task in multiple simulators, and integrate the collected running results to obtain simulation results, wherein the simulation results include the total running time and video memory change data during the running process.

[0078] For example, the result collector can collect the running results of each simulation task in each simulator, and integrate and analyze the running results of each simulation task to obtain the final desired result.

[0079] Fig.12 A schematic diagram of a task execution timeline provided by at least one embodiment of the present disclosure is shown.

[0080] like Fig.12As shown, taking the interleaved1f1b distributed strategy as an example, the overall running timeline can be drawn according to the execution time and execution duration of each simulation task, where each block can represent a computing task with a data volume of one micro batch and a number of layers involved of one chunk. For example, F1_1 represents the forward computing task corresponding to chunk1 and micro batch 0, and B1_1 represents the reverse computing task corresponding to chunk1 and micro batch 0. The length of each block can be related to the running time of the corresponding computing task, and the time occupied by the communication task can also be inserted between the computing tasks. Through the task execution timeline, simulation results related to the total running time and other time can be analyzed.

[0081] For example, a video memory change curve can be drawn based on the video memory information of each computing task. For example, for any point in time, the total video memory value occupied by all computing tasks running at the current time point can be counted, and the total video memory amount can be used as the video memory consumption value at the current time point. This can be counted every predetermined time period, and then the video memory change curve of the entire running process can be obtained.

[0082] According to the distributed simulation system of the embodiment of the present disclosure, the performance data (such as the performance of operators in different shapes, the performance of various communication operators between different machines) obtained by running the model in the real distributed system is stored in a database, and each simulation node extracts the performance data from the database to simulate the execution process of various real distributed strategies, thereby obtaining simulation results such as running time and video memory information. In this way, there is no need to occupy a large amount of resources, and fewer resources can be used to implement model training of various distributed strategies.

[0083] For example, each device / module in the simulation node can be implemented by a hardware (e.g., circuit) module, a software module, or any combination of the two. For example, these devices / modules can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities and corresponding computer instructions. It should be noted that Fig.10 The distributed simulation system shown and Fig.11 The components and structures of the simulation nodes shown are merely exemplary and non-limiting. The distributed simulation system and simulation nodes may also have other components and structures as required.

[0084] Some embodiments of the present disclosure provide a simulation method for a distributed simulation system, wherein the distributed simulation system includes multiple simulation nodes.

[0085] Fig.13A flowchart of a simulation method provided by at least one embodiment of the present disclosure is shown.

[0086] like Fig.13 As shown, the method may include steps S1310 to S1320.

[0087] Step S1310: Each simulation node obtains at least one simulation task and allocates computing resources to each simulation task.

[0088] Step S1320: Each simulation node obtains the performance data corresponding to its own simulation task from the database, and based on the computing resources allocated to each simulation task and the performance data corresponding to each simulation task, runs at least one simulation task according to a predetermined distributed strategy to obtain an operation result. The database stores the performance data when running multiple tasks in a real distributed system.

[0089] For example, a real distributed system includes multiple computing nodes; multiple tasks include multiple communication tasks and multiple computing tasks, multiple communication tasks include tasks for transmitting multiple different data amounts within the computing nodes and tasks for transmitting multiple different data amounts between different computing nodes, and multiple computing tasks include computing tasks corresponding to multiple operators respectively; the performance data corresponding to each communication task includes the communication bandwidth and the time required for communication when running the communication task, and the performance data corresponding to each computing task includes the video memory consumed by the computing task and the time required for execution.

[0090] For example, at least one simulation task includes at least one simulation computing task and at least one communication simulation task. Based on the computing resources allocated to each simulation task and the performance data corresponding to each simulation task, the at least one simulation task is run in accordance with a predetermined distributed strategy to obtain the running result, including: for each simulation computing task, searching the database for performance data corresponding to the operator and data volume of the simulation computing task as the performance data corresponding to the simulation computing task, and determining the running result of the simulation computing task based on the performance data, wherein the running result of the simulation computing task includes the video memory consumed by the simulation computing task and the time required for execution; for each communication simulation task, searching the database for performance data corresponding to the communication range and data volume of the communication simulation task as the performance data corresponding to the communication simulation task, and determining the running result of the communication simulation task based on the performance data, wherein the running result of the communication simulation task includes the communication bandwidth of the communication simulation task and the time required for communication.

[0091] For example, based on the computing resources allocated to each simulation task and the performance data corresponding to each simulation task, the above-mentioned at least one simulation task is run according to a predetermined distributed strategy to obtain an operation result, including: placing at least one simulation task into one of the computing task queue and the communication task queue in a predetermined order, and executing the tasks in the computing task queue and the communication task queue in sequence.

[0092] For example, the computing task queue includes a first computing task, and the communication task queue includes a first communication task and a second communication task; there is no dependency between the first computing task and the first communication task, and the first computing task and the first communication task are executed in parallel; the second communication task depends on the first computing task, and the second communication task waits for the first computing task to be completed before starting to execute.

[0093] For example, the predetermined order is determined based on a distributed strategy, and the distributed strategy includes one of data parallelism, pipeline parallelism, and tensor parallelism.

[0094] For example, the simulation method further includes: collecting the running results of each simulation task in multiple simulation nodes, and integrating the collected running results to obtain simulation results, wherein the simulation results include the total running time and video memory change data during the running process.

[0095] For example, allocating computing resources to each simulation task includes: allocating or releasing tensors; allocating or releasing video memory.

[0096] At least one embodiment of the present disclosure further provides an electronic device, the electronic device comprising a processor and a memory, wherein the memory stores one or more computer program modules. The one or more computer program modules are configured to be executed by the processor to implement the above-mentioned simulation method.

[0097] Fig.14 A schematic block diagram of an electronic device provided in some embodiments of the present disclosure. Fig.14 As shown, the electronic device 1400 includes a processor 1410 and a memory 1420. The memory 1420 stores non-temporary computer-readable instructions (e.g., one or more computer program modules). The processor 1410 is used to run non-temporary computer-readable instructions, and the non-temporary computer-readable instructions are executed by the processor 1410 when they are executed. One or more steps in the simulation method described above. The memory 1420 and the processor 1410 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown). For the specific implementation of each step of the simulation method and the related explanation content, please refer to the embodiment of the above-mentioned simulation method, and the repetitions will not be repeated here.

[0098] It should be noted that Fig.14The components of the electronic device 1400 shown are merely exemplary and non-limiting. The electronic device 1400 may also have other components according to actual application requirements.

[0099] For example, the processor 1410 and the memory 1420 may communicate with each other directly or indirectly.

[0100] For example, the processor 1410 and the memory 1420 may communicate via a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 1410 and the memory 1420 may also communicate with each other via a system bus, which is not limited in the present disclosure.

[0101] For example, the processor 1410 and the memory 1420 may be disposed on a server side (or a cloud side).

[0102] For example, the processor 1410 can control other components in the electronic device 1400 to perform desired functions. For example, the processor 1410 can be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) can be an X86 or ARM architecture, etc. The processor 1410 can be a general-purpose processor or a special-purpose processor, and can control other components in the electronic device 1400 to perform desired functions.

[0103] For example, the memory 1420 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 1410 may run one or more computer program modules to implement various functions of the electronic device 1400. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.

[0104] It should be noted that, in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 1400 can refer to the description of the simulation method above, which will not be repeated here.

[0105] Fig.15This is a schematic block diagram of another electronic device provided in some embodiments of the present disclosure. The electronic device 1500 is suitable for implementing the simulation method provided in the embodiments of the present disclosure. The electronic device 1500 may be a terminal device, etc. It should be noted that: Fig.15 The electronic device 1500 shown is merely an example and does not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.

[0106] like Fig.15 As shown, the electronic device 1500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1510, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1520 or a program loaded from a storage device 1580 to a random access memory (RAM) 1530. In the RAM 1530, various programs and data required for the operation of the electronic device 1500 are also stored. The processing device 1510, the ROM 1520, and the RAM 1530 are connected to each other via a bus 1540. An input / output (I / O) interface 1550 is also connected to the bus 1540.

[0107] Typically, the following devices may be connected to the I / O interface 1550: an input device 1560 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1570 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1580 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1590. The communication device 1590 may allow the electronic device 1500 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Fig.15 The electronic device 1500 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown, and the electronic device 1500 may alternatively implement or possess more or fewer devices.

[0108] For example, according to an embodiment of the present disclosure, the above-mentioned simulation method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the above-mentioned simulation method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1590, or installed from the storage device 1580, or installed from the ROM 1520. When the computer program is executed by the processing device 1510, the functions defined in the simulation method provided in the embodiment of the present disclosure can be implemented.

[0109] At least one embodiment of the present disclosure further provides a computer-readable storage medium, which stores non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by a computer, the above-mentioned simulation method can be implemented. Using the computer-readable storage medium, the tediousness of users preparing video clips of different image styles can be reduced, and the switching between video clips of different image styles can be made smoother.

[0110] Fig.16 A schematic diagram of a storage medium provided in some embodiments of the present disclosure. Fig.16 As shown, the storage medium 1600 stores non-transitory computer-readable instructions 1610. For example, when the non-transitory computer-readable instructions 1610 are executed by a computer, one or more steps in the simulation method described above are performed.

[0111] For example, the storage medium 1600 may be applied to the electronic device 1400. Fig.14 The memory 1420 in the electronic device 1400 is shown. For example, the relevant description of the storage medium 1600 can be referred to Fig.14 The corresponding description of the memory 1420 in the electronic device 1400 is shown and will not be repeated here.

[0112] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0113] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0114] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0115] There are a few points to note about this disclosure:

[0116] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures may refer to the general design.

[0117] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0118] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A distributed simulation system, comprising: A database, configured to store performance data when running various tasks in a real distributed system; A plurality of simulation nodes, wherein each of the simulation nodes comprises: a scheduler configured to obtain at least one simulation task; an allocator, configured to allocate virtual computing resources to each of the simulation tasks; The task simulator is configured to run the at least one simulation task according to a predetermined distributed strategy based on the virtual computing resources allocated to each simulation task and the performance data corresponding to each simulation task obtained from the database to obtain the running result.

2. The distributed simulation system according to claim 1, wherein: The real distributed system includes a plurality of computing nodes; The multiple tasks include multiple communication tasks and multiple computing tasks, the multiple communication tasks include tasks for transmitting multiple different data amounts within the computing nodes and tasks for transmitting multiple different data amounts between different computing nodes, and the multiple computing tasks include computing tasks corresponding to multiple operators respectively; The performance data corresponding to each of the communication tasks includes the communication bandwidth and the time required for communication when running the communication task, and the performance data corresponding to each of the computing tasks includes the video memory consumed by the computing task and the time required for execution.

3. The distributed simulation system according to claim 2, wherein: The at least one simulation task includes at least one simulation computing task and at least one communication simulation task; The task simulator is configured as follows: For each of the simulation computing tasks, searching the database for performance data corresponding to the operator and data volume of the simulation computing task as the performance data corresponding to the simulation computing task, and determining the running result of the simulation computing task based on the performance data, wherein the running result of the simulation computing task includes the video memory consumed by the simulation computing task and the time required for execution; For each of the communication simulation tasks, performance data corresponding to the communication range and data volume of the communication simulation task are searched from the database as the performance data corresponding to the communication simulation task, and the running result of the communication simulation task is determined based on the performance data, wherein the running result of the communication simulation task includes the communication bandwidth of the communication simulation task and the time required for communication.

4. The distributed simulation system according to any one of claims 1 to 3, wherein: The task simulator includes a computing task queue and a communication task queue; The task simulator is configured to place the at least one simulation task into one of the computing task queue and the communication task queue in a predetermined order, and execute the tasks in the computing task queue and the communication task queue in order.

5. The distributed simulation system according to claim 4, wherein: The computing task queue includes a first computing task, and the communication task queue includes a first communication task and a second communication task; There is no dependency relationship between the first computing task and the first communication task, and the first computing task and the first communication task are executed in parallel; The second communication task depends on the first computing task, and the second communication task starts to execute after the first computing task is completed.

6. The distributed simulation system according to claim 4, wherein: The predetermined order is determined based on the distributed strategy, which includes one of data parallelism, pipeline parallelism, and tensor parallelism.

7. The distributed simulation system according to any one of claims 1 to 3, further comprising: The result collector is configured to collect the running results of each simulation task in the multiple simulation nodes, and integrate the collected running results to obtain simulation results, wherein the simulation results include the total running time and the video memory change data during the running process.

8. The distributed simulation system according to any one of claims 1 to 3, wherein: The distributor includes: Tensor allocator, configured to allocate or release tensors; Video memory emulator, configured to allocate or release video memory.

9. A simulation method for a distributed simulation system, wherein the distributed simulation system comprises a plurality of simulation nodes, the method comprising: Each of the simulation nodes obtains at least one simulation task, and allocates computing resources to each of the simulation tasks; Each of the simulation nodes obtains the performance data corresponding to its own simulation task from the database, and based on the computing resources allocated to each of the simulation tasks and the performance data corresponding to each of the simulation tasks, runs the at least one simulation task according to a predetermined distributed strategy to obtain an running result, wherein the database stores the performance data when running multiple tasks in a real distributed system.

10. An electronic device comprising: processor; a memory storing one or more computer program modules; The one or more computer program modules are configured to be executed by the processor to implement the simulation method according to claim 9.

11. A computer-readable storage medium storing non-transitory computer-readable instructions, which can implement the simulation method of claim 9 when the non-transitory computer-readable instructions are executed by a computer.

Images