Network traffic modeling method and simulation system for large model training clusters

Abstract

The present invention discloses a network traffic modeling method and simulation system for large model training clusters. The method first constructs the overall process of large model distributed training by obtaining cluster configuration and model training information, and then quantifies the computing resource requirements of the model calculation process and the data transmission requirements of processes such as parameter or gradient synchronization during distributed training through computing and communication process modeling, and analyzes the communication traffic characteristics of parameter or gradient synchronization and intermediate activation value transmission under different parallel configurations. On this basis, a simulation system is constructed that can simulate the effects of different traffic scheduling strategies in a cluster network environment, providing practical and verification support for optimizing network scheduling strategies in large model training. Through this method and system, it is possible to quantitatively analyze the computing and communication efficiency during distributed training, and evaluate and compare the optimization effects of different traffic scheduling strategies.

Description

Technical Field

[0001] The present invention relates to the field of data center network technology, and in particular to a network traffic modeling method and simulation system for a large model training cluster. Background Art

[0002] The demand for computing resources during the training of large-scale deep learning models is increasing. Especially in distributed training environments, communication between nodes has a crucial impact on overall training efficiency. As the model scale increases, the communication bottleneck during training becomes more prominent. Therefore, building high-performance networks for intelligent computing has become an urgent issue.

[0003] Currently, network traffic scheduling algorithms often rely on real-world experimental validation. However, this approach is not only costly but can also affect the generalizability of the results due to the complexity of actual deployment conditions. To address this issue, some research has turned to simulation validation. However, differences between simulation tools and platforms result in poor comparability and consistency of experimental results. Furthermore, current network traffic involved in large-scale model training mostly relies on intuitive estimation or simple modeling methods, lacking precise network traffic quantification and simulation tools. This often limits strategies for optimizing training efficiency in practical applications and fails to fully reflect actual performance under diverse network conditions. Large-scale distributed model training relies on various parallel strategies that often require collaborative training across multiple computing nodes. These training tasks often involve large-scale data synchronization and transmission, frequent parameter or gradient synchronization, and uneven network load. This has made network traffic scheduling and optimization a hot topic of research. To address these challenges, it is particularly important to propose a network traffic modeling method and simulation system for large-scale model training clusters. By accurately modeling the characteristics of network traffic generation and transmission during training, it can provide a theoretical basis for optimizing network scheduling strategies in large-scale model training. Through simulation analysis, researchers can effectively verify different network architectures and traffic scheduling algorithms, thereby quantifying the computing and communication of the distributed training process and evaluating the effectiveness of traffic scheduling strategies. Summary of the Invention

[0004] The present invention addresses the shortcomings of existing technologies by providing a network traffic modeling method and simulation system for large-scale model training clusters. This system aims to address existing issues such as incomplete network traffic modeling and high traffic scheduling strategy verification costs during large-scale distributed training. By accurately modeling the characteristics of network traffic generation and transmission during training, and combining the cluster's computing resources and network topology, it provides an efficient quantitative analysis and simulation tool for network communication traffic.

[0005] To achieve the above objectives, the present invention provides a network traffic modeling method for a large model training cluster, comprising the following steps:

[0006] (1) Obtain basic parameter information related to model training, including the number of input samples and model structure; obtain performance parameter information of the equipment to be used;

[0007] (2) Process modeling: Conduct process modeling for the target distributed training strategy and construct a training process for large models in a hybrid parallel environment of data parallelism, tensor parallelism, and pipeline parallelism;

[0008] (3) Computational modeling: Model each computing node in the distributed training of large models and analyze the computational process of each layer during training, including the input layer, Transformer decoder, and output layer. At the same time, based on the number of input samples, model parameter scale, and training process, quantify the computational load and computational time of each computing node.

[0009] (4) Communication traffic modeling: Analyze the data transmission requirements of each training process based on the hybrid parallel configuration and quantify the amount of data exchange between each communication node; combine different parallel strategies and communication modes to quantify the communication volume and communication time of each communication node.

[0010] Furthermore, in step (2), the distributed training process is simulated according to the set hybrid parallel strategy: data parallelism is implemented in multiple rows of servers, and each row of servers represents the training process of a single model copy; tensor parallelism is implemented within each server, and a single server contains multiple GPUs, and the training tasks are split and distributed to these GPUs according to tensor parallelism; pipeline parallelism is implemented in each row of servers, and a single server in each row of servers represents a pipeline stage.

[0011] Furthermore, in step (3), the computational effort in the distributed training of the large model is concentrated in the multi-head attention layer, the feedforward network layer, and the fully connected layer.

[0012] Furthermore, the calculation of the multi-head attention layer includes query, key, value matrix mapping, attention calculation and multi-head splicing; the calculation of the feedforward network layer includes three linear transformations and bias term addition, one Swish transformation and one element-by-element multiplication; the calculation of the fully connected layer includes one matrix multiplication and one softmax transformation; the model calculation amount is: (the calculation amount of the multi-head attention layer + the calculation amount of the feedforward network layer) × the number of Transformer block layers + the calculation amount of the fully connected layer; based on the model calculation amount, the computational time of the model training is estimated.

[0013] Furthermore, step (4) includes the following sub-steps:

[0014] (4.1) Quantification from the perspective of data parallelism: Model parameters and optimizer state are copied to multiple GPUs, and then the training data is evenly distributed to these GPUs. The communication generated by data parallelism is mainly the process of synchronizing gradients and parameters between different model copies. The AllReduce communication mode is used to calculate the single communication volume between GPU cards.

[0015] (4.2) Quantification from the perspective of tensor parallelism: The parameter matrix of the Transformer block layer is split and computed in parallel on multiple GPUs. The communication caused by tensor parallelism mainly involves synchronizing the parameters or gradient matrices of different model parts. The AllReduce mode is used for data exchange, and the single communication volume between GPU cards is calculated.

[0016] (4.3) Quantification from the perspective of pipeline parallelism: Parameters of different layers of the model are distributed to different GPUs. The communication generated by pipeline parallelism is mainly the context interaction process between different stages of the model. A point-to-point communication model is used to calculate the single communication volume between GPU cards.

[0017] (4.4) Based on the above communication volume and the bandwidth information of related devices, estimate the single communication time required for model training.

[0018] To achieve the above objectives, the present invention also provides a network traffic simulation system for a large model training cluster, comprising:

[0019] Input component module: includes a parameter input component and a cluster configuration component; the parameter input component is used to obtain the total model parameters and related training parameters, including the weight matrix and optimizer status of each Transformer layer; the cluster configuration component is used to set the cluster's computing power standard and training strategy, including the configuration of each dimension in hybrid parallelism;

[0020] Computation and Communication Quantification Module: This module is used to use the information obtained and set by the input component module to perform quantitative analysis on each computing node and communication node in the distributed training process, thereby estimating the computing and communication requirements of each node.

[0021] Network optimization module: includes a network topology simulation submodule, a network status monitoring submodule, and a traffic scheduling submodule; the network topology simulation submodule is used to simulate the network topology of the distributed training cluster; the network status monitoring submodule is used to monitor the network status; the traffic scheduling submodule is used to execute the traffic scheduling strategy to achieve congestion control in the cluster network;

[0022] Effect evaluation module: used to evaluate the training and communication time consumption and analyze the optimization effect based on the quantitative results of computing and communication and the traffic scheduling strategy implemented by the network optimization module.

[0023] To achieve the above-mentioned objectives, the present invention also provides a network traffic modeling device for a large model training cluster, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the network traffic modeling method for a large model training cluster is implemented.

[0024] To achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the network traffic modeling method for large model training clusters is implemented.

[0025] To achieve the above-mentioned objectives, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the network traffic modeling method for a large model training cluster.

[0026] Compared with the existing technology, the beneficial effects of the present invention are: by accurately modeling the generation and transmission characteristics of network traffic during training, it can provide a theoretical basis for optimizing network scheduling strategies in large-scale model training. Through simulation analysis, different network architectures and traffic scheduling algorithms can be effectively verified, thereby achieving the quantification of computing and communication in the distributed training process and the evaluation of the effectiveness of traffic scheduling strategies. In general, the present invention can not only provide important technical support for large-scale model training, but also provide a powerful tool for application optimization in related fields. Compared with traditional real-machine training methods, the present invention provides a powerful tool for the evaluation of network traffic scheduling algorithms in large-scale model training, significantly reducing the high economic cost of verifying traffic scheduling algorithms in actual large-scale model distributed training clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 This is a diagram of the architecture of the present invention;

[0030] Figure 3 This is a schematic diagram of the distributed training process of the present invention;

[0031] Figure 4 is a schematic diagram of the system of the present invention;

[0032] Figure 5It is a structural diagram of the device of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0034] In the scenario of large-model distributed training, the present invention combines the Transformer architecture with a hybrid parallel strategy to design a modeling and quantification method for computing and communication, so as to realize a network traffic simulation system for large-model distributed training intelligent computing clusters.

[0035] like Figure 1 and Figure 2 As shown, the present invention provides a network traffic modeling method for a large model training cluster, the method comprising the following steps:

[0036] (1) Obtain information related to cluster size, model parameters, and training configuration: Obtain basic parameters related to model training, including the number of input samples, model structure, vocabulary dimension, and dimensional information of each parameter matrix (such as batch size, embedding dimension, input sequence length, etc.). At the same time, obtain the performance parameter information of the device to be used and the hybrid parallel strategy adopted; among them, the performance parameter information of the device to be used includes GPU card computing power (i.e., the floating-point computing power of GPU (Graphics Processing Unit)), cluster network topology, link bandwidth (i.e., the end-to-end communication bandwidth of interconnected devices), etc.; the hybrid parallel strategy adopted includes data parallelism, tensor parallelism, and pipeline parallelism; and obtain information such as the expected overall cluster size and GPU parallel dimension under each parallel dimension. This information will be used for subsequent computational and communication modeling and support quantitative analysis of modeling results.

[0037] (2) Process modeling: Conduct process modeling for the target distributed training strategy and construct a training process for large-scale models in a hybrid parallel environment such as data parallelism, tensor parallelism, and pipeline parallelism.

[0038] See also Figure 3 , modeling the process of the proposed distributed training strategy. Based on the set hybrid parallelism strategy, the distributed training process is simulated: data parallelism is implemented across multiple rows of servers, with each row representing the training process of a single model replica; tensor parallelism is implemented within each server, with a single server containing multiple GPUs, and training tasks are split and distributed across these GPUs based on tensor parallelism; and pipeline parallelism is implemented within each row of servers, with each server in each row representing a pipeline stage.

[0039] (3) Computational Modeling: Based on model-related information, analyze the computational process of each layer during training, especially the computation of the input layer, Transformer decoder, and output layer. At the same time, quantify the computational load and computational time of each computing node based on factors such as the number of input samples, the scale of model parameters (i.e., the number of neurons in each layer of the neural network, the size of the weight matrix, etc.), and the training steps (i.e., the task division and training process of large models based on hybrid parallel strategies in distributed intelligent computing clusters).

[0040] Specifically, we modeled each computing node in the distributed training of the large model and analyzed the specific computational process of each layer during the entire training process, including the input layer, Transformer decoder, and output layer. Based on the basic parameter information related to model training obtained in step (1), we estimated the computing power required by each computing node during the training process. The computational workload during the distributed training of the large model is concentrated in the multi-head attention layer (MHA), feed forward network layer (FFN), and fully connected layer. Here, we take the decoder architecture (the decoder part of the transformer architecture) model as an example for analysis.

[0041] (3.1) The calculation of the multi-head attention layer includes query, key, value matrix mapping, attention calculation and multi-head splicing.

[0042] Query, key, value matrix mapping ,in, represents the input matrix of the multi-head attention layer, represents the query weight matrix in the multi-head attention layer, represents the key weight matrix in the multi-head attention layer, represents the value weight matrix in the multi-head attention layer, , , the operations involved are [ b,s,h ] dimension and [ h,d ] dimension matrix multiplication, the amount of operation is 3×2bshd=6bshd; b Indicates the input batch size, that is, the number of sequences contained in a batch; s represents the length of the input sequence, h Indicates the embedding dimension; d Denotes the dimension of a single head in the multi-head attention layer (d=h / n), and n denotes the number of heads in the multi-head attention layer.

[0043] Single-head attention calculation , the calculations involved include 、 and ,in Zoom and The computational cost of the transformation is 4 bs 2 , which is quite different from other computational costs and can be ignored. The other two multiplication operations involving query, key, and value matrices have a computational cost of 4bs. 2 d; represents the tensor parallelism, Indicates the degree of data parallelism.

[0044] Multi-head splicing ,in, Represents the mapping weight matrix after multiple attention heads are concatenated, , the amount of calculation involved in matrix splicing is negligible. The operations involved are [ b, s,h ] dimension and [ h,h ] dimension matrix multiplication, the amount of operation is 2bsh 2 .

[0045] The total computational cost of the multi-head attention layer is: (6bshd+4bs 2 d)×n+2bsh 2 =8bsh 2 +4bs 2 h.

[0046] (3.2) The calculation of the feedforward network layer FFN includes three linear transformations and bias term additions, one Swish transformation and one element-by-element multiplication.

[0047]

[0048] in, Represent the first linear transformation weight matrix and bias term of the SwiGLU activation function, respectively. Represent the second linear transformation weight matrix and bias term of the SwiGLU activation function, respectively. Represent the weight matrix and bias term of the second layer linear transformation of the feedforward network layer, , represents the hidden dimension of the feedforward network layer. The computational cost of bias addition, Swish transform and element-by-element multiplication is 2bsh'+bsh, 3bsh' and bsh' respectively, which can be ignored. The computational cost of the cubic linear transform occupies the main part of the computational cost of the feedforward network layer, specifically The total amount of matrix multiplication is 3×2bshh'=6bshh'.

[0049] (3.3) Fully connected layer The calculation includes one matrix multiplication and one softmax transformation.

[0050]

[0051] in, represents the input matrix of the last layer, represents the mapping weight matrix that maps the input matrix to the vocabulary dimension, , the operations involved are [ b,s,h ] dimension and [ h,V ] dimension matrix multiplication, the amount of operation is 2bshV; V Indicates the size of the input mapping vocabulary.

[0052] (3.4) According to the above analysis of the amount of calculation, the total amount of calculation of the model is (8bsh 2 +4bs 2 h+6bshh')×L+2bshV, where L represents the number of Transformer block layers. Combined with the computing power information of related devices, the computational time of model training is preliminarily estimated. The computational time is equal to the total computational amount divided by the total computing power, where the total computing power is the product of GPU utilization, the computing power of a single GPU, and the number of GPUs.

[0053] (4) Communication traffic modeling: Analyze the data transmission requirements of each training step based on the hybrid parallel configuration and quantify the amount of data exchanged between communication nodes, including parameter transmission, gradient synchronization, model update, etc. Combining different parallel strategies and communication modes, quantify the communication volume and communication time of each communication node.

[0054] Analyze the data transfer requirements of each training step based on the hybrid parallel configuration, and analyze the specific communication volume from the perspective of different parallel methods. Here, we can quantify it from the perspectives of data parallelism, pipeline parallelism, and tensor parallelism. The specific analysis process is as follows:

[0055] (4.1) Data parallelism is a method for improving training throughput. It replicates model parameters and optimizer state across multiple GPUs and then evenly distributes training data across these GPUs. The communication generated by data parallelism primarily involves synchronizing gradients and parameters between different model replicas, using the AllReduce communication model. The amount of communication between GPUs in a single transaction can be expressed as: , at this time G=D. Among them 4h 2 Represents the four weight matrices to be updated in the MHA layer (W Q 、W Q 、W Q and W O ) parameter; 3hh' represents the three weight matrices to be updated in the FFN layer (W U 、W G and WD ) parameter quantity; Indicates the number of Transformer block layers contained in a single card; Indicates the amount of data transmitted by Ring AllReduce data shards; 2Bytes indicates that data is transmitted with 16-bit precision, and each parameter size is 2Bytes. represents the hidden layer dimension of the feedforward network layer, represents the tensor parallelism, represents the pipeline parallelism, Indicates the number of Transformerblock layers, Indicates the number of GPUs in the current parallel mode (usually equal to the degree of parallelism), and D represents the data parallelism.

[0056] (4.2) Tensor parallelism splits the parameter matrix of the Transformer block layer and computes it in parallel on multiple GPUs, thereby more efficiently utilizing the GPU's computing power. The communication triggered by tensor parallelism mainly involves synchronizing the parameters or gradient matrices of different model parts. The communication method uses the AllReduce mode for data exchange. The single communication volume between GPU cards can be expressed as: , at this time G=T. Among them, Indicates the size of the activation value (forward propagation) / gradient (backward propagation) parameters to be synchronized between GPU cards; Indicates the number of stages contained in a single server; Indicates the amount of data transmitted by RingAllReduce data shards; 2Bytes means each parameter size is 2Bytes. Indicates the micro-batch size used in pipeline parallelization.

[0057] (4.3) Pipeline parallelism aims to distribute the parameters of different layers of a large language model to different GPUs. In practice, consecutive Transformer layers can be loaded onto the same GPU to reduce the cost of transferring hidden states or gradients between GPUs. Pipeline parallelism primarily involves contextual interactions between different stages of the model, using a point-to-point communication model. The amount of single inter-GPU communication can be expressed as: .in Indicates the size of the activation value parameters to be transmitted between pipeline stages; Indicates the number of tensor parallelism within a single pipeline stage; 2Bytes means that the size of each parameter is 2Bytes.

[0058] (4.4) Based on the above communication volume analysis and the bandwidth information of related devices, a preliminary estimate of the single communication time required for model training is made; the communication time is equal to the communication volume divided by the bandwidth.

[0059] A simulation system is constructed based on the quantitative characteristics of computation and communication in steps (3) and (4), and each quantization process is modularized to simulate the computation and communication process of distributed training of large models.

[0060] Traffic scheduling strategy evaluation: Through the simulation system, the effects of different traffic scheduling strategies in the cluster network environment are simulated, multiple scheduling strategies are set and the corresponding quantitative results are obtained, so as to analyze and evaluate the optimization effect of each strategy.

[0061] Specifically, based on the constructed simulation system, a network optimization module was added to further simulate traffic scheduling strategies in cluster network environments. This module includes a network topology simulation submodule, a network status monitoring submodule, and a traffic scheduling submodule, which are respectively used to simulate network topology, monitor real-time traffic characteristics in the network, and implement different traffic scheduling strategies. By monitoring network congestion in real time, this module can promptly implement traffic scheduling strategies, thereby achieving effective congestion control.

[0062] Corresponding to the aforementioned embodiment of the network traffic modeling method for large model training clusters, the present application also provides a network traffic simulation system for large model training clusters.

[0063] Figure 4 This is a schematic diagram of the network traffic simulation system for large model training clusters. Figure 4 , the system may include:

[0064] Input component module: This includes the parameter input component and the cluster configuration component. The parameter input component is used to obtain the total model parameters and related training parameters, including the weight matrix and optimizer status of each Transformer layer. The cluster configuration component is used to set the cluster's computing power standards and training strategies, including GPU (Graphics Processing Unit) card computing power, link bandwidth, and the configuration of various dimensions in hybrid parallelism.

[0065] Computation and Communication Quantification Module: This module uses the information obtained and set by the input component module to perform quantitative analysis on each computing node and communication node in the distributed training process, thereby estimating the computing and communication requirements of each node.

[0066] Network Optimization Module: This module includes a network topology simulation submodule, a network status monitoring submodule, and a traffic scheduling submodule. The network topology simulation submodule simulates the network topology of the distributed training cluster. The network status monitoring submodule monitors the network status, recording interface statistics, bandwidth usage, and network activity flows. The traffic scheduling submodule implements various traffic scheduling strategies to achieve congestion control in the cluster network.

[0067] Effect evaluation module: Based on the quantitative results of computing and communication and the traffic scheduling strategy implemented by the network optimization module, the training and communication time consumption is evaluated and the optimization effect is analyzed.

[0068] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0069] See also Figure 5 An embodiment of the present invention provides a network traffic modeling device for a large model training cluster, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the network traffic modeling method for a large model training cluster in the above embodiment.

[0070] The embodiment of the network traffic modeling device for large model training clusters provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5 As shown, this is a hardware structure diagram of a network traffic modeling device for a large model training cluster provided by the present invention, where any device with data processing capability is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0071] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0072] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0073] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the network traffic modeling method for a large model training cluster in the above embodiment is implemented.

[0074] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0075] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the network traffic modeling method for a large model training cluster.

[0076] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A network traffic modeling method for large model training clusters, characterized by: The following steps are involved: (1) Obtain basic parameter information related to model training, including the number of input samples and model structure; Obtain performance parameter information of the equipment to be adopted; (2) Process modeling: Conduct process modeling for the target distributed training strategy and construct a training process for large models in a hybrid parallel environment of data parallelism, tensor parallelism, and pipeline parallelism; (3) Computational modeling: Model each computing node in the distributed training of large models and analyze the computational process of each layer during training, including the input layer, Transformer decoder, and output layer. At the same time, based on the number of input samples, model parameter scale, and training process, quantify the computational load and computational time of each computing node. (4) Communication traffic modeling: Analyze the data transmission requirements of each training process based on the hybrid parallel configuration and quantify the amount of data exchange between each communication node; combine different parallel strategies and communication modes to quantify the communication volume and communication time of each communication node; (4.1) Quantification from the perspective of data parallelism: Model parameters and optimizer state are copied to multiple GPUs, and then the training data is evenly distributed to these GPUs. The communication generated by data parallelism is mainly the process of synchronizing gradients and parameters between different model copies. The AllReduce communication mode is used to calculate the single communication volume between GPU cards. (4.2) Quantification from the perspective of tensor parallelism: The parameter matrix of the Transformer block layer is split and computed in parallel on multiple GPUs. The communication caused by tensor parallelism mainly involves synchronizing the parameters or gradient matrices of different model parts. The AllReduce mode is used for data exchange, and the single communication volume between GPU cards is calculated. (4.3) Quantification from the perspective of pipeline parallelism: Parameters of different layers of the model are distributed to different GPUs. The communication generated by pipeline parallelism is mainly the context interaction process between different stages of the model. A point-to-point communication model is used to calculate the single communication volume between GPU cards. (4.4) Based on the above communication volume and the bandwidth information of related devices, estimate the single communication time required for model training.

2. The network traffic modeling method for large model training cluster according to claim 1 is characterized in that: In step (2), the distributed training process is simulated according to the set hybrid parallel strategy: data parallelism is implemented in multiple rows of servers, and each row of servers represents the training process of a single model replica; Tensor parallelism is implemented within each server. A single server contains multiple GPUs, and training tasks are split according to tensor parallelism and distributed to these GPUs. Pipeline parallelism is implemented in each row of servers. A single server in each row represents a pipeline stage.

3. The network traffic modeling method for large model training cluster according to claim 1 is characterized in that: In step (3), the computational workload during the distributed training of the large model is concentrated in the multi-head attention layer, feedforward network layer, and fully connected layer.

4. The network traffic modeling method for large model training cluster according to claim 3 is characterized in that: The calculation of the multi-head attention layer includes query, key, value matrix mapping, attention calculation and multi-head splicing; the calculation of the feedforward network layer includes three linear transformations and bias term addition, one Swish transformation and one element-by-element multiplication; the calculation of the fully connected layer includes one matrix multiplication and one softmax transformation; the model calculation amount is: (the calculation amount of the multi-head attention layer + the calculation amount of the feedforward network layer) × the number of Transformer block layers + the calculation amount of the fully connected layer; based on the model calculation amount, the computational time of the model training is estimated.

5. A network traffic simulation system for a large model training cluster that implements the method according to claim 1, characterized in that: include: Input component module: includes parameter input component and cluster configuration component; The parameter input component is used to obtain the total model parameters and related training parameters, including the weight matrix and optimizer status of each Transformer layer; the cluster configuration component is used to set the cluster's computing power standard and training strategy, including the configuration of each dimension in hybrid parallelism; Computation and Communication Quantification Module: This module is used to use the information obtained and set by the input component module to perform quantitative analysis on each computing node and communication node in the distributed training process, thereby estimating the computing and communication requirements of each node. Network optimization module: includes a network topology simulation submodule, a network status monitoring submodule, and a traffic scheduling submodule; the network topology simulation submodule is used to simulate the network topology of the distributed training cluster; the network status monitoring submodule is used to monitor the network status; the traffic scheduling submodule is used to execute the traffic scheduling strategy to achieve congestion control in the cluster network; Effect evaluation module: used to evaluate the training and communication time consumption and analyze the optimization effect based on the quantitative results of computing and communication and the traffic scheduling strategy implemented by the network optimization module.

6. A network traffic modeling device for a large model training cluster, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements the network traffic modeling method for a large model training cluster as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the network traffic modeling method for a large model training cluster as described in any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the network traffic modeling method for a large model training cluster as described in any one of claims 1 to 4 is implemented.

Images