Simulation optimization method and apparatus for network architecture of intelligent computing center, and device and medium

Abstract

The present application relates to the technical field of electronic digital data processing, and particularly relates to a simulation optimization method and apparatus for a network architecture of an intelligent computing center, and a device and a medium. The method comprises: acquiring training configuration information of a large language model to be trained; on the basis of the training configuration information and a preset discrete event simulation strategy, using a preset simulator to perform training simulation on said large language model, so as to obtain a simulation result, wherein the preset simulator is obtained by using a preset ECS framework; and on the basis of the simulation result, adjusting a network architecture of an intelligent computing center, and obtaining an optimal network architecture of the intelligent computing center. In this way, a network simulator of an intelligent computing center is designed by using an entity component system framework, such that a complete simulation process of the training of a large language model can be implemented, so as to search for the optimal network architecture of the intelligent computing center; and the problem of it not being possible to support the simulation of a large-scale network of the intelligent computing center due to the fact that existing network simulators lack scalability is solved, thereby improving the efficiency and accuracy of simulating a large-scale network of an intelligent computing center.

Description

Simulation optimization method, device and equipment of intelligent computing center network architecture and medium

[0001] Cross-reference of related applications

[0002] The present application is based on the Chinese patent application with the application number 202410466606.8 and the application date of April 18, 2024, and claims the priority of the Chinese patent application, the entire content of which is hereby incorporated by reference into the present application. TECHNICAL FIELD

[0003] The present application relates to the technical field of digital data processing, in particular to a simulation optimization method, device and equipment of intelligent computing center network architecture and medium. BACKGROUND

[0004] In recent years, with the rapid development of large language models such as ChatGPT (Chat Generative Pre-trained Transformer), intelligent computing centers have become key computing infrastructure for hosting large model training. Intelligent computing centers are new data centers that are intensively built around intelligent computing power such as GPUs (Graphic Processing Units) and AI (Artificial Intelligence) accelerator cards, and provide the computing power, data services and network services required for artificial intelligence applications. Intelligent computing center networks provide high-speed interconnection for large-scale intelligent computing power, and the architecture, protocol and other parameters of the network have an important impact on the performance of AI applications (such as large model training and inference). Currently, companies such as Google and Amazon are building intelligent computing centers containing tens of thousands of servers, and there is an urgent need for a set of integrated simulation techniques and devices from computing to network to support the simulation of large-scale intelligent computing center networks to find the optimal intelligent computing center network architecture through simulation.

[0005] In related technologies, the most advanced simulator OMNet++ (Objective Modular Network Testbed in C++, a modular network simulation framework based on C++) is widely used in modeling, simulation and analysis of network fields and is considered a very powerful tool. It takes more than 9 days to simulate a large-scale data center network (DCNs) on 8 machines.

[0006] However, in the face of intelligent computing centers with tens of thousands of servers, the simulation performance of existing network simulators cannot meet the demand. Since existing simulators do not have scalability, they cannot efficiently support the simulation of large-scale intelligent computing center networks.

[0007] SUMMARY

[0008] The application provides a simulation optimization method of an intelligent computing center network architecture to solve the problem that existing network simulators do not have scalability and cannot support simulation of a large-scale intelligent computing center network, thereby improving the efficiency and accuracy of simulating a large-scale intelligent computing center network.

[0009] To achieve the above-mentioned purpose, the first aspect of the application provides a simulation optimization method of an intelligent computing center network architecture, comprising the following steps:

[0010] Obtaining training configuration information of a to-be-trained large language model;

[0011] Based on the training configuration information and a preset discrete event simulation strategy, training simulation is performed on the to-be-trained large language model by using a preset simulator, and a simulation result is obtained, wherein the preset simulator is obtained by using a preset ECS (Entity Component System) framework.

[0012] Adjusting the intelligent computing center network architecture according to the simulation result, and obtaining an optimal intelligent computing center network architecture.

[0013] According to one embodiment of the application, before the training simulation is performed on the to-be-trained large language model by using the preset simulator based on the training configuration information and the preset discrete event simulation strategy, the following steps are further included:

[0014] Modeling and simulating a traffic generation process, a set communication process, a traffic sending process, a congestion control process, a traffic forwarding process and a traffic control process in the training process of the target large language model, respectively, to obtain a traffic modeling module, a set communication simulation module, a transmission protocol simulation module and a bottom-layer network simulation module;

[0015] Based on the preset ECS framework, the traffic modeling module, the set communication simulation module, the transmission protocol simulation module and the bottom-layer network simulation module are combined to obtain a preset simulator.

[0016] According to one embodiment of the application, the modeling and simulation of the traffic generation process in the training process of the target large language model comprises:

[0017] Obtaining traffic data of the training process of the target large language model, and generating a training set based on the traffic data;

[0018] Constructing an initial deep learning generative model, training the initial deep learning generative model by using the training set based on a preset loss function and a preset optimization algorithm, and obtaining a final deep learning generative model;

[0019] Obtain network structure information, parameter size and communication mode of the target large language model, input the network structure information, the parameter size and the communication mode to the final deep learning generative model, and output a predicted traffic mode, wherein the predicted traffic mode includes the size and interval of traffic.

[0020] According to an embodiment of the present application, the set communication process in the training process of the target large language model is modeled and simulated, comprising:

[0021] The set communication process is divided into at least one communication event, and a discrete communication event model is established based on the at least one communication event;

[0022] Using a preset event-driven strategy, the at least one communication event is triggered to perform a corresponding communication operation according to a first preset trigger condition.

[0023] According to an embodiment of the present application, the traffic sending process and the congestion control process in the training process of the target large language model are modeled and simulated, comprising:

[0024] Using the preset ECS framework, a first network node entity and a first communication component are constructed, wherein each first network node entity includes at least one first communication component;

[0025] Based on the preset event-driven strategy, each network transmission operation is defined as a corresponding at least one transmission event, and based on a preset event scheduler, the execution order and timing relationship of the at least one transmission event are managed;

[0026] According to the first network node entity and the first communication component, it is judged whether the trigger condition of the at least one transmission event meets a second preset trigger condition, and when the trigger condition of the at least one transmission event meets the second preset trigger condition, the at least one transmission event is triggered to perform a corresponding transmission operation.

[0027] According to an embodiment of the present application, the traffic forwarding process and the traffic control process in the training process of the target large language model are modeled and simulated, comprising:

[0028] Using the preset ECS framework, a second network node entity and a second communication component are constructed, wherein each second network node entity includes at least one second communication component;

[0029] Based on the preset event-driven strategy, each network transmission operation is defined as a corresponding at least one transmission event, and based on a preset event scheduler, the execution order and timing relationship of the at least one transmission event are managed;

[0030] According to the second network node entity and the second communication component, it is judged whether a trigger condition of the at least one transmission event meets a third preset trigger condition, and when the trigger condition of the at least one transmission event meets the third preset trigger condition, the at least one transmission event is triggered to perform a corresponding transmission operation.

[0031] According to the simulation optimization method of the intelligent computing center network architecture provided in the embodiments of the present application, by obtaining the training configuration information of the to-be-trained large language model, the training configuration information and the preset discrete event simulation strategy can be used to train and simulate the to-be-trained large language model by using the preset simulator, so as to obtain a simulation result, wherein the preset simulator is obtained by using a preset ECS framework, and the intelligent computing center network architecture is adjusted according to the simulation result, so as to obtain an optimal intelligent computing center network architecture. Therefore, by using the entity component system framework to design the intelligent computing center network simulator, a complete simulation process of the large language model training can be realized to find the optimal intelligent computing center network architecture, and the problem that the existing network simulator does not have scalability and cannot support simulation of a large-scale intelligent computing center network is solved, thereby improving the efficiency and accuracy of simulating the large-scale intelligent computing center network.

[0032] To achieve the above object, the second aspect of the present application provides a simulation optimization device of an intelligent computing center network architecture, comprising:

[0033] The obtaining module is configured to obtain training configuration information of a to-be-trained large language model.

[0034] The training module is configured to train and simulate the to-be-trained large language model by using a preset simulator based on the training configuration information and a preset discrete event simulation strategy, so as to obtain a simulation result, wherein the preset simulator is obtained by using a preset entity component system (ECS) framework.

[0035] The processing module is configured to adjust the intelligent computing center network architecture according to the simulation result, and obtain an optimal intelligent computing center network architecture.

[0036] According to an embodiment of the present application, before the training and simulation of the to-be-trained large language model by using the preset simulator based on the training configuration information and the preset discrete event simulation strategy, the training module further comprises:

[0037] The modeling and simulation unit is configured to model and simulate a traffic generation process, a set communication process, a traffic sending process, a congestion control process, a traffic forwarding process and a traffic control process in the training process of the target large language model, respectively, to obtain a traffic modeling module, a set communication simulation module, a transmission protocol simulation module and a bottom-layer network simulation module.

[0038] A combination unit is configured to combine the traffic modeling module, the set communication simulation module, the transmission protocol simulation module and the bottom-layer network simulation module based on the preset ECS framework to obtain a preset simulator.

[0039] According to an embodiment of the present application, the modeling simulation unit is specifically configured to:

[0040] Obtain traffic data of a training process of the target large language model, and generate a training set based on the traffic data;

[0041] Construct an initial deep learning generative model, train the initial deep learning generative model based on a preset loss function and a preset optimization algorithm using the training set, and obtain a final deep learning generative model;

[0042] Obtain network structure information, parameter size and communication mode of the target large language model, input the network structure information, the parameter size and the communication mode into the final deep learning generative model, and output a predicted traffic mode, wherein the predicted traffic mode includes the size and interval of traffic.

[0043] According to an embodiment of the present application, the modeling simulation unit is specifically configured to:

[0044] Decompose a set communication process into at least one communication event, and establish a discrete communication event model based on the at least one communication event;

[0045] Trigger the at least one communication event to perform a corresponding communication operation according to a first preset trigger condition using a preset event-driven strategy.

[0046] According to an embodiment of the present application, the modeling simulation unit is specifically configured to:

[0047] Construct a first network node entity and a first communication component using the preset ECS framework, wherein each first network node entity includes at least one first communication component;

[0048] Define each network transmission operation as a corresponding at least one transmission event based on the preset event-driven strategy, and manage the execution order and timing relationship of the at least one transmission event based on a preset event scheduler;

[0049] Determine whether a trigger condition of the at least one transmission event meets a second preset trigger condition according to the first network node entity and the first communication component, and trigger the at least one transmission event to perform a corresponding transmission operation when the trigger condition of the at least one transmission event meets the second preset trigger condition.

[0050] According to one embodiment of the present application, the modeling simulation unit is specifically used for:

[0051] The second network node entity and the second communication component are constructed by using the preset ECS framework, wherein each second network node entity includes at least one second communication component;

[0052] Based on the preset event-driven strategy, each network transmission operation is defined as a corresponding at least one transmission event, and based on the preset event scheduler, the execution order and the timing relationship of the at least one transmission event are managed;

[0053] According to the second network node entity and the second communication component, it is judged whether the triggering condition of the at least one transmission event meets a third preset triggering condition, and when the triggering condition of the at least one transmission event meets the third preset triggering condition, the at least one transmission event is triggered to perform a corresponding transmission operation.

[0054] According to the simulation optimization device for the intelligent computing center network architecture provided in the embodiments of the present application, by obtaining the training configuration information of the to-be-trained large language model, the training configuration information and the preset discrete event simulation strategy can be used to train and simulate the to-be-trained large language model by using the preset simulator, so as to obtain a simulation result. The preset simulator is obtained by using the preset ECS framework, and the intelligent computing center network architecture is adjusted according to the simulation result, so as to obtain an optimal intelligent computing center network architecture. Therefore, by using the entity component system framework to design the intelligent computing center network simulator, the complete simulation process of the large language model training can be realized to find the optimal intelligent computing center network architecture, thereby solving the problem that the existing network simulator does not have scalability and cannot support the simulation of large-scale intelligent computing center networks, and thereby improving the efficiency and accuracy of simulating large-scale intelligent computing center networks.

[0055] To achieve the above object, the third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the simulation optimization method for the intelligent computing center network architecture as described in the above embodiments.

[0056] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the simulation optimization method for the intelligent computing center network architecture as described in the above embodiments.

[0057] To achieve the above object, the fifth aspect of the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the simulation optimization method for the intelligent computing center network architecture as described in the above embodiments.

[0058] Additional aspects and advantages of the present application will be made apparent from the following description of the application. BRIEF DESCRIPTION OF DRAWINGS

[0059] The above and / or additional aspects and advantages of the present application will become apparent and be more readily understood from the following description, by reference to which:

[0060] FIG. 1 is a flowchart of a simulation optimization method of a network architecture of an intelligent computing center according to an embodiment of the present application;

[0061] FIG. 2 is a schematic diagram of a network design of an intelligent computing center according to an embodiment of the present application;

[0062] FIG. 3 is a simulation schematic diagram of a large language model training based on a Data Oriented Infrastructure Simulator (DOIS) of an intelligent computing center network simulator according to an embodiment of the present application;

[0063] FIG. 4 is a schematic diagram of a large language model training process modeling flow according to an embodiment of the present application;

[0064] FIG. 5 is a block schematic diagram of a simulation optimization device of a network architecture of an intelligent computing center according to an embodiment of the present application;

[0065] FIG. 6 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the drawings, in which the same or similar components are denoted by the same or similar reference numerals throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0067] The simulation optimization method, device, equipment and medium of the network architecture of an intelligent computing center according to the embodiments of the present application are described below with reference to the accompanying drawings.

[0068] FIG. 1 is a flowchart of a simulation optimization method of a network architecture of an intelligent computing center according to an embodiment of the present application;

[0069] Before introducing the simulation optimization method of the network architecture of an intelligent computing center according to the embodiments of the present application, the background and related technologies of the present application are briefly introduced.

[0070] The computing power requirement of a deep neural network (DNN) model is rapidly growing, far exceeding the performance and capacity of a single neural-network processing unit (NPU, such as a GPU or a tensor processing unit (TPU)). For example, from the BERT (Bidirectional Encoder Representations from Transformers, a pre-training model) model to the GPT-3 (Generative Pre-trained Transformer 3, third-generation general pre-training transformer), the required second computing power to train the GPT-3 model is 1800 times in two years. The field of artificial intelligence has now entered an era of models requiring hundreds of billions of parameters, which require tens of terabytes (TB) of memory and ZB (Zettabyte, 100 trillion bytes) level floating point operations to train the model, so efficient distributed training of large language models is of great significance.

[0071] It can be understood that it is challenging to design an efficient distributed training system, because there are many design choices for a wisdom computing center network, such as parallel strategies, AI accelerator selection, network topology, network bandwidth, and scheduling strategies, as shown in FIG. 2, and these design choices are interdependent, requiring collaborative design of the hardware and software of the training platform.

[0072] In the related art, the most advanced simulator, OMNet++, takes more than 9 days to simulate a large-scale data center network on 8 machines. Currently, companies such as Google and Amazon are building wisdom computing centers containing tens of thousands of servers, and such simulation performance is far from meeting the needs of today's "super large-scale" networks.

[0073] The problems existing in the existing simulators include: on the one hand, they cannot completely simulate the end-to-end process of large language model training; on the other hand, the existing simulators have poor performance and cannot support simulation of large-scale wisdom computing center networks. The fundamental reason for the poor performance of the existing simulators is that they are based on an object-oriented design (OOD) software architecture. Research shows that under the current computer architecture, the OOD software architecture is not friendly to CPU (Central Processing Unit) cache mechanisms and automatic parallelism, because the OOD architecture ignores the importance of data layout, and inefficient data layout leads to a very high cache miss rate, thereby reducing the efficiency of network simulation.

[0074] In contrast, the program design method based on data-oriented design considers data layout, and thus has better cache friendliness. DOD stores data of the same type (e.g., the same attribute of all objects) in adjacent memory, and in DOD, data and processing logic are decoupled, and all processing logic is only responsible for the conversion of data, so the processing logic is easy to parallelize, and in network simulation, the data required by network functions (such as forwarding) can be loaded into the CPU cache at the same time and processed in parallel on a multi-core CPU.

[0075] Therefore, the embodiments of the present application adopt a DOD software framework widely used in the game field, that is, an ECS framework, to design a brand-new intelligent center network simulator, so that researchers can simulate a large model distributed training system in a virtual world to efficiently train future DNN models.

[0076] For example, as shown in FIG. 1, the simulation optimization method of the intelligent center network architecture includes the following steps:

[0077] In step S101, training configuration information of a large language model to be trained is obtained.

[0078] It can be understood that the training configuration information of the large language model to be trained can include, for example, model structure, training iteration number, collective communication algorithm, network topology and bandwidth, etc. The training configuration information can be provided by the user. Among them, the model structure determines how the model organizes and processes input data, and how to generate output; the training iteration number refers to the number of times the large language model to be trained is trained on the entire training set; in distributed training, the collective communication algorithm is used to synchronize model parameters and gradients between multiple nodes, and these algorithms help to ensure that the model on each node remains consistent, so that they can learn together throughout the training process; the network topology describes the connection relationship between computing nodes, including the communication path and delay between nodes, and in large-scale distributed training, the network topology is crucial to training efficiency; the bandwidth refers to the amount of data that the network can transmit in a certain time, and in large language model training, the bandwidth determines the speed of transmission of model parameters and gradients between different nodes, the higher the bandwidth, the faster the data transmission speed, and the higher the training efficiency.

[0079] As can be seen, obtaining and understanding the training configuration information of the large language model to be trained is crucial to optimizing the training efficiency, performance of the large language model and solving potential problems.

[0080] In step S102, based on the training configuration information and a preset discrete event simulation strategy, a preset simulator is used to train and simulate the large language model to be trained, to obtain a simulation result, wherein the preset simulator is obtained by using a preset ECS framework.

[0081] The preset discrete event simulation strategy refers to a computer simulation technique for simulating a system driven by a series of discrete events, in which the system state changes only when a specific event occurs.

[0082] That is, based on the training configuration information of the large language model to be trained provided by the user, the preset simulator can simulate the training process of the large language model using the preset discrete event simulation strategy, and obtain the training simulation result, which includes training time and power consumption, etc.

[0083] In the embodiments of the present application, the preset ECS framework can be used to implement the preset simulator (i.e., the network simulator of the intelligent computing center), which can support multi-thread parallelism and can concurrently execute the same logic of all devices when executing a certain specific logic. The preset ECS framework includes three core concepts: (1) Entity: an entity is a general object in a simulation scenario, which is different from an object in OOD. The entity in DOD is only used to identify data belonging to the same object, and is usually represented by a unique identifier (Unique Identifier, UID); (2) Component: a component represents a specific attribute or data of an entity. For example, in a physics engine, each entity has a "mass" component. Component data of the same type are stored together in physical memory and are indexed by the UID of the entity, so that the logical operation of processing multiple entities can efficiently access the memory; (3) System: a system represents the logic for processing components, which acts on all entities with certain components and converts component data from the current state to the next state. A system can describe a certain aspect of the network simulation process.

[0084] For ease of understanding, how to obtain the preset simulator is described in detail below.

[0085] As a possible implementation manner, in some embodiments, before training simulation of the large language model to be trained is performed based on the training configuration information and the preset discrete event simulation strategy using the preset simulator, the method further includes: modeling and simulating a traffic generation process, a collective communication process, a traffic sending process, a congestion control process, a traffic forwarding process and a traffic control process in the training process of the target large language model, respectively, to obtain a traffic modeling module, a collective communication simulation module, a transmission protocol simulation module and a bottom-layer network simulation module; and combining the traffic modeling module, the collective communication simulation module, the transmission protocol simulation module and the bottom-layer network simulation module based on the preset ECS framework to obtain the preset simulator.

[0086] Among them, the traffic generation process is the data exchange and exchange generated when the model is constantly optimizing parameters to more accurately predict the next word or character after a given context in the model training process; the collective communication process involves data transmission and synchronization between multiple processing units, usually used in parallel algorithms to achieve cooperation and information exchange between processes; the congestion control process is an important mechanism in computer networks to prevent too much data from being injected into the network, causing network performance to decline or even deadlock, when the network's resources (such as link bandwidth, node cache, etc.) are insufficient to handle the current data traffic, congestion occurs, in order to avoid this situation, the congestion control process will take appropriate measures to limit the sending rate of data, to prevent network overload.

[0087] Specifically, in the embodiments of the present application, by deeply analyzing various components that affect the training time of the target large language model, the traffic generation process, the collective communication process, the traffic sending process, the congestion control process, the traffic forwarding process and the traffic control process in the training process of the target large language model are modeled and simulated respectively to accurately simulate the network communication behavior in the training process of the large language model, thereby obtaining a traffic modeling module, a collective communication simulation module, a transmission protocol simulation module and a bottom network simulation module, as shown in FIG. 3, based on the preset ECS framework, the traffic modeling module, the collective communication simulation module, the transmission protocol simulation module and the bottom network simulation module are combined, thereby obtaining the preset simulator.

[0088] Further, as shown in FIG. 4, in order to realize the simulation of the target large language model training in the preset ECS framework based on the DOD idea, the present application first analyzes the network traffic life cycle in the target large language model training, and abstracts the entities, components and systems in the simulation of the target large language model training.

[0089] Among them, the preset simulator defines two entities, including: (1) host: the host represents a computing node with computing power in the intelligent computing center, such as a server, which contains CPU, GPU, AI accelerator card, etc. Computing power, responsible for running large language model training and other specific applications, these applications can generate network traffic and call communication modules for network transmission; (2) switch: the switch represents a network forwarding device in the intelligent computing center, used to forward the received network traffic to the corresponding port according to its destination. In addition, in a lossless network, the switch also needs to be responsible for traffic control to avoid packet loss caused by buffer overflow.

[0090] The preset emulator defines six systems related to network traffic, including: (1) a traffic generation system: used to generate traffic in the host according to the mode of the application, which includes the size, interval and timestamp of the traffic; (2) a collective communication system: used to simulate the algorithm used for parameter synchronization during large language model training, such as Ring All Reduce, PS (Parameter Server), etc., which can determine which hosts the network traffic generated by the application will be sent to; (3) a traffic sending system: after determining the destination of the network traffic, the traffic sending system can reliably transmit the network traffic to the destination, and according to the user's configuration, determine which transmission protocol to use, such as TCP (Transmission Control Protocol), RDMA (Remote Direct Memory Access (network communication protocol)); (4) a congestion control system: used to handle network congestion at the sending end, i.e., if congestion is observed in the network, the sending end needs to slow down for the network traffic, and according to the user's configuration, determine which congestion control algorithm to use, such as DCTCP (Data Center Transmission Control Protocol), DCQCN (Data Center Quantized Congestion Notification), HPCC (High Performance Congestion Control), etc.; (5) a traffic forwarding system: used for network traffic forwarding behavior in the switch, according to the destination of the network traffic packet, the packet is forwarded to other switches, and then by other switches continue to forward, until the packet reaches the destination host; (6) a traffic control system: used for flow control (Priority-based Flow Control, referred to as PFC) in lossless networks, if the buffer queue of a switch exceeds a certain threshold, the upstream switch needs to pause the forwarding of network traffic packets.

[0091] Next, how to model and simulate the traffic generation process, collective communication process, traffic sending process, congestion control process, traffic forwarding process and traffic control process in the training process of the target large language model, respectively, is described in detail.

[0092] (I) Model and simulate the traffic generation process to obtain a traffic modeling module.

[0093] As a possible implementation manner, in some embodiments, a traffic generation process in a training process of a target large language model is modeled and simulated, including: obtaining traffic data of the training process of the target large language model, generating a training set based on the traffic data; constructing an initial deep learning generative model, training the initial deep learning generative model based on a preset loss function and a preset optimization algorithm using the training set to obtain a final deep learning generative model; obtaining network structure information, parameter scale and communication mode of the target large language model, inputting the network structure information, parameter scale and communication mode into the final deep learning generative model, and outputting a predicted traffic mode, wherein the predicted traffic mode includes the size and interval of the traffic.

[0094] The embodiments of the present application collect traffic data during training of a target large language model, and design a deep learning generative model. The deep learning generative model can generate a corresponding training traffic mode according to given DNN model structure and distributed parallel, scheduling mode and other inputs, which includes the size and interval of the traffic, and provides an important traffic prediction and optimization means for training of the target large language model. The "traffic generation system" will call the model to generate traffic load simulating training of the large language model to be trained.

[0095] It can be understood that the design and training of the deep learning generative model is the core technology of this stage. The model is usually based on deep learning architectures such as recurrent neural network (RNN), variational auto encoder (VAE) or generative adversarial networks (GAN), and can generate traffic patterns consistent with actual conditions by learning the traffic data distribution in the training process of the large language model. In the design of the deep learning generative model, the characteristics of the target large language model training need to be considered, such as parallel computing, data transmission and communication in a distributed computing environment, etc. Therefore, the inputs of the deep learning generative model should include the network structure information, parameter scale and communication mode of the target large language model, etc. to ensure that the generated traffic pattern can fully reflect these characteristics. In addition, in the training process of the large language model, a suitable loss function and optimization algorithm need to be selected to fit the real traffic data distribution to the greatest extent. At the same time, in order to improve the generalization ability of the large language model, data enhancement techniques or regularization methods can also be used to prevent model overfitting.

[0096] Specifically, traffic data of a training process of a target large language model is acquired, including network traffic, data transmission rate and other information during training of the target large language model, and the traffic data can be used as a training set to train the deep learning generative model. An initial deep learning generative model is constructed, a preset loss function and a preset optimization algorithm are adopted, and the initial deep learning generative model is trained using the training set to fit the real traffic data distribution to the greatest extent, so as to ensure that the deep learning generative model can accurately capture the characteristics and rules of the training traffic, and obtain a final deep learning generative model. The preset loss function is a function used to measure the difference between the predicted results of the model and the real results in machine learning and deep learning, and the purpose is to optimize the parameters of the model so that the model can better fit the data; the optimization algorithm is a mathematical method used to find the optimal solution of a function, and common optimization algorithms include gradient descent method, Newton method, adaptive learning rate optimization algorithm, etc., which are not limited here. The final deep learning generative model after training can be used to predict the traffic pattern of the large language model training, and the network structure information, parameter size and communication mode of the target large language model are obtained. By inputting the network structure information, parameter size and communication mode into the final deep learning generative model, the predicted traffic pattern can be output, wherein the predicted traffic pattern includes the size and interval of the traffic. The predicted traffic pattern is applied to the training simulation of the to-be-trained large language model, which can be used for traffic generation and simulation.

[0097] (ii) modeling and simulating the collective communication process to obtain a collective communication simulation module.

[0098] As a possible implementation manner, in other embodiments, the collective communication process in the training process of the target large language model is modeled and simulated, including: decomposing the collective communication process into at least one communication event, establishing a discrete communication event model based on the at least one communication event; using a preset event-driven strategy, triggering at least one communication event to perform a corresponding communication operation according to a first preset trigger condition.

[0099] It can be understood that the collective communication process is an important process involving the exchange and synchronization of data between multiple nodes. The collective communication process is divided into at least one communication event, including message sending, message receiving, routing selection and other processes. Based on at least one communication event, a discrete communication event model is established, so as to abstract the collective communication process into a discrete communication event model. Then, a preset event-driven strategy is used to describe the collective communication algorithm, such as Ring All Reduce, Parameter Server, RHD (Recursive Halving and Doubling, recursive halving and doubling), etc. That is, data communication is no longer performed according to fixed time intervals or fixed steps, but is triggered according to actual needs or certain conditions (first preset trigger condition). Each communication event corresponds to a specific communication operation or behavior. The actual communication process is simulated through the occurrence and processing of communication events, so as to better reflect the timing and concurrency of communication operations. At the same time, various communication scheduling mechanisms such as FIFO (First in First out, first in first out), LIFO (Last in First out, last in first out) are implemented. Different scheduling strategies are used to manage and optimize inter-node communication, thereby improving communication efficiency, reducing communication delay, and maximizing system resources.

[0100] As can be seen, by establishing a discrete communication event model and combining the data-oriented design idea, fine modeling and simulation of communication behavior can be realized, and the association and influence between different communication operations can be accurately described.

[0101] (Three) Model and simulate the traffic sending process and the congestion control process to obtain a transmission protocol simulation module, that is, the "traffic sending system" and the "congestion control system" constitute the transmission protocol simulation module.

[0102] As a possible implementation manner, in other embodiments, the traffic sending process and the congestion control process in the training process of the target large language model are modeled and simulated, including: constructing a first network node entity and a first communication component by using a preset ECS framework, wherein each first network node entity includes at least one first communication component; based on a preset event-driven strategy, each network transmission operation is defined as a corresponding at least one transmission event, and based on a preset event scheduler, the execution order and timing relationship of the at least one transmission event are managed; according to the first network node entity and the first communication component, it is judged whether the trigger condition of the at least one transmission event satisfies a second preset trigger condition, and when the trigger condition of the at least one transmission event satisfies the second preset trigger condition, the at least one transmission event is triggered to perform a corresponding transmission operation.

[0103] The network node entity refers to a general term for various devices (including node devices, communication devices, terminal devices, storage devices, power systems, etc.) and other hardware devices serving the same in a computer network.

[0104] It can be understood that the network transmission protocol plays a crucial role in the distributed training of large language models, which determines the transmission mode and efficiency of data in the network, and therefore, the modeling and simulation of the network transmission protocol are of great significance. In combination with data-oriented design, the data-oriented design emphasizes starting from the perspective of data, optimizing algorithms and structures to improve computing efficiency.

[0105] Specifically, a first network node entity and a first communication component are constructed using a preset ECS framework, wherein each first network node entity includes a plurality of first communication components, the first network node entity includes a transmission control block (TCB), a data buffer, etc., each first communication component defines the state and behavior of the first network node entity, such as connection management, data transmission, etc., and the preset ECS framework is a high-performance and highly scalable game development framework, whose component-based design philosophy is consistent with the data-oriented design philosophy; based on a preset event-driven strategy, the traffic sending process and the congestion control process are simulated, each network transmission operation is defined as at least one corresponding transmission event (i.e., each transmission event represents a network transmission operation), such as connection establishment, data sending, data receiving, etc., the execution order and timing relationship of at least one transmission event are managed by a preset event scheduler to ensure the accuracy and reliability of the simulation; according to the first network node entity and the first communication component, the data processing and transmission process is simulated in each transmission event, i.e., according to the state information of the transmission control block, it is determined whether the triggering condition of at least one transmission event (such as data sending and receiving) meets a second preset triggering condition, and when the triggering condition of at least one transmission event meets the second preset triggering condition, at least one transmission event is triggered to perform a corresponding transmission operation, which includes the simulation of the congestion control algorithm.

[0106] (Four) The traffic forwarding process and the traffic control process are modeled and simulated to obtain the bottom-layer network simulation module, i.e., the "traffic forwarding system" and the "traffic control system" constitute the bottom-layer network simulation module.

[0107] As a possible implementation, in some embodiments, the traffic forwarding process and the traffic control process in the training process of the target large language model are modeled and simulated, including: constructing second network node entities and second communication components using a preset ECS framework, wherein each second network node entity includes at least one second communication component; defining each network transmission operation as a corresponding at least one transmission event based on a preset event-driven strategy, and managing the execution order and timing relationship of the at least one transmission event based on a preset event scheduler; determining whether the trigger condition of the at least one transmission event satisfies a third preset trigger condition according to the second network node entity and the second communication component, and triggering the at least one transmission event to perform a corresponding transmission operation when the trigger condition of the at least one transmission event satisfies the third preset trigger condition.

[0108] It can be understood that the underlying network includes various network topologies and Ethernet protocols, such as lossy networks and lossless networks, where lossless networks are a special network environment commonly used in scenarios where packet loss is not allowed during data transmission, especially when performing high-performance computing within a data center. To ensure the reliability of data transmission, the PFC (Priority-based Flow Control) protocol is usually used for traffic control, and in this case, modeling and simulating lossless networks is of great significance, which can be used to evaluate network performance and optimize communication efficiency.

[0109] Specifically, second network node entities and second communication components are constructed using a preset ECS framework, wherein each second network node entity includes at least one second communication component, the second network node entity includes a PFC controller and a data buffer, etc., the second communication component defines the state and behavior of the second network node entity, including traffic control and data transmission functions of the PFC protocol; the traffic forwarding process and the traffic control process are simulated based on a preset event-driven strategy, each network transmission operation is defined as a corresponding at least one transmission event (i.e., each transmission event represents a network transmission operation), including data sending and receiving, and the execution order and timing relationship of the at least one transmission event are managed by a preset event scheduler to ensure the accuracy and reliability of the simulation; according to the second network node entity and the second communication component, the data processing and transmission process is simulated in each transmission event, i.e., according to the state information of the PFC controller, it is determined whether the trigger condition of the at least one transmission event satisfies a third preset trigger condition, and when the trigger condition of the at least one transmission event satisfies the third preset trigger condition, the at least one transmission event is triggered to perform a corresponding transmission operation, while considering the priority control mechanism of the PFC protocol to realize the data transmission process simulation.

[0110] In step S103, the intelligent computing center network architecture is adjusted according to the simulation result, and an optimal intelligent computing center network architecture is obtained.

[0111] That is, after the simulation result is obtained by training and simulating the large language model to be trained by using the preset simulator in step S102, the current intelligent computing center network architecture can be adjusted according to the simulation result, so as to determine the optimal intelligent computing center network architecture.

[0112] Therefore, based on the program design idea of data-oriented design, the simulation optimization method of the intelligent computing center network architecture is proposed. The preset simulator includes modules such as large language model training flow modeling, set communication simulation, transmission protocol simulation, and bottom network simulation. Based on the data-oriented design idea, the preset simulator abstractly models each process of the typical application (such as AI large model training) in the intelligent computing center network, and can accurately simulate the time and power consumption required for training a large language model under different network architectures. In addition, the preset simulator can also perform automatic parallelism in a single machine, in which a hybrid parallel and serial computing mode is applied, thereby improving the speed and scalability of simulating a large-scale intelligent computing center network.

[0113] According to the simulation optimization method of the intelligent computing center network architecture proposed in the embodiments of the present application, by obtaining the training configuration information of the large language model to be trained, the training configuration information and the preset discrete event simulation strategy can be used to train and simulate the large language model to be trained by using the preset simulator, so as to obtain the simulation result. The preset simulator is obtained by using the preset ECS framework, and the intelligent computing center network architecture is adjusted according to the simulation result, so as to obtain the optimal intelligent computing center network architecture. Therefore, by designing the intelligent computing center network simulator based on the entity component system framework, the complete simulation process of the large language model training can be realized to find the optimal intelligent computing center network architecture, thereby solving the problem that the existing network simulator does not have scalability and cannot support simulation of a large-scale intelligent computing center network, and improving the efficiency and accuracy of simulating a large-scale intelligent computing center network.

[0114] Secondly, the simulation optimization device of the intelligent computing center network architecture according to the embodiments of the present application is described with reference to the accompanying drawings.

[0115] FIG. 5 is a block schematic diagram of the simulation optimization device of the intelligent computing center network architecture according to an embodiment of the present application.

[0116] As shown in FIG. 5, the simulation optimization device 10 of the intelligent computing center network architecture includes an obtaining module 100, a training module 200, and a processing module 300.

[0117] The obtaining module 100 is configured to obtain the training configuration information of the large language model to be trained.

[0118] The training module 200 is configured to perform training simulation on the large language model to be trained by using a preset simulator based on the training configuration information and a preset discrete event simulation strategy, and obtain a simulation result, wherein the preset simulator is obtained by using a preset ECS framework.

[0119] The processing module 300 is configured to adjust the network architecture of the intelligent computing center according to the simulation result, and obtain an optimal network architecture of the intelligent computing center.

[0120] Further, in some embodiments, before performing the training simulation on the large language model to be trained by using the preset simulator based on the training configuration information and the preset discrete event simulation strategy, the training module 200 further comprises:

[0121] The modeling and simulation unit is configured to model and simulate a traffic generation process, a set communication process, a traffic sending process, a congestion control process, a traffic forwarding process and a traffic control process in the training process of the target large language model respectively, and obtain a traffic modeling module, a set communication simulation module, a transmission protocol simulation module and a bottom-layer network simulation module.

[0122] The combination unit is configured to combine the traffic modeling module, the set communication simulation module, the transmission protocol simulation module and the bottom-layer network simulation module based on the preset ECS framework, and obtain the preset simulator.

[0123] Further, in some embodiments, the modeling and simulation unit is specifically configured to:

[0124] Obtain traffic data of the training process of the target large language model, and generate a training set based on the traffic data;

[0125] Construct an initial deep learning generative model, train the initial deep learning generative model by using the training set based on a preset loss function and a preset optimization algorithm, and obtain a final deep learning generative model;

[0126] Obtain network structure information, a parameter size and a communication mode of the target large language model, input the network structure information, the parameter size and the communication mode into the final deep learning generative model, and output a predicted traffic mode, wherein the predicted traffic mode includes a size and an interval of traffic.

[0127] Further, in some embodiments, the modeling and simulation unit is specifically configured to:

[0128] Decompose the set communication process into at least one communication event, and establish a discrete communication event model based on the at least one communication event;

[0129] Trigger the at least one communication event to perform a corresponding communication operation according to a first preset trigger condition by using a preset event-driven strategy.

[0130] Further, in some embodiments, the modeling simulation unit is specifically configured to:

[0131] constructing first network node entities and first communication components by using a preset ECS framework, wherein each first network node entity comprises at least one first communication component;

[0132] defining each network transmission operation as a corresponding at least one transmission event based on a preset event-driven strategy, and managing execution order and timing relationship of the at least one transmission event based on a preset event scheduler;

[0133] judging whether a trigger condition of the at least one transmission event meets a second preset trigger condition according to the first network node entities and the first communication components, and triggering the at least one transmission event to execute a corresponding transmission operation when the trigger condition of the at least one transmission event meets the second preset trigger condition.

[0134] Further, in some embodiments, the modeling simulation unit is specifically configured to:

[0135] constructing second network node entities and second communication components by using a preset ECS framework, wherein each second network node entity comprises at least one second communication component;

[0136] defining each network transmission operation as a corresponding at least one transmission event based on a preset event-driven strategy, and managing execution order and timing relationship of the at least one transmission event based on a preset event scheduler;

[0137] judging whether a trigger condition of the at least one transmission event meets a third preset trigger condition according to the second network node entities and the second communication components, and triggering the at least one transmission event to execute a corresponding transmission operation when the trigger condition of the at least one transmission event meets the third preset trigger condition.

[0138] It should be noted that the above description of the simulation optimization method for the network architecture of the intelligent computing center also applies to the simulation optimization device for the network architecture of the intelligent computing center, which will not be described here.

[0139] According to the simulation optimization device of the intelligent computing center network architecture provided in the embodiments of the present application, by obtaining the training configuration information of the to-be-trained large language model, the training configuration information and the preset discrete event simulation strategy can be used to train and simulate the to-be-trained large language model by using the preset simulator, so as to obtain a simulation result. The preset simulator is obtained by using a preset ECS framework, and the intelligent computing center network architecture is adjusted according to the simulation result, so as to obtain an optimal intelligent computing center network architecture. Therefore, by using the entity component system framework to design the intelligent computing center network simulator, the complete simulation process of the large language model training can be realized to find the optimal intelligent computing center network architecture, and the problem that the existing network simulator does not have scalability and cannot support the simulation of large-scale intelligent computing center networks is solved, thereby improving the efficiency and accuracy of simulating large-scale intelligent computing center networks.

[0140] FIG. 6 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. The electronic device can include:

[0141] The memory 601, the processor 602, and the computer program stored in the memory 601 and executable on the processor 602.

[0142] The processor 602 implements the simulation optimization method of the intelligent computing center network architecture provided in the above embodiments when executing the program.

[0143] Further, the electronic device further includes:

[0144] The communication interface 603 is used for communication between the memory 601 and the processor 602.

[0145] The memory 601 is used to store the computer program executable on the processor 602.

[0146] The memory 601 can include a high-speed RAM (Random Access Memory, Random Access Memory) memory, and can also include a non-volatile memory, such as at least one disk memory.

[0147] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected with each other through a bus and complete communication between each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in FIG. 6, but it does not mean that there is only one bus or only one type of bus.

[0148] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete communication between each other through an internal interface.

[0149] The processor 602 can be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0150] The embodiments of the present application also provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the simulation optimization method of the intelligent computing center network architecture as above.

[0151] The embodiments of the present application also provide a computer program product, comprising a computer program, which, when executed by a processor, implements the simulation optimization method of the intelligent computing center network architecture as above.

[0152] In addition, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0153] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are contained in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0154] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present application, and the person skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A simulation optimization method of a smart center network architecture, characterized in that, The method comprises the following steps: obtaining training configuration information of a large language model to be trained; based on the training configuration information and a preset discrete event simulation strategy, training simulation is performed on the large language model to be trained by using a preset simulator, and a simulation result is obtained, wherein the preset simulator is obtained by using a preset entity component system (ECS) framework; adjusting the network architecture of the intelligent computing center according to the simulation result, and obtaining an optimal network architecture of the intelligent computing center. 2.The simulation and optimization method of the intelligent algorithm center network architecture according to claim 1, wherein, Before the training simulation is performed on the large language model to be trained by using the preset simulator based on the training configuration information and the preset discrete event simulation strategy, the method further comprises the following steps: modeling and simulating a traffic generation process, a set communication process, a traffic sending process, a congestion control process, a traffic forwarding process and a traffic control process in the training process of the target large language model, respectively, to obtain a traffic modeling module, a set communication simulation module, a transmission protocol simulation module and a bottom layer network simulation module; based on the preset ECS framework, the traffic modeling module, the set communication simulation module, the transmission protocol simulation module and the bottom layer network simulation module are combined to obtain a preset simulator. 3.The simulation and optimization method of the intelligent algorithm center network architecture according to claim 2, characterized in that, The method of modeling and simulating the traffic generation process in the training process of the target large language model comprises the following steps: obtaining traffic data of the training process of the target large language model, and generating a training set based on the traffic data; constructing an initial deep learning generative model, training the initial deep learning generative model based on a preset loss function and a preset optimization algorithm by using the training set, and obtaining a final deep learning generative model; obtaining network structure information, parameter size and communication mode of the target large language model, inputting the network structure information, the parameter size and the communication mode into the final deep learning generative model, and outputting a predicted traffic mode, wherein the predicted traffic mode includes the size and interval of traffic. 4.The simulation and optimization method of the intelligent algorithm center network architecture according to claim 3, characterized in that, The method of modeling and simulating the set communication process in the training process of the target large language model comprises the following steps: decomposing the set communication process into at least one communication event, and establishing a discrete communication event model based on the at least one communication event; triggering the at least one communication event to perform corresponding communication operations according to a first preset trigger condition by using a preset event-driven strategy.

5. The simulation optimization method of the intelligent center network architecture according to claim 4, characterized in that, The method of modeling and simulating the traffic sending process and the congestion control process in the training process of the target large language model comprises the following steps: constructing a first network node entity and a first communication component by using the preset ECS framework, wherein each first network node entity comprises at least one first communication component; defining each network transmission operation as a corresponding at least one transmission event based on the preset event-driven strategy, and managing the execution order and timing relationship of the at least one transmission event based on a preset event scheduler. According to the first network node entity and the first communication component, it is judged whether a trigger condition of the at least one transmission event meets a second preset trigger condition, and when the trigger condition of the at least one transmission event meets the second preset trigger condition, the at least one transmission event is triggered to perform a corresponding transmission operation. 6.The simulation and optimization method of the intelligent algorithm center network architecture according to claim 5, characterized in that, The traffic forwarding process and the traffic control process in the training process of the target large language model are modeled and simulated, including: A second network node entity and a second communication component are constructed by using the preset ECS framework, wherein each second network node entity includes at least one second communication component; Based on the preset event-driven strategy, each network transmission operation is defined as a corresponding at least one transmission event, and based on a preset event scheduler, the execution order and the timing relationship of the at least one transmission event are managed; According to the second network node entity and the second communication component, it is judged whether a trigger condition of the at least one transmission event meets a third preset trigger condition, and when the trigger condition of the at least one transmission event meets the third preset trigger condition, the at least one transmission event is triggered to perform a corresponding transmission operation.

7. A simulation optimization device for an intelligent computing center network architecture, characterized in that: It includes: An acquisition module is configured to acquire training configuration information of a large language model to be trained; A training module is configured to train and simulate the large language model to be trained by using a preset simulator based on the training configuration information and a preset discrete event simulation strategy, to obtain a simulation result, wherein the preset simulator is obtained by using a preset entity component system (ECS) framework; A processing module is configured to adjust a network architecture of an intelligent computing center according to the simulation result, and obtain an optimal network architecture of the intelligent computing center.

8. An electronic device, comprising: It includes: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the simulation optimization method of the network architecture of the intelligent computing center according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the simulation optimization method of the network architecture of the intelligent computing center according to any one of claims 1-6.

10. A computer program product, characterised in that, It includes a computer program, which, when executed by a processor, is used to implement the simulation optimization method of the network architecture of the intelligent computing center according to any one of claims 1-6.

Images