Integrated computing power cluster scheduling management system based on optical interconnection

Abstract

The invention relates to the technical field of artificial intelligence model training, in particular to an integrated computing power cluster scheduling management system based on optical interconnection, which comprises a management platform and a plurality of computing power servers, the computing power servers are connected through the optical internet, and the management platform and the computing power servers are connected through the optical internet. The management platform collects the load condition on the computing power server through the optical internet, and distributes a model training task to the computing power server through the optical internet according to the load condition. In order to solve the problem that the model training efficiency of the computing power cluster in the prior art is restricted by the network transmission rate, in the scheme, the optical internet is introduced to connect the computing power servers and the management platform, the data synchronization between the computing power servers and the training data issuing efficiency of the management platform are improved through the optical internet, and the training efficiency of the computing power cluster is improved. And the model training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence model training technology, and in particular to an integrated computing power cluster scheduling and management system based on optical interconnection. Background Art

[0002] A computing cluster is a computer system that connects multiple servers through a network to complete computing tasks together. This system is usually composed of multiple computing nodes, each of which contains a high-performance processor, a graphics processor, a large-capacity memory, and a high-speed storage device. The nodes are interconnected through a high-speed network to ensure efficient data transmission between nodes. The computing cluster also includes a cluster management system, which is responsible for monitoring node status, resource allocation and scheduling, load balancing, and fault recovery to ensure that tasks can run efficiently on available nodes. Because computing clusters can achieve higher overall computing power than traditional architectures, they are widely used in the training process of artificial intelligence models.

[0003] In the prior art, there are artificial intelligence model training methods based on computing power clusters and distributed training. For example, Chinese patent CN201910755676.4 discloses a model training method, including: obtaining multiple groups of batch data for model training; training the same neural network language model in parallel based on different batch data through multiple computing nodes running on multiple graphics processors; determining the gradient of each parameter through each of the multiple computing nodes in the current round of iteration; summarizing the gradient of each parameter to obtain the global gradient of each parameter, and controlling multiple computing nodes to synchronously update the parameters based on the global gradient; evaluating the performance of the neural network language model updated in the current round based on the test data set by the first computing node to obtain the current language model performance index value; broadcasting the current language model performance index value through the first computing node; and controlling whether multiple computing nodes stop training based on the relationship between the current language model performance index value and the language model performance index value determined in the previous round.

[0004] However, during the actual implementation process, the inventors found that since this type of technical solution still uses a three-layer network design based on electrical interconnection technology, the overall computing efficiency of the computing cluster will be restricted by the transmission network, especially the synchronization of training set data and model parameter changes during the distributed training process of the model, which will be seriously affected by the network transmission rate. Summary of the invention

[0005] In view of the above-mentioned problems existing in the prior art, an integrated computing power cluster scheduling and management system based on optical interconnection is now provided.

[0006] The specific technical solutions are as follows:

[0007] An integrated computing power cluster scheduling and management system based on optical interconnection, including a management platform and multiple computing power servers;

[0008] The computing power servers and the management platform and the computing power servers are connected via an optical internet network;

[0009] The management platform collects the load status on the computing power server through the optical Internet network, and the management platform distributes model training tasks to the computing power server via the optical Internet network according to the load status.

[0010] On the other hand, the optical internet network includes a plurality of regional gateways, and a plurality of computing power servers are respectively arranged under each of the regional gateways;

[0011] The management platform includes a routing allocation module, and the routing allocation module includes:

[0012] A topology acquisition module, wherein the topology acquisition module collects topology information between a plurality of the regional gateways and the computing power server from the optical interconnection network;

[0013] A convergence ratio determination module, the convergence ratio determination module is connected to the topology acquisition module, and the convergence ratio determination module determines the convergence ratio of switches under each group of regional gateways;

[0014] A training unit scheduling module, wherein the training unit scheduling module is connected to the convergence ratio determination module, and the training unit scheduling module splits the training unit of the model training task and allocates a plurality of training containers in the training unit to the computing power server according to the switch convergence ratio;

[0015] The training unit scheduling module preferentially allocates the training containers of the same training unit to the computing power servers under the same group of regional gateways when the convergence ratio of the switch meets the requirement.

[0016] On the other hand, the management platform includes:

[0017] A queue generation module, wherein the queue generation module receives the training task input from the outside and adds it to the training queue;

[0018] A load collection module, wherein the load collection module obtains load information on a plurality of computing power servers;

[0019] A task scheduling module, wherein the task scheduling module is connected to the queue generation module and the load acquisition module respectively, and the task scheduling module extracts the training tasks from the training queue according to the load information and distributes them to the corresponding computing power servers.

[0020] On the other hand, the task scheduling module includes:

[0021] A container number determination module, wherein the container number determination module splits the training task to determine a minimum number of training containers;

[0022] A load determination module, wherein the load determination module determines the number of idle containers that can be deployed by the computing power server under the current load according to the load information;

[0023] An allocation module, wherein the allocation module is connected to the container number determination module and the load determination module respectively, and the allocation module allocates the training task to the computing power server when the number of free containers meets the minimum number of training containers.

[0024] On the other hand, the task scheduling module also includes:

[0025] A pre-screening module, the pre-screening module is connected to the allocation module, the pre-screening module collects node heterogeneous resource information from the computing power server, and the pre-screening module pre-marks the computing power server that can be allocated for the training task in the training queue according to the node heterogeneous resource information to generate a resource classification attribute;

[0026] The allocation module allocates the training tasks according to the resource classification attributes.

[0027] On the other hand, the task scheduling module also includes:

[0028] A centralized scheduling module, wherein the centralized scheduling module is connected to the allocation module, wherein the centralized scheduling module reads the task type of the training task, and the centralized scheduling module centrally allocates the training task whose task type is marked as a GPU training task.

[0029] On the other hand, the management platform also includes:

[0030] A load balancing module, wherein the load balancing module is connected to the load acquisition module, and the load balancing module performs load balancing on the computing power server according to the load information.

[0031] On the other hand, the computing power server includes:

[0032] A cache module, which receives and caches the training task transmitted from the optical internet network;

[0033] A cache reading module, wherein the cache reading module is connected to the cache module and the processing module, and the cache reading module directly reads the training task data of the training task from the cache module and transmits it to the processing module.

[0034] On the other hand, the management platform also includes:

[0035] A front-end page generation module, wherein the front-end page generation module creates a front-end page through which the user uploads the training task.

[0036] The above technical solution has the following advantages or beneficial effects:

[0037] In order to solve the problem that the model training efficiency of the computing power cluster in the existing technology is restricted by the network transmission rate, this solution introduces an optical Internet network to connect the computing power server and the management platform. The optical Internet network improves the data synchronization between the computing power servers and the efficiency of the management platform in issuing training data, thereby improving the model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.

[0039] Figure 1 It is an overall schematic diagram of an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a routing allocation module in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of a management platform in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of a task scheduling module in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of a pre-screening module in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of a centralized scheduling module in an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of a load balancing module in an embodiment of the present invention;

[0046] Figure 8 This is a schematic diagram of a computing power server in an embodiment of the present invention;

[0047] Fig. 9 Schematic diagram of a front-end page generation module in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0050] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0051] The present invention comprises:

[0052] An integrated computing cluster scheduling and management system based on optical interconnection, including a management platform A and multiple computing servers B;

[0053] The computing servers B and the management platform A and computing servers B are connected via an optical internet network;

[0054] Management platform A collects the load status on computing server B through the optical Internet network, and distributes model training tasks to computing server B through the optical Internet network according to the load status.

[0055] Specifically, in order to address the problem in the prior art that the model training efficiency of the computing power cluster is restricted by the network transmission rate, in this embodiment, an optical Internet network is introduced to connect the computing power server B and the management platform A. The optical Internet network improves the data synchronization between the computing power servers B and the efficiency of the management platform A in issuing training data, thereby improving the model training efficiency.

[0056] Specifically, taking a distributed model training task as an example, the training task of this type of model usually includes the following steps:

[0057] Users upload the built model to the management platform and select a public training set or upload their own training set;

[0058] The management platform builds distributed training tasks corresponding to the model and allocates corresponding computing nodes;

[0059] The management platform transfers the model and the corresponding training set to the computing node, configures the training parameters and starts training;

[0060] During the training process, for some models with self-learning characteristics, there may be certain parameter transfer and data exchange processes between multiple computing nodes;

[0061] When the preset number of training times is reached, the management platform evaluates the model, optimizes the parameters, and synchronizes them to each computing node, and then continues training;

[0062] When the iteration conditions are met, stop training and output the model.

[0063] In the above process, by replacing the data transmission method with an optical Internet network, it helps to increase the transmission speed for training sets with large amounts of data, and improve the efficiency of data synchronization between multiple computing nodes.

[0064] During the implementation process, the management platform A is embodied as a cloud server or cloud platform configured with a specific computer program, which receives training models, training parameters, training sets and other information input by users through front-end web pages, clients, etc. The computing power server B is a computer device that is configured with a corresponding computer program and can be used for model training. Compared with traditional computer devices, it may have better CPU resources, GPU resources, etc., and can perform specific computing tasks under the control of the management platform A. The optical interconnection network is a local area network system built on an optical switch, which mainly includes an optical switch / optical router or other equivalent equipment, which is connected to the optical port of the management platform A and the computing power server B through optical fiber, and can achieve a large amount of data throughput and improve computing efficiency.

[0065] In one embodiment, the optical internet network includes a plurality of regional gateways, and a plurality of computing servers are respectively arranged under each regional gateway;

[0066] The management platform includes a routing distribution module 1, and the routing distribution module 1 includes:

[0067] A topology acquisition module 11, which collects topology information between multiple regional gateways and computing servers from the optical interconnection network;

[0068] A convergence ratio determination module 12, the convergence ratio determination module 12 is connected to the topology acquisition module 11, and the convergence ratio determination module 12 determines the convergence ratio of switches under each group of regional gateways;

[0069] The training unit scheduling module 13 is connected to the convergence ratio determination module 12. The training unit scheduling module 13 splits the training units of the model training task and distributes multiple training containers in the training units to the computing power server according to the switch convergence ratio.

[0070] The training unit scheduling module 13 preferentially allocates the training containers of the same training unit to the computing power servers under the same group of regional gateways when the switch convergence ratio meets the requirements.

[0071] Specifically, in order to solve the problem in the prior art that the efficiency of distributed training is restricted by the network transmission rate, in this embodiment, the transmission efficiency is improved by reducing cross-switch communication.

[0072] Specifically, the optical Internet network includes multiple regional gateways, each of which is equipped with multiple computing servers, and the computing servers under each regional gateway are configured in different network segments.

[0073] Before starting training, the topology acquisition module 11 collects topology information between multiple regional gateways and computing power servers from the optical Internet network, that is, which computing power servers are included under each gateway.

[0074] At the same time, during the model training process, there may be multiple models that need to be trained in parallel. In order to achieve better training efficiency, on the computing server side, training tasks are executed by creating containers, that is, each model training task will create a container in the computing server, and each computing server will create multiple containers corresponding to training tasks of different models, and containers are used to isolate computing resources and memory data.

[0075] After determining the topology information, the convergence ratio determination module 12 calculates the convergence ratio of the switch under each group of regional gateways based on the hardware resources of the switch itself, the number of optical ports, the optical port throughput of the computing server and other information contained in the topology information.

[0076] As an optional implementation, during calculation, the real-time convergence ratio obtained by subtracting the current load of the regional gateway from the convergence ratio of the switch may be used as a judgment condition.

[0077] Then, the training unit scheduling module 13 splits the training units of the model training task. For the training units in a single training node, the training unit scheduling module 13 first estimates the amount of cross-node data exchange that may be generated by the training unit, and then compares it with the switch convergence ratio to determine the regional gateway that can meet the cross-node exchange, and then creates a container on multiple computing servers under the regional gateway and specifies the training for the training unit.

[0078] During the actual training process, the training unit will only exchange data under the gateway of this area, and there will be no delay problem caused by cross-gateway transmission.

[0079] In one embodiment, the management platform A includes:

[0080] Queue generation module 2, which receives external input training tasks and adds them to the training queue;

[0081] Load collection module 3, load collection module 3 obtains load information on multiple computing power servers;

[0082] The task scheduling module 4 is connected to the queue generation module 2 and the load collection module 3 respectively. The task scheduling module 4 extracts training tasks from the training queue according to the load information and distributes them to the corresponding computing power servers.

[0083] Specifically, in order to achieve a better scheduling effect, in this embodiment, a queue generation module 2 is first created in the management platform. The queue generation module 2 receives external input training tasks and adds them to the training queue, and obtains the load information on multiple computing servers based on the heartbeat message through the load collection module 3. Then, the task scheduling module 4 makes a real-time judgment based on the load information to determine whether there is a computing server that meets the computing requirements. If so, the training task is extracted from the training queue and allocated to the corresponding computing server.

[0084] In one embodiment, the task scheduling module 4 includes:

[0085] A container number determination module 41, which splits the training task to determine the minimum number of training containers;

[0086] A load determination module 42, which determines the number of idle containers that can be deployed on the computing server under the current load according to the load information;

[0087] The allocation module 43 is connected to the container number determination module 41 and the load determination module 42 respectively. The allocation module 43 allocates the training task to the computing power server when the number of free containers meets the minimum number of training containers.

[0088] Specifically, in order to achieve a better allocation effect, in this embodiment, for non-distributed training tasks, the training task is first split through the container number determination module 41 to determine the minimum number of training containers to control the minimum resources occupied by the model during training. Then the load discrimination module 42 determines the number of idle containers that can be deployed by the computing server under the current load according to the load information, and finally the allocation module 43 allocates the training task to multiple computing servers when the number of idle containers meets the minimum number of training containers. The above settings can reduce the number of containers created during model training, and concentrate the training process as much as possible in a small number of computing servers, thereby reducing resource usage.

[0089] In one embodiment, the task scheduling module 4 further includes:

[0090] A pre-screening module 44, the pre-screening module 44 is connected to the allocation module 43, the pre-screening module 44 collects node heterogeneous resource information from the computing power server, and the pre-screening module 44 pre-marks the computing power server that can be allocated for the training tasks in the training queue according to the node heterogeneous resource information to generate resource classification attributes;

[0091] The allocation module 43 allocates training tasks according to resource classification attributes.

[0092] Specifically, in view of the fact that the actual computing power server may be a node heterogeneous resource with different platforms and configurations, in this embodiment, a pre-screening module 44 is also provided. Before starting to allocate training tasks, the pre-screening module 44 will first collect node heterogeneous resource information, that is, the platform architecture, hardware parameters, NVLink, PcleSwitch and other communication methods between GPU cards, NUMA topology structure of CPU and other information of each computing power server, so as to determine the computing functions that can be called by each computing power server, the computing tasks to be executed, etc.

[0093] When a training task is added to the training queue, the pre-screening module 44 pre-marks the computing servers that can be allocated to the training tasks in the training queue according to the node heterogeneous resource information to generate resource classification attributes, thereby determining the computing servers that can be used to execute the training task. During actual allocation, the allocation module 43 allocates the training task according to the resource classification attributes.

[0094] In one embodiment, the task scheduling module 4 further includes:

[0095] The centralized scheduling module 45 is connected to the allocation module 43. The centralized scheduling module 45 reads the task type of the training task, and the centralized scheduling module 45 centrally allocates the training tasks whose task types are marked as GPU training tasks.

[0096] Specifically, in order to reduce the waste of GPU resources, in this embodiment, a centralized scheduling module 45 is also configured in the task scheduling module 4. When the allocation module 43 reads the training task from the training queue, the centralized scheduling module 45 identifies the task type of the training task, and when it is determined to be a GPU training task that mainly requires GPU training resources, the GPU training task is cached until the relevant conditions are met, such as the existence of multiple GPU training tasks and the total GPU resources on the computing power server side are idle to a certain threshold before being released, thereby improving the scheduling efficiency.

[0097] At the same time, a parallel computing plug-in is also configured on the computing server side to perform parallel computing on the GPU resources when executing such GPU training tasks to improve processing efficiency.

[0098] In one embodiment, the management platform further includes:

[0099] The load balancing module 5 is connected to the load collection module 3, and the load balancing module 5 performs load balancing on the computing power servers according to the load information.

[0100] Specifically, in order to achieve better resource scheduling, a load balancing module 5 is also set in the management platform in this embodiment. The load balancing module 5 will identify the computing power server based on the load information. The identification process mainly includes two parts. The first part is performed for each computing power server. When the load of a computing power server triggers the load threshold, the container on the computing power server is selectively closed and transferred to other computing power servers with lower loads to avoid overload. The second part is to measure the load balancing situation of the computing power server as a whole, which can be done in the form of standard deviation. When the standard deviation is too large, it indicates that there is a load imbalance, and the tasks running on the computing power server are reallocated.

[0101] In one embodiment, the computing server includes:

[0102] Cache module B1, which receives and caches training tasks transmitted from the optical interconnection network;

[0103] The cache reading module B2 connects the cache module B1 and the processing module B3. The cache reading module B2 directly reads the training task data of the training task from the cache module B1 and transmits it to the processing module B3.

[0104] Specifically, in order to achieve better training efficiency, in this embodiment, a method of establishing a node local cache on the computing power server side is chosen to improve efficiency.

[0105] Specifically, on the computing server side, a local cache module B1 is established. The cache module B1 receives and caches the training tasks transmitted from the optical Internet. Its physical carrier is a high-performance local storage disk. The platform can cache the user's original data set directly into the high-performance storage of each computing node. The training task reads data from the cache module B1 through the cache reading module B2 according to the address information, and then transmits it to the processing module B3 for calculation.

[0106] As an optional implementation, the computing server can also mount an external parallel storage system, and directly read the training data set from the storage without passing through any local disk, and directly use it for training tasks. For example, if the customer uses a high-performance parallel storage Beegfs system, all data sets can be placed on the Beegfs system, and all cluster computing nodes are installed with the Beegfs client. In this way, the training task will automatically pull the data set directly from the storage.

[0107] In one embodiment, the management platform further includes:

[0108] The front-end page generation module 6 creates a front-end page through which the user uploads the training task.

[0109] Specifically, in order to achieve a better user experience, the front-end page is also created by the front-end page generation module 6 in this embodiment.

[0110] The user configures the relevant parameters through the UI, then submits an application, performs unified configuration of distributed training through kubeflow, and then starts the container task according to the corresponding configuration through k8s and docker, and then the training task starts. The input and output of the training task are synchronized with the data storage, and the user can view the progress of the task in real time. The visualization function will also display the progress of the task, monitor the resource utilization of the task in real time, and so on. When the user applies for a training task, the system will present a detailed configuration list, and then start the corresponding container to load the image according to the configuration list. The training task is then started based on the startup file and input data. The log output file during the operation will be saved to the storage space (personal and temporary storage space), and will be visualized as required. The output of the training task will be saved to the corresponding personal storage space for subsequent visualization and calling.

[0111] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. An integrated computing cluster scheduling and management system based on optical interconnection, characterized in that: Including management platform and multiple computing servers; The computing power servers and the management platform and the computing power servers are connected via an optical internet network; The management platform collects the load status on the computing power server through the optical Internet network, and the management platform distributes model training tasks to the computing power server via the optical Internet network according to the load status.

2. The integrated computing cluster scheduling and management system for optical interconnection according to claim 1 is characterized in that: The optical internet network includes a plurality of regional gateways, and each of the regional gateways is respectively provided with a plurality of computing power servers; The management platform includes a routing allocation module, and the routing allocation module includes: A topology acquisition module, wherein the topology acquisition module collects topology information between a plurality of the regional gateways and the computing power server from the optical interconnection network; A convergence ratio determination module, the convergence ratio determination module is connected to the topology acquisition module, and the convergence ratio determination module determines the convergence ratio of switches under each group of regional gateways; A training unit scheduling module, wherein the training unit scheduling module is connected to the convergence ratio determination module, and the training unit scheduling module splits the training unit of the model training task and allocates a plurality of training containers in the training unit to the computing power server according to the switch convergence ratio; The training unit scheduling module preferentially allocates the training containers of the same training unit to the computing power servers under the same group of regional gateways when the convergence ratio of the switch meets the requirement.

3. The integrated computing cluster scheduling and management system for optical interconnection according to claim 1, characterized in that: The management platform includes: A queue generation module, wherein the queue generation module receives the training task input from the outside and adds it to the training queue; A load collection module, wherein the load collection module obtains load information on a plurality of computing power servers; A task scheduling module, wherein the task scheduling module is connected to the queue generation module and the load acquisition module respectively, and the task scheduling module extracts the training tasks from the training queue according to the load information and distributes them to the corresponding computing power servers.

4. The integrated computing cluster scheduling and management system for optical interconnection according to claim 3 is characterized in that: The task scheduling module includes: A container number determination module, wherein the container number determination module splits the training task to determine a minimum number of training containers; A load determination module, wherein the load determination module determines the number of idle containers that can be deployed by the computing power server under the current load according to the load information; An allocation module, wherein the allocation module is connected to the container number determination module and the load determination module respectively, and the allocation module allocates the training task to the computing power server when the number of free containers meets the minimum number of training containers.

5. The integrated computing cluster scheduling and management system for optical interconnection according to claim 4 is characterized in that: The task scheduling module also includes: A pre-screening module, the pre-screening module is connected to the allocation module, the pre-screening module collects node heterogeneous resource information from the computing power server, and the pre-screening module pre-marks the computing power server that can be allocated for the training task in the training queue according to the node heterogeneous resource information to generate a resource classification attribute; The allocation module allocates the training tasks according to the resource classification attributes.

6. The integrated computing cluster scheduling and management system for optical interconnection according to claim 4, characterized in that: The task scheduling module also includes: A centralized scheduling module, wherein the centralized scheduling module is connected to the allocation module, wherein the centralized scheduling module reads the task type of the training task, and the centralized scheduling module centrally allocates the training task whose task type is marked as a GPU training task.

7. The integrated computing cluster scheduling and management system for optical interconnection according to claim 3 is characterized in that: The management platform also includes: A load balancing module, wherein the load balancing module is connected to the load acquisition module, and the load balancing module performs load balancing on the computing power server according to the load information.

8. The integrated computing cluster scheduling and management system for optical interconnection according to claim 1, characterized in that: The computing power server includes: A cache module, which receives and caches the training task transmitted from the optical internet network; A cache reading module, wherein the cache reading module is connected to the cache module and the processing module, and the cache reading module directly reads the training task data of the training task from the cache module and transmits it to the processing module.

9. The integrated computing cluster scheduling and management system for optical interconnection according to claim 1, characterized in that: The management platform also includes: A front-end page generation module, wherein the front-end page generation module creates a front-end page through which the user uploads the training task.

Images