Heterogeneous intelligent computing power optimization management scheduling system for accelerating large model training tasks

Abstract

The invention provides a heterogeneous intelligent computing power optimization management scheduling system for accelerating a large model training task, and the system comprises the steps: obtaining the performance of heterogeneous intelligent computing power equipment through the actual measurement of an algorithm and a framework, carrying out the resource description management through taking the computing power equipment with the highest performance as a reference standard conversion coefficient, and forming a heterogeneous intelligent computing power resource topology; meanwhile, a parallel method is provided, heterogeneous intelligent computing power resources are organized in an innovative mode, the parallel method comprises two layers, in the first layer, low-performance computing power nodes are combined into equivalent high-performance nodes, and therefore the effect of increasing the number of the intelligent computing power nodes is achieved; and in the second layer, parallel computing is performed among a plurality of low-performance nodes in the equivalent high-performance nodes, so that the processing results of the equivalent high-performance nodes are not lower than those of the reference high-performance computing power nodes. By adding the intelligent computing power nodes, the computing amount of a single node can be reduced through parallel computing, the processing time is shortened, and finally the purpose of accelerating a large model training task is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence computing power load scheduling, and specifically relates to a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model training tasks. Background Art

[0002] In recent years, with the rapid development of artificial intelligence, especially the development of large models, it has shown great potential in improving industry efficiency, innovating business models, supporting decision-making and developing new products and services. The application of artificial intelligence requires the support of corresponding intelligent computing power. Large group companies have many branches, or each branch has many business scenarios. Among them, large model training tasks and fine-tuning require a large amount of intelligent computing resources to carry out. In this case, the scale of group enterprise intelligent computing resources is relatively large, and they are often built in the form of heterogeneous intelligent computing resources. There should be technical means to realize the scheduling and management of the group's heterogeneous intelligent computing resources, so as to make fuller use of intelligent computing resources and better carry intelligent computing business. Large model training tasks generally use parallel solutions, including data parallelism, model parallelism, and pipeline hybrid parallelism. Parallel solutions require the collaborative work of multiple intelligent computing resource nodes. Currently, the implementation solutions for large-scale heterogeneous intelligent computing scheduling management on the market are mostly carried by computing nodes with the same processing performance, and the user specifies the type and quantity of intelligent computing resources when creating the task. In this case, parallel solution planning does not need to consider the differences brought by heterogeneous intelligent computing nodes and the secondary parallel solutions within heterogeneous intelligent computing, which is relatively simple. Therefore, it is necessary to introduce a new set of methods to make better use of the group's heterogeneous intelligent computing resources and better serve related users. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies in the prior art and provide a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model training tasks, thereby improving the utilization rate of heterogeneous intelligent computing power resources and accelerating large model training tasks.

[0004] The present invention specifically provides a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model training tasks. The system uses a computing power resource performance comparison conversion coefficient expression method to perform topological management on heterogeneous computing power resources. In the parallel plan planning of model training allocation tasks, the heterogeneous computing power resources are topologically organized into equivalent computing power nodes, and these equivalent power nodes are allowed to participate in the tasks to achieve the purpose of accelerating large model training tasks. The system includes a model development application layer, a computing framework layer, a resource management and scheduling layer, and a heterogeneous intelligent computing power resource layer;

[0005] The model development application layer is used to provide an interface for user operations, and the user operations include setting model training tasks;

[0006] The computing framework layer is used to provide a learning framework for model training tasks. It also formulates a parallel planning scheme based on the selected learning framework. The parallel scheme is planned according to the model size. It can be model parallelism and data parallelism or combined pipeline parallelism. The learning framework provides underlying support for the execution of parallel schemes.

[0007] The resource management scheduling layer calls intelligent computing resources for tasks according to model training tasks and heterogeneous intelligent computing resource topology, and the intelligent computing resources can be a heterogeneous combination;

[0008] The heterogeneous intelligent computing power resource layer is used to complete the management of heterogeneous intelligent computing power, including the measured single-node performance of various intelligent computing power resources under various commonly used learning frameworks, and the performance equivalent coefficient value of the intelligent computing power resource type with the best measured performance.

[0009] The model development application layer includes a data management module, an algorithm development module and a model training module;

[0010] Users upload data sets through the data management module, as well as delete, modify, and export data sets;

[0011] Users upload algorithms through the algorithm development module, as well as modify and delete algorithms. Algorithms can be publicly released or developed by users themselves;

[0012] The user sets the model training task through the model training module, including setting the algorithm, data set, and learning framework used for model training;

[0013] The computing framework layer includes a parallel planning module, which is used to perform parallel scheme planning for model training tasks and has pipeline parallelism, data parallelism, and model parallelism capabilities;

[0014] The resource management scheduling layer includes a task queue management module, a resource topology generation module, a scheduling decision generation module and a Kubernetes (container cluster management system) module;

[0015] The scheduling decision generation module realizes the allocation and scheduling of intelligent computing resources in the heterogeneous intelligent computing resource layer through the Kubernetes module, specifically including: the scheduling decision generation module allocates intelligent computing resources of the heterogeneous intelligent computing resource layer to the model training task according to the setting of the model training task and the resource topology generation information, and then calls the Kubernetes module to establish a container for the model training task, and performs resource scheduling in units of containers;

[0016] The heterogeneous intelligent computing power resource layer includes a resource overall description module and a heterogeneous intelligent computing power resource performance description module. The heterogeneous resource performance description module is used to provide various performance indicators and available quantity statistics of multiple types of heterogeneous intelligent computing power resources. The various performance indicators of the multiple types of heterogeneous intelligent computing power resources include the peak computing power, video memory size, and inter-card interconnection bandwidth of a single card under the corresponding model and corresponding AI framework. This part of data needs to be tested to obtain corresponding performance data when the system first introduces a new model, AI framework, and heterogeneous computing power type; the resource overall description module is used to provide global data statistics of heterogeneous intelligent computing power resources, and the statistical method can be manual entry by the administrator or an automated method using device detection. Heterogeneous intelligent computing power topology data is generated based on the resource overall description module and the heterogeneous intelligent computing power resource performance description module.

[0017] The learning framework includes a deep learning framework and a reinforcement learning framework, specifically including TensorFlow, PyTorch, Caffe, Mindspore, MXNet, and PaddlePaddle.

[0018] The models in the model training tasks include natural language processing models, image processing models, text processing models, audio and video processing models, and multi-modal processing models;

[0019] The resource management and scheduling layer constructs a suitable parallel scheme based on the model training task and the heterogeneous intelligent computing power topology data, and generates an intelligent computing power resource scheduling decision.

[0020] The heterogeneous intelligent computing power resource layer is also used to generate a database for storing heterogeneous intelligent computing power topology data; the heterogeneous intelligent computing power resource layer uses the intelligent computing power resource type G with the best measured performance in the performance dimension i as the standard quantity 1, and the measured performance of other heterogeneous intelligent computing power resources is converted into the corresponding equivalent coefficient value A by referring to the performance value of the resource type G as the standard quantity 1. ij , where j is the number of heterogeneous intelligent computing power types;

[0021] Set the required video memory capacity size for model training as C, and the video memory capacity of a single intelligent computing power node as g. When the number of intelligent computing power nodes n ≥ C / g, these n intelligent computing power nodes can complete one sub-dataset training iteration of the model, that is, model parallelism; when n < C / g, then the combination of n intelligent computing power nodes cannot complete one sub-dataset training iteration of the model, and some or all of the nodes must be selected from the n nodes to replace the sub-model and continue to complete the sub-dataset training. When all batches of sub-datasets of the dataset are completed, one training iteration is completed;

[0022] When the computing framework layer formulates a parallel planning scheme, the first-layer parallel scheme allocates the number of intelligent computing resources in units of standard quantity, and the number of intelligent computing resources allocated is n1=(Σ*N j ), *N j It is the converted quantity of the jth type of intelligent computing power resource type; the second-layer parallel solution is a multi-node combination converted by an equivalent coefficient within the first-layer intelligent computing power resource node group. The various indicators after the multi-node combination are not less than the standard quantity 1, and the various indicators include peak computing power, video memory size, and inter-card interconnection bandwidth.

[0023] The heterogeneous intelligent computing resource layer, for the j-th type of intelligent computing resource, combines the values ​​of the performance dimension i of the j-th type of intelligent computing resource to be no less than the standard amount 1, and the corresponding number of heterogeneous intelligent computing resources after combination is N i , N i is an integer not less than 0, N i Construct a one-dimensional array;

[0024] When the resource management scheduling layer calls the number of intelligent computing resources, it obtains the number of available intelligent computing resources N of performance dimension i. i , as the basis for resource allocation quantity, N i The maximum value should not be greater than the N value converted according to the video memory size dimension, * N = min (N i , N), to ensure that the combined resource group can equivalently complete the training phase calculation required by the standard intelligent computing resource node. The equivalent here includes the calculation amount and completion time. Assuming that the video memory size of the standard node is 80G and the video memory size of type j is 64G, the equivalent coefficient is 0.8, that is, the logical number of type j combined according to the graphics card size and performance dimension N = 0.8*the original number of nodes of type j.

[0025] The resource management scheduling layer generates resource scheduling decision data based on training tasks and parallel solutions combined with intelligent computing power topology data;

[0026] The resource scheduling decision data includes intelligent computing power node task cluster information generated according to the node information topology corresponding to the allocation scheduling event.

[0027] The resource management scheduling layer includes a kubernetes cluster management module, which schedules and manages the generated intelligent computing power node task cluster, saves the number of allocated intelligent computing power nodes, and dynamically updates the number of available intelligent computing power resources.

[0028] The resource management scheduling layer also includes a task queue management module, which is used to receive model training tasks initiated by users, transfer the model training tasks to the task queue, and trigger the scheduling decision generation module; the task queue includes more than two model training tasks.

[0029] The system includes a console, which is connected to each module of the system and is used to receive background task input, parameter input, task execution status, and heterogeneous intelligent computing power resource status for visual display.

[0030] Beneficial effects:

[0031] The intelligent computing resources required for large model training are large, and the training completion time is long. Generally, a parallel solution is adopted. In the parallel training process, each node calculation, communication interaction and other links must be completed. Poor parallel solution planning may result in the inability to play the advantage of the number of nodes. Existing parallel solutions generally use homogeneous nodes to avoid the complexity of the solution, optimize the use of heterogeneous intelligent computing resources, or divide the calculation subgraph according to the target model and then globally decompose it into heterogeneous intelligent computing resources for pipeline calculation, or adopt a rougher strategy to directly merge low-performance intelligent computing resources step by step to form a merged system to undertake training calculation tasks. The above-mentioned processing scheme has great limitations. Scheme 1 is too complicated to implement, and it cannot be divided and implemented for closed models; Scheme 2 strategy may cause the overall processing capacity of the system to decrease when the model is large due to the forced use of low-performance intelligent computing resources. The present invention has a certain universality and the implementation method is relatively simple. It ensures that the processing performance of the combined nodes participating in the training task is not lower than that of other nodes through performance conversion. At the same time, only homogeneous nodes are used inside the combined nodes, which can fully reduce the impact of communication performance loss. The present invention is particularly suitable for the unified construction and use of heterogeneous intelligent computing resources, and for running large model training task application scenarios. While improving task execution efficiency, it can also improve system resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0033] Figure 1 This is an architectural diagram of a heterogeneous intelligent computing power optimization management and scheduling method for accelerating large model training tasks according to an embodiment of the present application.

[0034] Figure 2 It is a flowchart of the steps of a heterogeneous intelligent computing power optimization management and scheduling method for accelerating large model training tasks according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] The present invention specifically provides a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model training tasks, including a model development application layer, a computing framework layer, a resource management and scheduling layer, and a heterogeneous intelligent computing power resource layer;

[0036] The model development application layer is used to provide an interface for user operations, and the user operations include setting model training tasks;

[0037] The computing framework layer is used to provide a learning framework for model training tasks. It also formulates a parallel planning scheme based on the selected learning framework. The parallel scheme is planned according to the model size. It can be model parallelism and data parallelism or combined pipeline parallelism. The learning framework provides underlying support for the execution of parallel schemes.

[0038] The resource management scheduling layer calls intelligent computing resources for tasks according to model training tasks and heterogeneous intelligent computing resource topology, and the intelligent computing resources can be a heterogeneous combination;

[0039] The heterogeneous intelligent computing power resource layer is used to complete the management of heterogeneous intelligent computing power, including the measured single-node performance of various intelligent computing power resources under various commonly used learning frameworks, and the performance equivalent coefficient value of the intelligent computing power resource type with the best measured performance.

[0040] The model development application layer includes a data management module, an algorithm development module and a model training module;

[0041] Users upload data sets through the data management module, as well as delete, modify, and export data sets;

[0042] Users upload algorithms through the algorithm development module, as well as modify and delete algorithms. Algorithms can be publicly released or developed by users themselves;

[0043] The user sets the model training task through the model training module, including setting the algorithm, data set, and learning framework used for model training;

[0044] The computing framework layer includes a parallel planning module, which is used to perform parallel scheme planning for model training tasks and has pipeline parallelism, data parallelism, and model parallelism capabilities;

[0045] The resource management scheduling layer includes a task queue management module, a resource topology generation module, a scheduling decision generation module and a Kubernetes (container cluster management system) module;

[0046] The scheduling decision generation module realizes the allocation and scheduling of intelligent computing resources in the heterogeneous intelligent computing resource layer through the Kubernetes module, specifically including: the scheduling decision generation module allocates intelligent computing resources of the heterogeneous intelligent computing resource layer to the model training task according to the setting of the model training task and the resource topology generation information, and then calls the Kubernetes module to establish a container for the model training task, and performs resource scheduling in units of containers;

[0047] The heterogeneous intelligent computing power resource layer includes a resource overall description module and a heterogeneous intelligent computing power resource performance description module. The heterogeneous resource performance description module is used to provide various performance indicators and available quantity statistics of various types of heterogeneous intelligent computing power resources. The various performance indicators of various types of heterogeneous intelligent computing power resources include the peak computing power, video memory size, and inter-card interconnection bandwidth of a single card under the corresponding model and corresponding AI framework. This part of data needs to be tested to obtain corresponding performance data when the system introduces a new model, AI framework, and heterogeneous computing power type for the first time; the resource overall description module is used to provide global data statistics of heterogeneous intelligent computing power resources. The statistical method can be manual entry by the administrator or automated by device detection. Heterogeneous intelligent computing power topology data is generated according to the resource overall description module and the heterogeneous intelligent computing power resource performance description module.

[0048] The learning framework includes a deep learning framework and a reinforcement learning framework, specifically including TensorFlow, PyTorch, Caffe, Mindspore, MXNet, and PaddlePaddle.

[0049] The models in the model training task include natural language processing models, image processing models, text processing models, audio and video processing models, and multimodal processing models;

[0050] The resource management scheduling layer constructs a suitable parallel solution based on model training tasks and heterogeneous intelligent computing power topology data, and generates intelligent computing power resource scheduling decisions.

[0051] The heterogeneous intelligent computing power resource layer is also used to generate a database for storing heterogeneous intelligent computing power topology data; the heterogeneous intelligent computing power resource layer uses the intelligent computing power resource type G with the best measured performance as the standard quantity 1 in the performance dimension i, and the measured performance of other heterogeneous intelligent computing power resources is converted to the corresponding equivalent coefficient value A by referring to the performance value of resource type G as the standard quantity 1. ij , j is the number of heterogeneous intelligent computing power types;

[0052] Large models are of large volume (assuming the required video memory capacity for training is C), and the video memory capacity of a single intelligent computing power node (assuming its size is g) cannot store the entire model. Therefore, it needs to be sliced and split into sub-models and stored on multiple intelligent computing power nodes. When the number of intelligent computing power nodes n ≥ C / g, these n intelligent computing power nodes can complete one sub-dataset training iteration of the model, that is, model parallelism; when n < C / g, then the combination of n intelligent computing power nodes cannot complete the sub-dataset training iteration of the model at one time. It is necessary to select some or all of the n nodes to replace the sub-model and continue to complete the training of this batch of sub-datasets. When all batches of sub-datasets of the dataset are completed, one training iteration is completed (including three calculations of forward propagation, backward propagation, and parameter update). Thus, it can be seen the impact of the number and performance of intelligent computing power nodes on the model training task. Organize heterogeneous intelligent computing power resources in an innovative way, combine low-performance computing power nodes into equivalent high-performance nodes to achieve the effect of increasing the number of intelligent computing power nodes, and ultimately achieve the purpose of accelerating the large model training task. When the computing framework layer formulates a parallel planning scheme, the first-layer parallel scheme allocates the intelligent computing power resources in units of standard quantities, and the number of intelligent computing power resource nodes allocated n1 = (Σ * N j ) * N j is the converted quantity of the jth type of intelligent computing power resource type; the second-layer parallel scheme is a multi-node combination within the first-layer intelligent computing power resource node group, converted according to the equivalent coefficient. All indicators after the multi-node combination are not lower than the standard quantity 1, and the indicators include peak computing power, video memory size, and inter-card interconnection bandwidth.

[0053] For the jth type of intelligent computing power resource type in the heterogeneous intelligent computing power resource layer, it is combined according to the value of the performance dimension i of the jth type of intelligent computing power resource type to be not less than the standard quantity 1. The corresponding quantity N i of the heterogeneous intelligent computing power resources after combination, N i is an integer not less than 0, and N i constitutes a one-dimensional array;

[0054] When the resource management and scheduling layer calls the number of intelligent computing power resources, the available number of intelligent computing power resources N i of the performance dimension i is obtained as the basis for resource allocation quantity. The maximum value in N i should not be greater than the N value converted according to the video memory size dimension, *N = min(N i , N), to ensure that the combined resource group can equivalently complete the computational workload of the training link required by the standard intelligent computing power resource node. Here, the equivalence includes computational workload and completion time. Assume the video memory size of the standard quantity node is 80G, and the video memory size of type j is 64G, then the equivalent coefficient is 0.8, that is, the logical quantity N of type j after combination according to the video memory size performance dimension = 0.8 * the original number of type j nodes.

[0055] The resource management scheduling layer generates resource scheduling decision data based on training tasks and parallel solutions combined with intelligent computing power topology data;

[0056] The resource scheduling decision data includes intelligent computing power node task cluster information generated according to the node information topology corresponding to the allocation scheduling event.

[0057] The resource management scheduling layer includes a kubernetes cluster management module, which schedules and manages the generated intelligent computing power node task cluster, saves the number of allocated intelligent computing power nodes, and dynamically updates the number of available intelligent computing power resources.

[0058] The resource management scheduling layer also includes a task queue management module, which is used to receive model training tasks initiated by users, transfer the model training tasks to the task queue, and trigger the scheduling decision generation module; the task queue includes more than two model training tasks.

[0059] The system includes a console, which is connected to each module of the system and is used to receive background task input, parameter input, task execution status, and heterogeneous intelligent computing power resource status for visual display.

[0060] Large-scale intelligent computing resources are required in large-model training tasks, and parallel scheme planning needs to be carried out in combination with computing power performance. Some computing power equipment cannot even bear the computing amount of one layer in the parallel scheme of large-model training tasks alone. This is undoubtedly a huge waste of resources for intelligent computing centers with a large number of heterogeneous intelligent computing resources. Therefore, it is necessary to optimize the existing intelligent computing resource scheduling method to improve the utilization rate of heterogeneous intelligent computing resources, and at the same time accelerate large-model training tasks. The present invention proposes a heterogeneous intelligent computing resource optimization management and scheduling system for accelerating large-model training tasks.

[0061] refer to Figure 1 , Figure 1 This is an architecture diagram of a heterogeneous intelligent computing resource management and scheduling system for accelerating large model training tasks provided in Embodiment 1 of the present invention, and the system includes:

[0062] The model development application layer 11 is used to provide an interface for user operations, wherein the user operations include setting model training tasks;

[0063] The computing framework layer 12 is used to provide a learning framework used by the model training task;

[0064] The resource management and scheduling layer 13 is used to allocate and schedule intelligent computing resources for the model training task according to the topology of heterogeneous intelligent computing resources to execute the model training task;

[0065] The heterogeneous intelligent computing resource layer 14 is used to provide heterogeneous intelligent computing resources, including an overall view of the heterogeneous intelligent computing resources and a description of the performance of the heterogeneous intelligent computing resources.

[0066] The model development application layer 11 includes a data management module 111, an algorithm development module 112, and a model training module 113. Users upload data sets, delete, modify, and export data sets through the data management module 111. Users upload algorithms, modify, and delete algorithms through the algorithm development module 112. Users set model training tasks through the model training module 113, including setting the algorithms, data sets, and learning frameworks used for model training.

[0067] The learning framework 121 provided by the computing framework layer 12 includes a deep learning framework and a reinforcement learning framework, including mainstream AI frameworks such as TensorFlow, PyTorch, Caffe, Mindspore, MXNet, and PaddlePaddle.

[0068] During use, the user can specify the computing framework used for model training through the model training module 113, so as to facilitate the quick deployment of the training process and improve the operation efficiency.

[0069] The computing framework layer 12 also includes a parallel planning module 122, which is used for parallel solution planning for large model training tasks, including pipeline parallelism, data parallelism, and model parallelism capabilities.

[0070] The resource management scheduling layer 13 includes a task queue management module 131, a resource topology generation module 132, a scheduling decision generation module 133, and a Kubernetes module 134. The scheduling decision generation module 133 implements the allocation and scheduling of the intelligent computing resources in the heterogeneous intelligent computing resource layer through the Kubernetes module 134. Specifically, the scheduling decision generation module 133 allocates the intelligent computing resources of the heterogeneous intelligent computing resource layer 14 to the model training task according to the setting of the model training task and the resource topology generation information, and then calls the Kubernetes module 134 to establish a container for the model training task, and performs resource scheduling in units of containers.

[0071] The heterogeneous intelligent computing resource layer 14 includes a resource overall description module 141 and a heterogeneous resource performance description module 142. The heterogeneous resource performance description module 142 is used to provide various performance indicators and available quantity statistics of various types of heterogeneous intelligent computing resources, such as NVIDIA, AMD, Huawei Ascend, Cambrian, Baidu Kunlun Core, Moore Thread, Haiguang DCU, Tianshu Zhixin, Suiyuan Technology and other different manufacturers or different generations of intelligent computing resources of the same manufacturer. The resource overall description module 141 is used to provide global data statistics of heterogeneous intelligent computing resources.

[0072] The present invention provides a method for optimizing management and scheduling of heterogeneous intelligent computing power resources for accelerating large model training tasks. By integrating heterogeneous intelligent computing power resources and refined parallel solutions, it solves the problem of the existing intelligent computing power management and scheduling platform adopting a single computing power architecture, so that the heterogeneous intelligent computing power of the intelligent computing center can be more fully utilized and large model training tasks can be further accelerated.

[0073] like Figure 2 As shown, another embodiment of the present invention provides a method for accelerating large model training tasks. Taking heterogeneous intelligent computing resources (including 32 A-type computing cards, 32 B-type computing cards, and 32 C-type computing cards) and training a large model with 65B parameters as an example, the measured single-node performance of various computing cards is shown in Table 1:

[0074] Table 1

[0075] Hash Card A Hash Card B Hash Card C Video memory size 80G 64G 32G FP16 computing power 313TFLOPS 313TFLOPS 128TFLOPS Inter-card connection 400GB / s 392GB / s 200GB / s

[0076] The heterogeneous intelligent computing resource management and scheduling system described in Example 1 is used for implementation, including:

[0077] S101, the user sets a model training task through the model training and application layer and starts the task, wherein setting the model training task includes selecting an algorithm, a data set, and a learning framework for training;

[0078] S102, the heterogeneous intelligent computing power resource layer manages the heterogeneous intelligent computing power resources and generates a dynamic topology of the heterogeneous intelligent computing power resources;

[0079] S103, the resource management scheduling layer obtains the dynamic topology of the heterogeneous intelligent computing resource layer according to the setting of the model training task, completes the parallel planning scheme, and generates a resource scheduling decision;

[0080] S104: The resource management scheduling layer allocates and calls the intelligent computing resources of the heterogeneous intelligent computing resource layer for the model training task according to the resource scheduling decision to perform model training.

[0081] Specifically, the user sets the model training task through the model training module 113, including setting the algorithm, data set, and learning framework used for model training. The algorithm used can be uploaded by the user through the algorithm development module 112, and the data set used can be uploaded by the user through the data management module 111. The algorithm and data set can also be pre-made in the system, and the learning framework used can be pre-made in the system.

[0082] After the settings are completed, the user starts the model training task. The platform calls the selected learning framework from the learning framework 121 of the computing framework layer 12 according to the user's settings, extracts the data set as training data, extracts the algorithm code and executes it;

[0083] The heterogeneous intelligent computing resource layer 14 generates heterogeneous intelligent computing resource topology data according to the measured performance data of the three types of computing cards A, B, and C, and the standard value is (video memory size: 80G; FP16 computing power: 313TFLOPS; card-to-card interconnection: 400GB / s); the heterogeneous intelligent computing resource topology data calculated according to the algorithm of claim 5 is {(1, 1, 1, 32); (0.8, 1, 0.98, 16); (0.4, 0.41, 0.5, 10); the number of computing nodes n=32+16+10=58 as the first-layer parallel scheme is calculated according to the algorithm of claim 6; the number of computing nodes in the second-layer parallel scheme is 2 for computing card B and 3 for computing card C;

[0084] When heterogeneous computing power cards are not used, the optimal choice for training tasks is to use 32 A computing power cards; after using heterogeneous computing power cards B and C for acceleration, the training task becomes to use 58 equivalent computing power cards, which will greatly shorten the overall model training time (because the equivalent computing power nodes need to be parallelized for the second time, it will actually cause some performance loss, mainly due to communication and synchronization between nodes; but because the performance of the equivalent computing power node combination is actually greater than that of the standard value node, it will also shorten the calculation time);

[0085] At the same time, the resource management scheduling layer 13 generates heterogeneous intelligent computing resource scheduling decisions based on the topology of heterogeneous intelligent computing resources and parallel scheme planning. When new intelligent computing resources of the specified type are available, the parallel scheme is updated, new intelligent computing resource nodes are added, and the heterogeneous intelligent computing resource scheduling decisions are updated and passed to the Kubernetes module.

[0086] Then, the resource management scheduling layer 13 calls the Kubernetes module 134 to create a container for the model training task. When there are multiple model training tasks executed in parallel, each task has a corresponding container. The resource management scheduling layer 13 can call the Kubernetes module 134 to uniformly manage each container.

[0087] The heterogeneous intelligent computing resource management and scheduling method for accelerating large model training tasks provided in the embodiments of the present invention can be applied to various large model training tasks including natural language processing models, image processing models, text processing models, audio and video processing models, and multimodal processing models.

[0088] The present invention provides a heterogeneous intelligent computing power optimization management and scheduling system for accelerating large model training tasks. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. Heterogeneous intelligent computing power optimization management and scheduling system to accelerate large model training tasks, It is characterized in that It includes model development application layer, computing framework layer, resource management scheduling layer and heterogeneous intelligent computing resource layer; The model development application layer is used to provide an interface for user operations, and the user operations include setting model training tasks; The computing framework layer is used to provide a learning framework for model training tasks, and also formulates a parallel planning scheme based on the selected learning framework; The resource management scheduling layer calls intelligent computing resources for tasks according to model training tasks and heterogeneous intelligent computing resource topology, and the intelligent computing resources can be a heterogeneous combination; The heterogeneous intelligent computing power resource layer is used to complete the management of heterogeneous intelligent computing power, including the measured single-node performance of various intelligent computing power resources under various commonly used learning frameworks, and the performance equivalent coefficient value of the intelligent computing power resource type with the best measured performance.

2. The system according to claim 1, It is characterized in that The model development application layer includes a data management module, an algorithm development module and a model training module; Users upload data sets through the data management module, as well as delete, modify, and export data sets; Users upload algorithms through the algorithm development module, and modify and delete algorithms; The user sets the model training task through the model training module, including setting the algorithm, data set, and learning framework used for model training; The computing framework layer includes a parallel planning module, which is used to perform parallel scheme planning for model training tasks and has pipeline parallelism, data parallelism, and model parallelism capabilities; The resource management scheduling layer includes a task queue management module, a resource topology generation module, a scheduling decision generation module and a Kubernetes module; The scheduling decision generation module realizes the allocation and scheduling of intelligent computing resources in the heterogeneous intelligent computing resource layer through the Kubernetes module, specifically including: the scheduling decision generation module allocates intelligent computing resources of the heterogeneous intelligent computing resource layer to the model training task according to the setting of the model training task and the resource topology generation information, and then calls the Kubernetes module to establish a container for the model training task, and performs resource scheduling in units of containers; The heterogeneous intelligent computing power resource layer includes a resource overall description module and a heterogeneous intelligent computing power resource performance description module. The heterogeneous resource performance description module is used to provide various performance indicators and available quantity statistics of various types of heterogeneous intelligent computing power resources. The various performance indicators of the various types of heterogeneous intelligent computing power resources include the peak computing power, video memory size, and inter-card interconnection bandwidth of a single card under the corresponding model and corresponding AI framework; the resource overall description module is used to provide global data statistics of heterogeneous intelligent computing power resources; and heterogeneous intelligent computing power topology data is generated according to the resource overall description module and the heterogeneous intelligent computing power resource performance description module.

3. The system according to claim 2, It is characterized in that The learning framework includes a deep learning framework and a reinforcement learning framework, specifically including TensorFlow, PyTorch, Caffe, Mindspore, MXNet, and PaddlePaddle.

4. The system according to claim 3, wherein, the models in the model training tasks include natural language processing models, image processing models, text processing models, audio-video processing models, and multi-modal processing models; the resource management and scheduling layer constructs a suitable parallel scheme based on the model training tasks and heterogeneous intelligent computing power topology data, and generates an intelligent computing power resource scheduling decision.

5. The system according to claim 4, wherein, The heterogeneous intelligent computing power resource layer is also used to generate a database for storing heterogeneous intelligent computing power topology data; the heterogeneous intelligent computing power resource layer uses the intelligent computing power resource type G with the best measured performance as the standard quantity 1 in the performance dimension i, and the measured performance of other heterogeneous intelligent computing power resources is converted to the corresponding equivalent coefficient value A by referring to the performance value of resource type G as the standard quantity 1. ij , j is the number of heterogeneous intelligent computing power types; assuming the required video memory capacity for model training is C, and the video memory capacity of a single intelligent computing power node is g, when the number of intelligent computing power nodes n ≥ C / g, these n intelligent computing power nodes can complete one sub-dataset training iteration of the model, i.e., model parallelism; when n < C / g, then the combination of n intelligent computing power nodes cannot complete the sub-dataset training iteration of the model at once, and it is necessary to select some or all of the n nodes to replace the sub-model and continue to complete the sub-dataset training. When all batches of sub-datasets of the dataset are completed, one training iteration is completed; When the computing framework layer formulates a parallel planning scheme, the first-layer parallel scheme allocates the number of intelligent computing resources in units of standard quantity. The number of intelligent computing resources allocated is n 1 =(Σ*N j ), *N j It is the converted quantity of the jth type of intelligent computing power resource type; the second-layer parallel solution is a multi-node combination converted by an equivalent coefficient within the first-layer intelligent computing power resource node group. The various indicators after the multi-node combination are not less than the standard quantity 1, and the various indicators include peak computing power, video memory size, and inter-card interconnection bandwidth.

6. The system according to claim 5, wherein, The heterogeneous intelligent computing resource layer, for the j-th type of intelligent computing resource, combines the values ​​of the performance dimension i of the j-th type of intelligent computing resource to be no less than the standard amount 1, and the corresponding number of heterogeneous intelligent computing resources after combination is N i , N i is an integer not less than 0, N i Construct a one-dimensional array; When the resource management scheduling layer calls the number of intelligent computing resources, it obtains the number of available intelligent computing resources N of performance dimension i. i , as the basis for resource allocation quantity, N i The maximum value should not be greater than the N value converted according to the video memory size dimension, * N = min (N i , N) to ensure that the merged resource group can equivalently complete the computing workload required by the standard intelligent computing resource node for the training phase.

7. The system according to claim 6, wherein, the resource management and scheduling layer generates resource scheduling decision data according to the training tasks and parallel scheme, in combination with the heterogeneous intelligent computing power topology data; the resource scheduling decision data includes the intelligent computing power node task cluster information generated according to the node information topology corresponding to the allocation scheduling event.

8. The system according to claim 7, wherein, the resource management and scheduling layer includes a kubernetes cluster management module, and the kubernetes cluster management module schedules and manages the generated intelligent computing power node task clusters, saves the number of allocated intelligent computing power nodes, and dynamically updates the number of available intelligent computing power resources.

9. The system according to claim 8, wherein, the resource management and scheduling layer further includes a task queue management module, and the task queue management module is used to receive the model training tasks initiated by the user, transmit the model training tasks to the task queue, and trigger the scheduling decision generation module; the task queue includes two or more model training tasks.

10. The system according to claim 1, wherein, the system includes a console, which is connected to each module of the system and is used to visually display the background task input, parameter input, task execution status, and heterogeneous intelligent computing power resource situation.

Images