Training method and system based on multi-machine multi-card large model
By adopting the data parallel and model parallel training mode of the Deepspeed framework in a multi-machine and multi-card environment, the problem of low training efficiency of large-scale data sets and complex models is solved, and efficient model training is achieved.
Patent Information
- Application Number
- CN202510006681.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-03
- Publication Date
- 2025-04-11
AI Technical Summary
When faced with large-scale data sets and complex models, existing model training requires huge computing resources and long training cycles, and cannot achieve parallel training of models, resulting in low training efficiency.
Using a multi-machine and multi-card training method, through the deep learning framework Deepspeed, the training data and model are divided into multiple sub-blocks and sub-models through the deep learning framework Deepspeed, using data parallelism and model parallel training modes, and parallel training is performed on multiple computing nodes. The central node coordinates the gradient information of each computing node and updates the model parameters through the ZeRO optimizer.
It realizes efficient training of large models, reduces dependence on huge computing resources, shortens the training cycle, and improves training efficiency.
Smart Images
Figure CN119808895A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method and system based on a multi-machine and multi-card large model. Background Art
[0002] In the field of artificial intelligence technology, with the continuous development and expansion of deep learning models, more and more tasks require the use of large-scale data sets and complex models for training.
[0003] However, existing model training usually requires huge computing resources and long training cycles when facing large-scale data sets and complex models, and cannot achieve parallel training of models, resulting in low training efficiency of large models. Summary of the invention
[0004] The embodiments of the present invention provide a training method and system based on a large model with multiple machines and multiple cards, which can improve the training efficiency of the large model.
[0005] In a first aspect, an embodiment of the present invention provides a training method based on a multi-machine multi-card large model, the method comprising:
[0006] Receive training data sent by the data distribution module;
[0007] Based on the deep learning framework Deepspeed, the training data is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node, wherein each computing node includes at least one server, and each server is configured with at least one graphics card;
[0008] Using a gradient merging module to merge the gradient information of the at least one computing node on a central node to generate an average value of the gradient information, wherein the central node is used to coordinate the gradient information of the at least one computing node;
[0009] Based on the average value of the gradient information, the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed are used to update the large model parameters.
[0010] Preferably,
[0011] After the training data is received from the data distribution module, the method further includes:
[0012] Based on the data parallel training mode of the deep learning framework Deepspeed, the training data is divided into at least one data block;
[0013] sending the at least one data block to a corresponding computing node;
[0014] The using the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node includes:
[0015] The at least one data block is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node.
[0016] Preferably,
[0017] Based on the deep learning framework Deepspeed, the at least one sub-model is trained in parallel using the training data to generate gradient information of at least one computing node, including:
[0018] Based on the model parallel training mode of the deep learning framework Deepspeed and the structure and computational complexity of the large model, the large model is divided into the at least one sub-model;
[0019] The at least one sub-model is trained in parallel using the at least one data block to generate gradient information of at least one computing node.
[0020] Preferably,
[0021] The step of merging the gradient information of the at least one computing node on the central node using the gradient merging module to generate the average value of the gradient information includes:
[0022] According to the first formula and the gradient information of the at least one computing node, the gradient information of the at least one computing node is merged on the central node using a gradient merging module to generate a gradient information average value, wherein the first formula is:
[0023]
[0024] in, is used to represent the average value of the gradient information, X1 is used to represent the gradient information of the first computing node, X2 is used to represent the gradient information of the second computing node, and X n It is used to represent the gradient information of the nth computing node, and N is used to represent the number of computing nodes;
[0025] Preferably,
[0026] After the large model parameters are updated based on the average value of the gradient information using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed, the method further includes:
[0027] According to a preset load threshold, determining whether there is a first current computing node among the at least one computing node whose load is higher than the load threshold;
[0028] When it is determined that the load of the first current computing node among the at least one computing node is higher than the load threshold, determining a second current computing node with the smallest load among the at least one computing node;
[0029] The submodel of the first current computing node is assigned to the second current computing node.
[0030] In a second aspect, an embodiment of the present invention provides a training system based on a multi-machine multi-card large model, the system comprising: a data receiving module, a data processing module, a gradient merging module and a parameter updating module;
[0031] The data receiving module is used to receive the training data sent by the data distribution module;
[0032] The data processing module is used to perform parallel training on at least one sub-model using the training data based on the deep learning framework Deepspeed, and generate gradient information of at least one computing node, wherein each computing node includes at least one server, and each server is configured with at least one graphics card;
[0033] The gradient merging module is used to merge the gradient information of the at least one computing node on the central node to generate an average value of the gradient information, wherein the central node is used to coordinate the gradient information of the at least one computing node;
[0034] The parameter updating module is used to update the large model parameters based on the average value of the gradient information using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed.
[0035] Preferably,
[0036] After the data receiving module, it further includes: a data segmentation module;
[0037] The data segmentation module is used to perform:
[0038] Based on the data parallel training mode of the deep learning framework Deepspeed, the training data is divided into at least one data block;
[0039] sending the at least one data block to a corresponding computing node;
[0040] The using the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node includes:
[0041] The at least one data block is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node.
[0042] Preferably,
[0043] The data processing module is further used to execute:
[0044] Based on the model parallel training mode of the deep learning framework Deepspeed and the structure and computational complexity of the large model, the large model is divided into the at least one sub-model;
[0045] The at least one sub-model is trained in parallel using the at least one data block to generate gradient information of at least one computing node.
[0046] Preferably,
[0047] The gradient merging module is also used to execute:
[0048] According to the first formula and the gradient information of the at least one computing node, the gradient information of the at least one computing node is merged on the central node to generate a gradient information average value, wherein the first formula is:
[0049]
[0050] in, is used to represent the average value of the gradient information, X1 is used to represent the gradient information of the first computing node, X2 is used to represent the gradient information of the second computing node, and X n It is used to represent the gradient information of the nth computing node, and N is used to represent the number of computing nodes.
[0051] Preferably,
[0052] After the parameter updating module, it further includes: a load balancing module;
[0053] The load balancing module is used to perform:
[0054] According to a preset load threshold, determining whether there is a first current computing node among the at least one computing node whose load is higher than the load threshold;
[0055] When it is determined that the load of the first current computing node among the at least one computing node is higher than the load threshold, determining a second current computing node with the smallest load among the at least one computing node;
[0056] The submodel of the first current computing node is assigned to the second current computing node.
[0057] In a third aspect, an embodiment of the present invention provides a training system based on a multi-machine multi-card large model, comprising: at least one memory and at least one processor;
[0058] The at least one memory is used to store a machine-readable program;
[0059] The at least one processor is used to call the machine-readable program to execute any method described in the first aspect of claim 1.
[0060] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.
[0061] The embodiment of the present invention provides a training method and system based on a multi-machine multi-card large model. The implementation of the method includes: configuring multiple servers on multiple computing nodes, and each server is configured with multiple graphics cards to form a multi-machine multi-card computing power cluster. The received training data is distributed to each computing node for processing through a data distribution module. At the same time, the large model is divided into multiple sub-models and distributed to different computing nodes for parallel training based on the deep learning framework Deepspeed, while the central node is used to coordinate the training work of each computing node. The gradient merging module can be used to merge the gradient information of each computing node on the central node to obtain the average value of the gradient information, and finally the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed are used to update the model parameters based on the obtained average value of the gradient information. Through the above training process, parallel training of data and models can be achieved, and when facing large-scale data sets and complex models, huge computing resources and long training cycles are not required, thereby improving the training efficiency of large models. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0063] Figure 1 It is a flow chart of a training method based on a multi-machine multi-card large model provided by an embodiment of the present invention;
[0064] Figure 2 It is a flowchart of another training method based on a multi-machine multi-card large model provided by an embodiment of the present invention;
[0065] Figure 3 It is a schematic diagram of a training system based on a multi-machine multi-card large model provided by an embodiment of the present invention;
[0066] Figure 4is a schematic diagram of another training system based on a multi-machine and multi-card large model provided by an embodiment of the present invention;
[0067] Figure 5 It is a schematic diagram of another training system based on a multi-machine and multi-card large model provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0069] like Figure 1 As shown, an embodiment of the present invention provides a training method based on a multi-machine multi-card large model, which may include the following steps:
[0070] Step 101: receiving training data sent by a data distribution module;
[0071] Step 102: Based on the deep learning framework Deepspeed, use the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node, wherein each computing node includes at least one server, and each server is configured with at least one graphics card;
[0072] Step 103: using a gradient merging module to merge the gradient information of at least one computing node on the central node to generate an average value of the gradient information, wherein the central node is used to coordinate the gradient information of at least one computing node;
[0073] Step 104: Based on the average value of the gradient information, the large model parameters are updated using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed.
[0074] In an embodiment of the present invention, a training method based on a multi-machine multi-card large model is provided. The implementation of the method includes: configuring multiple servers on multiple computing nodes, and each server is configured with multiple graphics cards to form a multi-machine multi-card computing power cluster. The received training data is distributed to each computing node for processing through a data distribution module. At the same time, the large model is divided into multiple sub-models and distributed to different computing nodes for parallel training based on the deep learning framework Deepspeed, while the central node is used to coordinate the training work of each computing node. The gradient merging module can be used to merge the gradient information of each computing node on the central node to obtain the average value of the gradient information, and finally the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed are used to update the model parameters based on the obtained average value of the gradient information. Through the above training process, parallel training of data and models can be achieved, and when facing large-scale data sets and complex models, huge computing resources and long training cycles are not required, thereby improving the training efficiency of large models.
[0075] In order to realize parallel training of data and further improve the training efficiency of the model, in one embodiment of the present invention, after receiving the training data sent by the data distribution module in step 101, the above embodiment further includes:
[0076] Based on the data parallel training mode of the deep learning framework Deepspeed, the training data is divided into at least one data block;
[0077] The at least one data block is sent to a corresponding computing node.
[0078] The using the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node includes:
[0079] The at least one data block is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node.
[0080] In an embodiment of the present invention, the training efficiency of the model can be further improved by implementing data parallel training. Based on this, the data distribution module can adopt the data parallelization technology of the deep learning framework Deepspeed to divide the training data into multiple data blocks and distribute them to different computing nodes for parallel processing.
[0081] In order to realize parallel training of the model and further improve the training efficiency of the model, in one embodiment of the present invention, step 102 in the above embodiment is based on the deep learning framework Deepspeed, and uses the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node, which may include the following steps:
[0082] Based on the model parallel training mode of the deep learning framework Deepspeed and the structure and computational complexity of the large model, the large model is divided into the at least one sub-model;
[0083] The at least one sub-model is trained in parallel using the at least one data block to generate gradient information of at least one computing node.
[0084] In an embodiment of the present invention, the training efficiency of the model can be further improved by implementing parallel model training. First, the model segmentation module can be used to segment the large model into multiple sub-models according to its structure and computational complexity, so that each sub-model can be trained on a computing node, and the gradient information can indicate in which direction the error increases or decreases under the current parameter value, and how much it should be adjusted. The model parameters can be updated accordingly to make the model's prediction closer to the true value, thereby reducing the error. Therefore, the gradient information of each computing node can be obtained for correcting the model.
[0085] Specifically, the following adjustments can be made in the software architecture: using the deep learning framework Deepspeed, and building a high-performance training technology base based on the 3D parallel training advantages of the Deepspeed framework.
[0086] 1. Use DeepSpeed's ZeRO (Zero Redundancy Optimizer) optimizer to optimize model parameters, optimizer state, and gradient storage in stages to reduce memory usage;
[0087] 2. Apply DeepSpeed’s Automatic Mixed Precision Training (AMP) technology to increase training speed and reduce video memory usage;
[0088] 3. Use DeepSpeed's communication optimization technology to reduce cross-node communication overhead;
[0089] 4. Enable data parallel training mode, split the training data into multiple subsets, each GPU processes one subset, and all GPUs calculate gradient information in parallel;
[0090] 5. Enable model parallel training mode to split the large model into multiple sub-models, with different GPUs responsible for the forward and backward calculations of different parts;
[0091] 6. Enable tensor parallel mode. Tensor parallel is a technology used for distributed training that splits and distributes the model's computing tasks across multiple GPU devices. Tensor parallelism is particularly suitable for large neural network models that cannot fit into the memory of a single device. By splitting computing tasks onto multiple devices, tensor parallelism can effectively utilize cluster resources and accelerate the training process.
[0092] Through the above configuration, data is distributed to different server GPU cards and training is performed in stages, which can effectively improve the training speed.
[0093] The specific configuration example is as follows:
[0094] "zero_optimization":{
[0095] "stage":2,#Number of parallel stages of the pipeline
[0096] "allgather_partitions":true,#Data parallelism, data is collected from multiple devices to one device
[0097] "allgather_bucket_size":5e8,
[0098] "reduce_scatter":true,#Scatter data from a tensor to multiple devices
[0099] "reduce_bucket_size":5e8,
[0100] "overlap_comm":false,
[0101] "contiguous_gradients":true}
[0102] 1. Build CUDA into the base image so that the container has the ability to schedule the host graphics card driver;
[0103] 2. Build the deep learning framework in a container image and run it as a container during training;
[0104] 3. Use the container orchestration framework to automatically schedule tasks and isolate resources between sub-models to prevent resource preemption and sub-model failure.
[0105] In order to coordinate the gradient information of each computing node, in one embodiment of the present invention, the step 103 in the above embodiment uses the gradient merging module to merge the gradient information of the at least one computing node on the central node to generate the average value of the gradient information, including:
[0106] According to the first formula and the gradient information of the at least one computing node, the gradient information of the at least one computing node is merged on the central node using a gradient merging module to generate a gradient information average value, wherein the first formula is:
[0107]
[0108] in, is used to represent the average value of the gradient information, X1 is used to represent the gradient information of the first computing node, X2 is used to represent the gradient information of the second computing node, and X n It is used to represent the gradient information of the nth computing node, and N is used to represent the number of computing nodes.
[0109] In an embodiment of the present invention, since the central node can coordinate the gradient information of each computing node, the gradient merging module can be used to merge the gradient information of multiple computing nodes on the central node to generate the gradient information average value. First, it is necessary to obtain the gradient information of each computing node and determine the gradient information average value through the above-mentioned first formula so as to adjust the model parameters based on the gradient information average value.
[0110] In order to achieve load balancing of each computing node, in one embodiment of the present invention, after updating the large model parameters based on the average value of the gradient information in step 104 using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed, the following is further included:
[0111] According to a preset load threshold, determining whether there is a first current computing node among the at least one computing node whose load is higher than the load threshold;
[0112] When it is determined that the load of the first current computing node among the at least one computing node is higher than the load threshold, determining a second current computing node with the smallest load among the at least one computing node;
[0113] The submodel of the first current computing node is assigned to the second current computing node.
[0114] In an embodiment of the present invention, in order to achieve load balancing of each computing node, the allocation of sub-models can be dynamically adjusted according to the load conditions of each computing node. When it is determined that the load of the first current computing node is higher than a preset load threshold, it indicates that the load of the current computing node is high, so the load of the current computing node can be reduced by allocating the sub-model of the current computing node to a computing node with a lower load for model training, thereby achieving load balancing of each computing node.
[0115] In the embodiment of the present invention, adjustments can also be made from the following network architecture, storage architecture and task scheduling architecture to further improve the training efficiency of the model. The specific adjustment methods are as follows:
[0116] 1. Network architecture: Frequent data exchange between computing server devices may lead to high communication overhead, affecting the overall training speed. Therefore, optimizing communication strategies and hardware architecture is also crucial to improving performance. High-speed IB networks can be used to reduce communication delays between computing servers and increase bandwidth. For example, IB network switches and IB network cables with a communication efficiency of more than 200G can be used to connect the GPU computing servers in the computing resource pool to the same IB network switch to reduce network losses caused by cross-network devices;
[0117] 2. Storage architecture: Use high-performance shared storage systems, such as distributed file systems or object storage, so that multiple machines can share training data and model parameters. There is no need for each computing node to redundantly store model weight files. At the same time, storage devices can use high-speed disk arrays such as NVMe SSD to store data to improve storage performance from a physical level.
[0118] 3. Task scheduling architecture: Use the container management tool Kubernetes to schedule and manage multiple sub-models and ensure the effective use of resources and the smooth execution of tasks. While improving resource utilization efficiency, it isolates resources between sub-models to avoid mutual preemption, and reasonably sets the steps for training to generate training results to avoid the loss of training results due to unexpected situations such as server failures and network failures.
[0119] In the embodiment of the present invention, since the large model training platform has the attribute of high load, real-time monitoring of the platform system is also particularly important. The health status and operation status of the platform can be grasped in real time through the monitoring system, and the platform log and task log can also be recorded. The platform log is used by the super administrator to record the problems of the service component itself, and the task log is used by the user who performs the training task to record the problems in the model training process. By recording the platform log and the task log, the accuracy of the model can be further improved.
[0120] For example, (1) you can use performance analysis tools, such as NVIDIA's Nsight Systems or TensorFlow Profiler, to identify and optimize bottlenecks in the training process; (2) you can use an integrated monitoring system to monitor the resource utilization, network traffic, and training progress of each node in the cluster in real time, and identify and solve problems in a timely manner; (3) you can use the mainstream monitoring components of the Kubernetes platform, prometheus and grfana, to build a monitoring system to collect platform resource and sub-model indicators in real time.
[0121] In the embodiment of the present invention, the solution greatly speeds up the training of large models through parallel computing of multiple machines and multiple cards; utilizes the resources of multiple computing nodes to effectively reduce hardware costs; has strong scalability and can easily increase or reduce computing nodes to adapt to model training of different scales; and at the same time, through technologies such as data parallelism and model parallelism, improves the training accuracy of large models.
[0122] like Figure 2 As shown, in order to more clearly illustrate the technical solution and advantages of the present invention, the following is a detailed description of a training method based on a multi-machine multi-card large model provided by an embodiment of the present invention, which may specifically include the following steps:
[0123] Step 201: receiving training data sent by a data distribution module;
[0124] Step 202: Based on the data parallel training mode of the deep learning framework Deepspeed, the training data is divided into at least one data block, and the at least one data block is sent to a corresponding computing node;
[0125] Step 203: Based on the model parallel training mode of the deep learning framework Deepspeed and the structure and computational complexity of the large model, the large model is divided into at least one sub-model, and at least one sub-model is trained in parallel using at least one data block to generate gradient information of at least one computing node, wherein each computing node includes at least one server, and each server is configured with at least one graphics card;
[0126] Specifically, the following adjustments can be made from the hardware architecture:
[0127] 1. Use multiple servers to build a computing cluster, and each server is equipped with multiple GPU cards to increase computing power;
[0128] 2. Choose high-performance server hardware, such as GPU servers with large page memory, high-speed processors, and PCIe4.0 or higher;
[0129] 3. Deploy the servers in the same computer room and connect them to the same switch.
[0130] Step 204: According to the first formula and the gradient information of at least one computing node, the gradient information of at least one computing node is merged on the central node using a gradient merging module to generate a gradient information average value, wherein the central node is used to coordinate the gradient information of at least one computing node;
[0131] Specifically, the first formula is:
[0132] in, It is used to represent the average value of gradient information, X1 is used to represent the gradient information of the first calculation node, X2 is used to represent the gradient information of the second calculation node, and X n It is used to represent the gradient information of the nth computing node, and N is used to represent the number of computing nodes;
[0133] Step 205: Based on the average value of the gradient information, the large model parameters are updated using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed;
[0134] Step 206: According to a preset load threshold, determine whether there is a first current computing node in at least one computing node whose load is higher than the load threshold;
[0135] Step 207: when it is determined that the load of the first current computing node among the at least one computing node is higher than the load threshold, determine a second current computing node with the smallest load among the at least one computing node;
[0136] Step 208: Allocate the sub-model of the first current computing node to the second current computing node.
[0137] like Figure 3 As shown, the embodiment of the present invention provides a training system based on a multi-machine multi-card large model, the system comprising: a data receiving module 301, a data processing module 302, a gradient merging module 303 and a parameter updating module 304;
[0138] The data receiving module 301 is used to receive the training data sent by the data distribution module;
[0139] The data processing module 302 is used to perform parallel training on at least one sub-model using the training data based on the deep learning framework Deepspeed, and generate gradient information of at least one computing node, wherein each computing node includes at least one server, and each server is configured with at least one graphics card;
[0140] The gradient merging module 303 is used to merge the gradient information of the at least one computing node on the central node to generate an average value of the gradient information, wherein the central node is used to coordinate the gradient information of the at least one computing node;
[0141] The parameter updating module 304 is used to update the large model parameters based on the average value of the gradient information using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed.
[0142] based on Figure 3 A training system based on a multi-machine multi-card large model is shown in FIG. Figure 4 As shown, in one embodiment of the present invention, after the data receiving module 301, it further includes: a data segmentation module 305;
[0143] The data segmentation module 305 is used to perform:
[0144] Based on the data parallel training mode of the deep learning framework Deepspeed, the training data is divided into at least one data block;
[0145] sending the at least one data block to a corresponding computing node;
[0146] The using the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node includes:
[0147] The at least one data block is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node.
[0148] like Figure 4 As shown, in one embodiment of the present invention, the data processing module 302 is further used to execute:
[0149] Based on the model parallel training mode of the deep learning framework Deepspeed and the structure and computational complexity of the large model, the large model is divided into the at least one sub-model;
[0150] The at least one sub-model is trained in parallel using the at least one data block to generate gradient information of at least one computing node.
[0151] like Figure 4 As shown, in one embodiment of the present invention, the gradient merging module is further used to execute:
[0152] According to the first formula and the gradient information of the at least one computing node, the gradient information of the at least one computing node is merged on the central node to generate a gradient information average value, wherein the first formula is:
[0153]
[0154] in, is used to represent the average value of the gradient information, X1 is used to represent the gradient information of the first computing node, X2 is used to represent the gradient information of the second computing node, and X n It is used to represent the gradient information of the nth computing node, and N is used to represent the number of computing nodes.
[0155] based on Figure 4 A training system based on a multi-machine multi-card large model is shown in FIG. Figure 5 As shown, in one embodiment of the present invention, after the parameter updating module, it further includes: a load balancing module 306;
[0156] The load balancing module 306 is used to execute:
[0157] According to a preset load threshold, determining whether there is a first current computing node among the at least one computing node whose load is higher than the load threshold;
[0158] When it is determined that the load of the first current computing node among the at least one computing node is higher than the load threshold, determining a second current computing node with the smallest load among the at least one computing node;
[0159] The submodel of the first current computing node is assigned to the second current computing node.
[0160] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a training system based on a multi-machine multi-card large model. In other embodiments of the present invention, a training system based on a multi-machine multi-card large model may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0161] The information interaction, execution process and other contents between the units in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.
[0162] The embodiment of the present invention also provides a training system based on a multi-machine multi-card large model, comprising: at least one memory and at least one processor;
[0163] at least one memory for storing a machine-readable program;
[0164] At least one processor is used to call a machine-readable program to execute a training method based on a multi-machine and multi-card large model in any embodiment of the present invention.
[0165] An embodiment of the present invention also provides a computer-readable medium on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes a training method based on a multi-machine and multi-card large model in any embodiment of the present invention.
[0166] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.
[0167] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
[0168] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.
[0169] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.
[0170] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.
[0171] Each embodiment of the present invention has at least the following beneficial effects:
[0172] 1. In an embodiment of the present invention, a training method based on a large model with multiple machines and multiple graphics cards is provided. The implementation of the method includes: configuring multiple servers on multiple computing nodes, and each server is configured with multiple graphics cards to form a multi-machine multi-card computing power cluster. The received training data is distributed to each computing node for processing through a data distribution module. At the same time, the large model is divided into multiple sub-models and assigned to different computing nodes for parallel training based on the deep learning framework Deepspeed. The central node is used to coordinate the training work of each computing node. The gradient merging module can be used to merge the gradient information of each computing node on the central node to obtain the average gradient information. Finally, the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed are used to update the model parameters based on the obtained average gradient information. Through the above training process, parallel training of data and models can be achieved. When facing large-scale data sets and complex models, there is no need for huge computing resources and long training cycles, thereby improving the training efficiency of large models;
[0173] 2. In the embodiment of the present invention, the training efficiency of the model can be further improved by implementing data parallel training. Based on this, the data distribution module can use the data parallelization technology of the deep learning framework Deepspeed to divide the training data into multiple data blocks and distribute them to different computing nodes for processing;
[0174] 3. In the embodiment of the present invention, the training efficiency of the model can be further improved by implementing parallel model training. First, the model segmentation module can be used to segment the large model into multiple sub-models according to its structure and computational complexity, so that each sub-model can be trained on a computing node. The gradient information can indicate in which direction the error increases or decreases under the current parameter value, and how much it should be adjusted. The model parameters can be updated accordingly to make the model prediction closer to the true value, thereby reducing the error. Therefore, the gradient information of each computing node can be obtained to correct the model.
[0175] It should be noted that not all steps and modules in the above-mentioned processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.
[0176] In the above embodiments, the hardware unit can be realized by mechanical means or electrical means. For example, a hardware unit can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operation. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete the corresponding operation. Concrete implementation (mechanical means or dedicated permanent circuits or temporarily set circuits) can be determined based on cost and time considerations.
[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A training method based on a multi-machine multi-card large model, characterized in that: The method includes: Receive training data sent by the data distribution module; Based on the deep learning framework Deepspeed, the training data is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node, wherein each computing node includes at least one server, and each server is configured with at least one graphics card; Using a gradient merging module to merge the gradient information of the at least one computing node on a central node to generate an average value of the gradient information, wherein the central node is used to coordinate the gradient information of the at least one computing node; Based on the average value of the gradient information, the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed are used to update the large model parameters.
2. The method according to claim 1, characterized in that After the training data is received from the data distribution module, the method further includes: Based on the data parallel training mode of the deep learning framework Deepspeed, the training data is divided into at least one data block; sending the at least one data block to a corresponding computing node; The using the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node includes: The at least one data block is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node.
3. The method according to claim 2, characterized in that Based on the deep learning framework Deepspeed, the at least one sub-model is trained in parallel using the training data to generate gradient information of at least one computing node, including: Based on the model parallel training mode of the deep learning framework Deepspeed and the structure and computational complexity of the large model, the large model is divided into the at least one sub-model; The at least one sub-model is trained in parallel using the at least one data block to generate gradient information of at least one computing node.
4. The method according to claim 1, characterized in that: The step of merging the gradient information of the at least one computing node on the central node using the gradient merging module to generate the average value of the gradient information includes: According to the first formula and the gradient information of the at least one computing node, the gradient information of the at least one computing node is merged on the central node using a gradient merging module to generate a gradient information average value, wherein the first formula is: in, is used to represent the average value of the gradient information, X1 is used to represent the gradient information of the first computing node, X2 is used to represent the gradient information of the second computing node, and X n It is used to represent the gradient information of the nth computing node, and N is used to represent the number of computing nodes; and / or, After the large model parameters are updated based on the average value of the gradient information using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed, the method further includes: According to a preset load threshold, determining whether there is a first current computing node among the at least one computing node whose load is higher than the load threshold; When it is determined that the load of the first current computing node among the at least one computing node is higher than the load threshold, determining a second current computing node with the smallest load among the at least one computing node; The submodel of the first current computing node is assigned to the second current computing node.
5. A training system based on a multi-machine multi-card large model, characterized in that: The system includes: a data receiving module, a data processing module, a gradient merging module and a parameter updating module; The data receiving module is used to receive the training data sent by the data distribution module; The data processing module is used to perform parallel training on at least one sub-model using the training data based on the deep learning framework Deepspeed, and generate gradient information of at least one computing node, wherein each computing node includes at least one server, and each server is configured with at least one graphics card; The gradient merging module is used to merge the gradient information of the at least one computing node on the central node to generate an average value of the gradient information, wherein the central node is used to coordinate the gradient information of the at least one computing node; The parameter updating module is used to update the large model parameters based on the average value of the gradient information using the optimizer module and the ZeRO optimizer of the deep learning framework Deepspeed.
6. The system according to claim 5, characterized in that After the data receiving module, it further includes: a data segmentation module; The data segmentation module is used to perform: Based on the data parallel training mode of the deep learning framework Deepspeed, the training data is divided into at least one data block; sending the at least one data block to a corresponding computing node; The using the training data to perform parallel training on at least one sub-model to generate gradient information of at least one computing node includes: The at least one data block is used to perform parallel training on at least one sub-model to generate gradient information of at least one computing node.
7. The system according to claim 6, characterized in that The data processing module is further used to execute: Based on the model parallel training mode of the deep learning framework Deepspeed and the structure and computational complexity of the large model, the large model is divided into the at least one sub-model; The at least one sub-model is trained in parallel using the at least one data block to generate gradient information of at least one computing node.
8. The system according to claim 5, characterized in that The gradient merging module is also used to execute: According to the first formula and the gradient information of the at least one computing node, the gradient information of the at least one computing node is merged on the central node to generate a gradient information average value, wherein the first formula is: in, is used to represent the average value of the gradient information, X1 is used to represent the gradient information of the first computing node, X2 is used to represent the gradient information of the second computing node, and X n It is used to represent the gradient information of the nth computing node, and N is used to represent the number of computing nodes; and / or, After the parameter updating module, it further includes: a load balancing module; The load balancing module is used to execute: According to a preset load threshold, determining whether there is a first current computing node among the at least one computing node whose load is higher than the load threshold; When it is determined that the load of the first current computing node among the at least one computing node is higher than the load threshold, determining a second current computing node with the smallest load among the at least one computing node; The submodel of the first current computing node is assigned to the second current computing node.
9. A training system based on a multi-machine multi-card large model, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 4.
10. A computer readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, cause the processor to perform any one of the methods of claims 1 to 4.