Model distributed parallel training method and device, electronic equipment and storage medium

Abstract

The invention provides a model distributed parallel training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: determining a hyper-parameter combination for training a target model in a distributed computing system; generating a particle swarm based on the hyper-parameter combination; the speed of each particle in the particle swarm is determined based on the value distribution probability of each hyper-parameter in the hyper-parameter combination; training the target model based on a distributed training execution script generated by each particle, updating the speed and position of each particle by taking the training throughput performance of the distributed computing system as the fitness value of each particle, and determining the optimal solution of the hyper-parameter combination; and on the basis of a distributed training execution script generated by the optimal solution of the hyper-parameter combination, continuing to train the target model. According to the method and the device provided by the invention, the training efficiency and the model performance of the model are improved in a distributed computing environment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model distributed parallel training method, device, electronic device and storage medium. Background Art

[0002] Large Language Model (LLM) distributed parallel training refers to training large models in a distributed computing environment. During the training process, the performance of the model depends not only on the training algorithm itself, but also on the selection of hyperparameters such as the distribution strategy.

[0003] Related technologies perform hyperparameter search through grid search, random search, dynamic programming and other methods, but these methods have many shortcomings in terms of search efficiency, search cost, search accuracy and resource consumption, resulting in low training efficiency of large models, poor model performance and high resource consumption.

[0004] Therefore, how to improve model training efficiency and model performance in a distributed computing environment has become a technical problem that needs to be urgently solved in the industry. Summary of the invention

[0005] The present invention provides a model distributed parallel training method, device, electronic device and storage medium, which are used to solve the technical problem of how to improve the training efficiency and model performance of the model in a distributed computing environment.

[0006] The present invention provides a model distributed parallel training method, comprising: Determine a combination of hyperparameters for training a target model in a distributed computing system; A particle swarm is generated based on the hyperparameter combination; a speed of each particle in the particle swarm is determined based on a value distribution probability of each hyperparameter in the hyperparameter combination; The target model is trained based on the distributed training execution script generated by each particle, the speed and position of each particle are updated with the training throughput performance of the distributed computing system as the fitness value of each particle, and the optimal solution of the hyperparameter combination is determined; Based on the distributed training execution script generated by the optimal solution of the hyperparameter combination, the target model continues to be trained.

[0007] In some embodiments, determining a hyperparameter combination for training a target model in a distributed computing system includes: Determining multiple parallel strategies for training the target model; The hyperparameter combination is determined based on the data segmentation dimensions and model segmentation dimensions corresponding to each parallel strategy.

[0008] In some embodiments, generating a particle swarm based on the hyperparameter combination includes: Determine a value set of each hyperparameter in the hyperparameter combination; Based on the value set, determining the value distribution probability of each hyperparameter; The particle swarm is generated by taking the value distribution probability of each hyperparameter as the speed of each particle.

[0009] In some embodiments, the distributed training execution script generated based on each particle trains the target model, updates the speed and position of each particle using the training throughput performance of the distributed computing system as the fitness value of each particle, and determines the optimal solution of the hyperparameter combination, including: Initialize the speed and position of each particle based on expert experience settings and / or random sampling generation; Generate a distributed training execution script based on each initialized particle, train the target model, determine the training throughput performance of the distributed computing system as the fitness value of each particle, and search for the individual optimal position and the global optimal position of each particle during the training process; When the search stop condition is not met, the individual optimal position and the global optimal position of each particle are updated based on the fitness value of each particle, and the speed of each particle is updated based on the individual optimal position and the global optimal position; When the search stop condition is met, the optimal solution of the hyperparameter combination is determined based on the position of each particle.

[0010] In some embodiments, the updating of the velocity of each particle based on the individual optimal position and the global optimal position includes: Based on the value distribution probability of each hyperparameter of each particle in the current search process, the value distribution probability of each hyperparameter of each particle at the individual optimal position, and the value distribution probability of each hyperparameter of each particle at the global optimal position, the speed of each particle, the speed of each particle at the individual optimal position, and the speed of each particle at the global optimal position are determined respectively; Determine a first velocity update amount for each particle based on the velocity of each particle and the velocity of each particle at the individual optimal position; Determine a second speed update amount of each particle based on the speed of each particle and the speed of each particle at the global optimal position; Determine the speed of each particle in the next search process based on the speed of each particle in the current search process, the first speed update amount, and the second speed update amount; Based on the speed of each particle in the next search process, the position of each particle in the next search process is updated.

[0011] In some embodiments, updating the position of each particle in the next search process based on the speed of each particle in the next search process includes: In the case where the positions of the particles are updated, determining the current training state of the target model based on the current model parameters of the target model, the training progress, and the state parameters of the optimizer in the distributed computing system; Save the current training state of the target model.

[0012] In some embodiments, after saving the current training state of the target model, the method further includes: Determine the value of each hyperparameter based on the updated position of each particle; Based on the values ​​of each hyperparameter, determining the data segmentation dimension and the model segmentation dimension in the next search process; Redistributing the training data of the target model based on the data segmentation dimension; The weights and layers of the target model are redistributed based on the model segmentation dimension.

[0013] The present invention provides a model distributed parallel training device, comprising: A parameter determination module, used to determine a hyperparameter combination for training a target model in a distributed computing system; A particle swarm generation module, used to generate a particle swarm based on the hyperparameter combination; the speed of each particle in the particle swarm is determined based on the value distribution probability of each hyperparameter in the hyperparameter combination; A particle swarm optimization module is used to train the target model based on the distributed training execution script generated by each particle, update the speed and position of each particle with the training throughput performance of the distributed computing system as the fitness value of each particle, and determine the optimal solution of the hyperparameter combination; The model training module is used to continue training the target model based on the distributed training execution script generated based on the optimal solution of the hyperparameter combination.

[0014] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the model distributed parallel training method when executing the computer program.

[0015] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the model distributed parallel training method is implemented.

[0016] The model distributed parallel training method, device, electronic device and storage medium provided by the present invention determine a hyperparameter combination for training a target model in a distributed computing system; generate a particle swarm based on the hyperparameter combination; train the target model based on a distributed training execution script generated by each particle, update the speed and position of each particle with the training throughput performance of the distributed computing system as the fitness value of each particle, and determine the optimal solution of the hyperparameter combination; continue to train the target model based on the distributed training execution script generated by the optimal solution of the hyperparameter combination; realize automatic search for optimal hyperparameters, greatly reduce manual intervention, and improve the efficiency of the search process, which not only optimizes each hyperparameter under parallel training, improves the training efficiency and model performance of the model in a distributed computing environment, but also speeds up the training speed of large models and improves the quality of the final model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the model distributed parallel training method provided by the present invention.

[0020] Figure 2 This is the second flow chart of the model distributed parallel training method provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the model distributed parallel training device provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units or modules is not necessarily limited to those steps or units or modules that are clearly listed, but may include other steps or units or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Different distributed parallel strategies will split the model or data in different dimensions, and introduce different communication modes during operation. The selection and combination of different distributed parallel strategies greatly affect the training performance and throughput. Factors affecting distributed performance include but are not limited to hardware architecture, node interconnection topology, and inter-node network topology. This poses a great challenge to the formulation of distributed training strategies. The selection of these strategies and parameters is crucial to the final performance of the model, and the manual adjustment method used in related technologies is not only time-consuming, but also difficult to find the optimal solution. There are also ways to perform automated parallel strategy search, which uses dynamic programming, grid search, random search and other methods to search in the strategy space, and conduct actual tests to find the optimal strategy. After finding it, the corresponding strategy script is used for training during actual training. These methods still have room for improvement in terms of efficiency, cost, accuracy and resource consumption.

[0026] In order to solve the shortcomings of related technologies, Figure 1 This is one of the flow charts of the distributed parallel training method of the model provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0027] Step 110: Determine a hyperparameter combination for training the target model in a distributed computing system.

[0028] Specifically, the executor of the model distributed parallel training method provided in the embodiment of the present invention is a model distributed parallel training device. The device can be implemented by software, such as a model distributed parallel training program; or by hardware, such as a computer or server that executes the model distributed parallel training method.

[0029] A distributed computing system is a system formed by multiple dispersed computers or servers connected through a network. These computers cooperate with each other to achieve one or more common goals, such as training or reasoning on a large language model.

[0030] The application scenario of the method provided in the embodiment of the present invention is to perform distributed training on a target model. The target model refers to a neural network model that needs to be trained, for example, various large language models.

[0031] For example, when a neural network model is used to perform tasks such as image processing, speech processing, or text processing, a distributed computing system can be used to train the neural network model. The neural network model can be trained using training data such as image samples, speech samples, or text samples.

[0032] Hyperparameters are parameters used to control the training process of the target model. Hyperparameters are variables that determine how the target model is trained. A hyperparameter combination is a set of multiple hyperparameters.

[0033] Step 120: Generate a particle swarm based on the hyperparameter combination; the speed of each particle in the particle swarm is determined based on the value distribution probability of each hyperparameter in the hyperparameter combination.

[0034] Specifically, Dynamic Particle Swarm Optimization (DPSO) is a swarm intelligence optimization algorithm inspired by the collective foraging behavior of bird flocks or fish swarms in nature. It is used to solve complex optimization problems by simulating the collaboration and information sharing mechanism of biological groups. Compared with traditional particle swarm optimization algorithms, DPSO performs better in dynamic environments and can respond to environmental changes in real time and adjust the optimization strategy, thereby improving search efficiency and global optimization capabilities.

[0035] The embodiment of the present invention can use a dynamic particle swarm algorithm to search for possible values ​​of hyperparameters to obtain an optimal solution. Each particle in the particle swarm can represent a potential solution, that is, a set of possible hyperparameter values.

[0036] A particle swarm can be constructed based on a combination of hyperparameters. Since the values ​​of hyperparameters are discrete and the position changes of particles in the search space show a jumping characteristic, a probability distribution is used to represent the speed of each particle. The value distribution probability is used to describe the possible values ​​of a hyperparameter (random variable) and its corresponding probability.

[0037] The speed of each particle in the particle swarm can be defined as the distribution probability of the value of each hyperparameter. The speed of each particle is no longer a speed vector in the traditional sense, but is transformed into a set of probability distributions, each of which corresponds to a possible value of a hyperparameter.

[0038] Step 130: Train the target model based on the distributed training execution script generated by each particle, update the speed and position of each particle using the training throughput performance of the distributed computing system as the fitness value of each particle, and determine the optimal solution for the hyperparameter combination.

[0039] Specifically, each particle can be initialized to determine the initial speed and initial position of each particle. Each particle represents a potential hyperparameter combination. The position of the particle is the specific value of each hyperparameter in the hyperparameter combination. The speed of the particle is the distribution probability of the value of each hyperparameter in the hyperparameter combination.

[0040] Based on each particle, a distributed training execution script can be generated, that is, the particle position (i.e., hyperparameter combination) is mapped to the actual command line parameters or configuration file to control the distributed computing system to execute the training of the target model.

[0041] Training throughput performance refers to the system's ability to process data during distributed training. Specifically, it measures the total number of requests or data that the system can handle in a certain period of time. In the training of large language models, the TGS (tokens per GPU per second) metric can be used to evaluate training throughput performance. This metric can measure the inference performance of large language models on a distributed computing system or a single graphics processing unit (GPU). This metric specifically indicates the number of tokens that a graphics processor can process per unit time.

[0042] The fitness value refers to the value of the objective function in the particle swarm algorithm, which is used to quantify the performance of each particle in the problem space. The fitness value not only affects the particle speed update, but also directly affects the particle position update. Since the particle speed is dynamically adjusted according to its current position, individual optimal position and global optimal position, and the individual optimal position and global optimal position are determined according to the fitness value, the quality of the fitness value directly determines the next moving direction and distance of the particle.

[0043] After initializing the position and speed of each particle in the particle swarm, calculate the fitness value of each particle, update the historical optimal position of each particle in the particle swarm, update the speed and position of each particle in the particle swarm, and adjust the speed and position of the particle according to the dynamic adjustment strategy; repeat the above steps until the search stop condition is met and the optimal solution of the hyperparameter combination is obtained.

[0044] The values ​​of each hyperparameter in the optimal solution of the hyperparameter combination can optimize the training throughput performance of the distributed computing system.

[0045] Step 140: Continue training the target model based on the distributed training execution script generated based on the optimal solution of the hyperparameter combination.

[0046] Specifically, the training state of the target model when the optimal solution of the hyperparameter combination is obtained can be saved. Thereafter, the target model is trained continuously according to the distributed training execution script generated by the optimal solution, and the possible values ​​of the hyperparameters are no longer searched.

[0047] The model distributed parallel training method provided by the embodiment of the present invention determines a hyperparameter combination for training a target model in a distributed computing system; generates a particle swarm based on the hyperparameter combination; trains the target model based on a distributed training execution script generated by each particle, updates the speed and position of each particle with the training throughput performance of the distributed computing system as the fitness value of each particle, and determines the optimal solution of the hyperparameter combination; continues to train the target model based on the distributed training execution script generated by the optimal solution of the hyperparameter combination; realizes automated search for optimal hyperparameters, greatly reduces manual intervention, improves the efficiency of the search process, not only optimizes each hyperparameter under parallel training, improves the training efficiency and model performance of the model in a distributed computing environment, but also accelerates the training speed of large models and improves the quality of the final model.

[0048] It should be noted that each implementation of the present invention can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0049] In some embodiments, determining a hyperparameter combination for training a target model in a distributed computing system includes: Determine multiple parallel strategies for training the target model; Determine the hyperparameter combination based on the data segmentation dimensions and model segmentation dimensions corresponding to each parallel strategy.

[0050] Specifically, when training the target model, multiple parallel strategies can be determined. A parallel strategy refers to a strategy of splitting a task into multiple subtasks and executing them simultaneously on multiple processors or computing nodes in order to improve computing efficiency and speed when processing a computing task.

[0051] In the embodiment of the present invention, the parallel strategies may include tensor parallelism, pipeline parallelism, data parallelism, sequence parallelism, etc.

[0052] The data split dimension refers to the hyperparameter that splits the training data into multiple batches. These batches are assigned to different computing units to train the target model. The model split dimension refers to the hyperparameter that splits the target model into multiple stages (each stage contains one or more layers) according to the layer structure. These stages are mapped to different computing units to achieve parallel processing of the model.

[0053] After determining multiple parallel strategies, the data segmentation dimensions and model segmentation dimensions corresponding to each parallel strategy can be determined to generate a hyperparameter combination.

[0054] For example, the hyperparameter combination can be defined as [dimtp, dimpp, mbsize, recompute, recomputelayers, distoptimizer, overlap]. dimtp represents the tensor parallel dimension, and the value set can be {1, 2, ..., 16}. dimpp represents the pipeline parallel dimension, and the value set can be {1, 2, ..., 16}. mbsize represents the micro batch size parallel dimension, and the value set can be {1, 2, ..., 128}. recompute represents whether recomputation needs to be enabled, and the value set can be {0, 1}. recomputelayers represents the number of model layers that need to be recomputed, and the value set can be {1, 2, ..., the total number of model layers}. distoptimizer represents whether the distributed optimizer needs to be enabled, and the value set can be {0, 1}. overlap represents whether communication and reverse calculation need to be enabled, and the value set can be {0, 1}.

[0055] The model distributed parallel training method provided in the embodiment of the present invention determines the hyperparameter combination according to the data segmentation dimensions and model segmentation dimensions corresponding to each parallel strategy, which can improve the accuracy of hyperparameter selection and improve the training efficiency and model performance of the model.

[0056] In some embodiments, generating a particle swarm based on a combination of hyperparameters includes: Determine the value set of each hyperparameter in the hyperparameter combination; Based on the value set, determine the value distribution probability of each hyperparameter; Generate a particle swarm by taking the value distribution probability of each hyperparameter as the speed of each particle.

[0057] Specifically, the value set of each hyperparameter in the hyperparameter combination can be determined first. For example, the tensor parallel dimension is determined as a hyperparameter, and its value set can be expressed as {1, 2, ..., 16}. Through methods such as probability distribution fitting or probability density function estimation, the possible values ​​of the hyperparameter can be converted into the value distribution probability of the hyperparameter according to the value set.

[0058] For the particle swarm The velocity of a particle can be expressed as a vector: .in, Indicates The particle in The probability distribution of the values ​​of the hyperparameters, that is, the speed; ; is the total number of hyperparameters.

[0059] The model distributed parallel training method provided by the embodiment of the present invention uses the value distribution probability of each hyperparameter as the speed of each particle, and can guide the particles to search for the optimal value in the value set of the hyperparameter.

[0060] In some embodiments, the target model is trained based on the distributed training execution script generated by each particle, the speed and position of each particle are updated with the training throughput performance of the distributed computing system as the fitness value of each particle, and the optimal solution of the hyperparameter combination is determined, including: Initialize the speed and position of each particle based on expert experience settings and / or random sampling generation; Generate a distributed training execution script based on each initialized particle, train the target model, determine the training throughput performance of the distributed computing system as the fitness value of each particle, and search for the individual optimal position and global optimal position of each particle during the training process; When the search stop condition is not met, the individual optimal position and the global optimal position of each particle are updated based on the fitness value of each particle, and the speed and position of each particle are updated based on the individual optimal position and the global optimal position; When the search stop condition is met, the optimal solution of the hyperparameter combination is determined based on the position of each particle.

[0061] Specifically, the speed and position of each particle can be initialized by expert experience setting and random sampling generation. Expert experience setting means setting the speed and position of some particles according to expert experience, which can give a higher probability of selection; random sampling generation means randomly sampling and generating the speed and position of each particle, so that the values ​​are evenly distributed, which can increase the diversity of values.

[0062] For example, the position of a particle can be expressed as: position = [dimtp, dimpp, mbsize, recompute, recomputelayers, distoptimizer, overlap]. Position initialization is to determine the initial value for each hyperparameter.

[0063] For each particle, initialize its velocity vector, where each element is a probability distribution, representing the probability of the possible value of the corresponding hyperparameter: .

[0064] where each element is a probability distribution, such as It is The speed (value distribution probability) of a particle on the hyperparameter dimtp indicates the probability of choosing each possible value in dimtp.

[0065] Generate a distributed training execution script based on each initialized particle to train the target model, that is, map the particle position (i.e., hyperparameter combination) to the actual command line parameters or configuration file. Save the execution command with the corresponding parameters to the script file, and the executor will schedule the execution of the training.

[0066] During the training process, the training throughput performance of the distributed computing system is monitored in real time and used as the fitness value of each particle. At the same time, the individual optimal position and global optimal position of each particle are searched.

[0067] Each particle in the particle swarm algorithm can have an individual optimal position (pbest) and a global optimal position (gbest). The individual optimal position is the optimal solution found by the particle itself in history, corresponding to the best configuration parameters in the particle's history. The global optimal position is the optimal solution found by all particles in the entire particle swarm, corresponding to the configuration parameters with the best performance in the world.

[0068] The training of the target model and the search for the optimal solution are carried out simultaneously. During the training of the target model, the training throughput performance of the distributed computing system is monitored, and the speed and position of each particle are updated as the fitness value. The update of the speed and position of each particle causes the training hyperparameters to change, and the change of the hyperparameters (representing the change of the parallel strategy) causes the training throughput performance to change, and so on, while training and searching.

[0069] The search stop condition means that when certain specific conditions are met during the operation of the particle swarm algorithm, the algorithm will stop the iterative search and output the currently found optimal solution.

[0070] When the search stop condition is not met, the individual optimal position and global optimal position of each particle can be updated according to the fitness value of each particle. If the performance of the current position is better than the individual optimal, the individual optimal is updated; if it is better than the global optimal, the global optimal is updated. The specific implementation is to save the distribution probability of each hyperparameter of the corresponding particle. Then, the speed and position of each particle are updated according to the individual optimal position and the global optimal position.

[0071] When the search stop condition is met, the optimal solution of the hyperparameter combination is determined based on the position of each particle. The position here is the global optimal position of the particle.

[0072] The model distributed parallel training method provided by the embodiment of the present invention updates the speed and position of each particle with the training throughput performance of the distributed computing system as the fitness value of each particle, determines the optimal solution of the hyperparameter combination, and can quickly adjust the strategy according to real-time feedback, so that the system can quickly respond to performance changes and continuously optimize the training process. Compared with traditional grid search and random search, the method of the embodiment of the present invention is more outstanding in search efficiency and effect, and reduces the computing resources and time required to achieve the same performance level. The characteristics of the particle swarm optimization algorithm make the present invention not easy to fall into the local optimum during the search process, effectively reducing the risk of overfitting.

[0073] In some embodiments, updating the velocity of each particle based on the individual optimal position and the global optimal position includes: Based on the value distribution probability of each hyperparameter of each particle in the current search process, the value distribution probability of each hyperparameter of each particle at the individual optimal position, and the value distribution probability of each hyperparameter of each particle at the global optimal position, the speed of each particle, the speed of each particle at the individual optimal position, and the speed of each particle at the global optimal position are determined respectively; Determine a first velocity update amount for each particle based on the velocity of each particle and the velocity of each particle at the individual optimal position; Determine a second speed update amount of each particle based on the speed of each particle and the speed of each particle at the global optimal position; Determine the speed of each particle in the next search process based on the speed of each particle in the current search process, the first speed update amount, and the second speed update amount; Based on the velocity of each particle in the next search process, the position of each particle in the next search process is updated.

[0074] Specifically, the particle speed and position in the particle swarm can be dynamically adjusted according to the monitored training throughput performance to optimize the process of searching for hyperparameters.

[0075] In the current search process, the speed of each particle, the speed of each particle at the individual optimal position, and the speed of each particle at the global optimal position can be determined respectively according to the value distribution probability of each hyperparameter of each particle, the value distribution probability of each hyperparameter of each particle at the individual optimal position, and the value distribution probability of each hyperparameter of each particle at the global optimal position.

[0076] According to the speed of each particle and the speed of each particle at the individual optimal position, the first speed update of each particle is determined; according to the speed of each particle and the speed of each particle at the global optimal position, the second speed update of each particle is determined; according to the speed of each particle in the current search process, the first speed update and the second speed update, the speed of each particle in the next search process is determined. In the above process, the particle speed update is to update the value distribution probability in each dimension, which can be expressed as: .

[0077] in, For the current search process Middle The particle in The probability distribution of values ​​on hyperparameters; For the current search process Middle The particle is in the best position for each The probability distribution of values ​​on hyperparameters; For the current search process Middle The particle is at the global optimal position. The probability distribution of values ​​on hyperparameters; For the next search process Middle The particle in The probability distribution of values ​​on hyperparameters; and are the calculation weights of the first speed update amount and the second speed update amount, respectively, and are used to control the influence of the individual optimal position and the global optimal position on the value distribution probability of the hyperparameter.

[0078] According to the speed of each particle in the next search process, the position of each particle in the next search process can be updated.

[0079] The model distributed parallel training method provided by the embodiment of the present invention updates the speed of each particle according to the individual optimal position and the global optimal position, can quickly adjust the strategy according to real-time feedback, so that the system can quickly respond to performance changes and continuously optimize the training process.

[0080] In some embodiments, updating the position of each particle in the next search process based on the speed of each particle in the next search process includes: In the case where the positions of the particles are updated, the current training state of the target model is determined based on the current model parameters of the target model, the training progress, and the state parameters of the optimizer in the distributed computing system; Save the current training state of the target model.

[0081] Specifically, the position of each particle is updated, indicating that the value of the hyperparameter has changed, which means that the parallel strategy has been adjusted. At this time, the current model parameters of the target model, the training progress, and the state parameters of the optimizer in the distributed computing system can be saved as the current training state (checkpoint) of the target model.

[0082] The purpose of saving the current training state of the target model is to be able to restore the training state of the target model at any time.

[0083] After determining the optimal solution of the hyperparameter combination, the target model can be continuously trained based on the training state at the time of determining the optimal solution until a predetermined training cycle is reached.

[0084] The model distributed parallel training method provided by the embodiment of the present invention saves the current training state of the target model when the parallel strategy is adjusted, which is conducive to continuing to train the target model, avoiding the need to start training from scratch, and saving time and computing resources.

[0085] In some embodiments, after saving the current training state of the target model, the method further includes: Determine the value of each hyperparameter based on the updated position of each particle; Based on the values ​​of each hyperparameter, determine the data segmentation dimension and model segmentation dimension in the next search process; Redistribute the training data of the target model based on the data segmentation dimension; Redistribute the weights and layers of the target model based on the model split dimension.

[0086] Specifically, in a large-scale distributed deep learning training environment, adjustments to parallel strategies, especially changes involving tensor parallel (TP) dimensions or pipeline parallel (PP) dimensions, usually require redistribution of training data to ensure compatibility with the new parallel architecture. This process is called dynamic re-sharding. In an embodiment of the present invention, it mainly includes: The values ​​of each hyperparameter are determined according to the updated positions of each particle, and then the data segmentation dimension and model segmentation dimension in the next search process are determined.

[0087] The training data of the target model is redistributed according to the data segmentation dimension so that the training data can be distributed to each computing unit in the system; the weights and layers of the target model are redistributed according to the model segmentation dimension so that different weights or different layers in the target model are distributed to different computing nodes for processing.

[0088] The method provided in the embodiment of the present invention combines dynamic resegmentation with continued model training, mainly including: Step 1: State preservation. Before changing the parallel strategy (the value of the hyperparameter changes, the position of the corresponding particle changes), the system must save the current training state, including model parameters, optimizer state, and training progress. This can be achieved through the checkpoint mechanism to ensure that it can be restored from a known training state.

[0089] Step 2: Data loader adaptation. Reconfigure the data loader according to the new parallel strategy (update of hyperparameter values) to ensure that data can be evenly and effectively distributed to each computing node. This may involve using Distributed Sampler or other similar tools to dynamically adjust the data segmentation strategy.

[0090] Step 3: Redistribution of models and layers. The weights and layers of the model need to be redistributed according to the new TP or PP dimensions. This may involve changing the physical layout of the model to adapt to the new parallel architecture to optimize communication and computational efficiency.

[0091] Step 4: Training recovery: Use the saved checkpoint to restore the state of the model and optimizer to ensure seamless continuation of training from the change point.

[0092] The model distributed parallel training method provided by the embodiment of the present invention reallocates the training data of the target model and reallocates the weights and layers of the target model when the parallel strategy is adjusted, thereby enhancing the robustness of the system and ensuring rapid recovery when the training process is interrupted.

[0093] Figure 2 This is the second flow chart of the distributed parallel training method of the model provided by the present invention. Figure 2 As shown, the method includes: Step 210: Determine a possible value set for each hyperparameter that needs to be searched for training the large model.

[0094] You can select distributed parallel strategies and other important hyperparameters, such as tensor parallelism (TP) dimension, pipeline parallelism (PP) dimension, sequence parallelism (SP) switch, micro-batch (MB) size, global batch (GB) size, whether to enable recompute, etc. as hyperparameters to be optimized.

[0095] Step 220: Create a particle swarm and define the speed and position of each particle.

[0096] Given that the values ​​of hyperparameters are discrete and the position changes of particles in the search space show a jumping characteristic, probability distribution is used to represent the particle speed of each hyperparameter. In DPSO, the speed of each particle is no longer a speed vector in the traditional sense, but is converted into a set of probability distributions, each of which corresponds to a possible value of a hyperparameter.

[0097] The velocity vector of a particle defines the probability of selecting a specific value of each hyperparameter in the next iteration. The velocity of each particle is a probability distribution that determines the probability of selecting the corresponding hyperparameter value in the next iteration. These probability distributions can be updated based on the individual optimal position of the particle and the global optimal position to guide the exploration and exploitation of the particle in the search space.

[0098] Step 230: Initialize the particle swarm.

[0099] The initial particle swarm consists of two parts: one part is based on particles set by expert experience, and the other part is generated by random sampling. This initialization method combines the advantages of expert knowledge and random exploration, providing a diverse starting point for the subsequent optimization process. The particle swarm consists of multiple particles, each of which represents a potential hyperparameter combination. The position and velocity of the particle are initialized.

[0100] Step 240: Train the target model.

[0101] Traverse the current particle swarm, generate a distributed training execution script for each particle, and start model training. Map the particle position (i.e., hyperparameter combination) to the actual command line parameter or configuration file.

[0102] Save the execution command with the corresponding parameters to the script file, and schedule the execution of the training by the executor. During the training process, the performance of the current parallel strategy is evaluated in real time. This step is the key to ensure the effectiveness of the strategy. In model training, the TGS indicator is used to evaluate the performance, and the optimal position of the particle is obtained by comparing this performance value.

[0103] Step 250: Dynamically adjust the speed and position of particles in the particle swarm.

[0104] According to the evaluation results, the particle positions in the particle swarm are dynamically adjusted to optimize the parallel strategy and hyperparameters. The DPSO algorithm can maintain efficient optimization capabilities in a dynamic environment by adaptively adjusting the speed and position of particles. The particle speed update is to update the probability distribution in each dimension.

[0105] With the new probability distribution, a new value is randomly selected based on the given probability distribution and the set of possible values. This function uses the np.random.choice method, which allows for random selection based on the provided probability distribution. This function is then called for each hyperparameter, updating the particle's position. This process ensures that the particle's position is dynamically adjusted based on the updated probability distribution, allowing the Particle Swarm Optimizer to explore the hyperparameter space more efficiently.

[0106] Step 260: When the parallel strategy is adjusted, the training data is redistributed to ensure compatibility with the new parallel architecture while saving the training status of the model.

[0107] Each time the strategy is adjusted, the model saving module is used to save the checkpoint of the current training. This step is crucial to ensure the continuity and stability of training, and also provides a guarantee for possible fault recovery.

[0108] Step 270: Determine the optimal solution for the hyperparameter combination.

[0109] Repeat the above steps, and gradually approach the global optimal solution through continuous adjustment and optimization. Determine the optimal solution by searching for the termination condition.

[0110] The termination conditions of the search can be as follows: Maximum number of iterations: Set a maximum number of iterations, when this number is reached, the algorithm stops searching. This prevents the algorithm from running indefinitely.

[0111] Performance threshold: Set a performance threshold. When the performance of the solution found reaches or exceeds this threshold, the algorithm stops. For example, setting TGS>300 will terminate the search early.

[0112] Improvement margin: Set an improvement margin threshold. When the performance improvement of the optimal solution in multiple consecutive iterations is less than this threshold, the algorithm stops.

[0113] In practical applications, you may need to combine multiple termination conditions to ensure that the algorithm can effectively search the solution space and stop in a reasonable time. For example, you can set a maximum number of iterations and a performance threshold at the same time to ensure that the algorithm stops when the performance requirement or the iteration limit is reached.

[0114] Step 280: Continue training the model according to the optimal solution of the hyperparameter combination until the training is completed.

[0115] Based on the optimal policy parameters found, continue training from the most recent checkpoint until the predetermined training cycle is reached. This step ensures that the model can complete training under the optimal policy and thus achieve the best performance.

[0116] The model distributed parallel training method provided in the embodiment of the present invention can automatically search for the optimal distributed parallel strategy and hyperparameter combination, solve some key challenges in the prior art, and improve the efficiency and performance of large model training; it has dynamic adjustment capabilities and global optimization characteristics, has significant advantages in a distributed training environment, can effectively cope with environmental changes and complexity, and achieve optimal utilization of resources.

[0117] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below and the method described above can refer to each other.

[0118] Figure 3 Schematic diagram of the structure of the model distributed parallel training device provided by the present invention. Figure 3 As shown, the device comprises: A parameter determination module 310, for determining a hyperparameter combination for training a target model in a distributed computing system; The particle swarm generation module 320 is used to generate a particle swarm based on a hyperparameter combination; the speed of each particle in the particle swarm is determined based on the value distribution probability of each hyperparameter in the hyperparameter combination; The particle swarm optimization module 330 is used to train the target model based on the distributed training execution script generated by each particle, update the speed and position of each particle with the training throughput performance of the distributed computing system as the fitness value of each particle, and determine the optimal solution of the hyperparameter combination; The model training module 340 is used to generate a distributed training execution script based on the optimal solution of the hyperparameter combination to continue training the target model.

[0119] The model distributed parallel training device provided by the embodiment of the present invention determines a hyperparameter combination for training a target model in a distributed computing system; generates a particle swarm based on the hyperparameter combination; trains the target model based on a distributed training execution script generated by each particle, updates the speed and position of each particle with the training throughput performance of the distributed computing system as the fitness value of each particle, and determines the optimal solution of the hyperparameter combination; continues to train the target model based on the distributed training execution script generated by the optimal solution of the hyperparameter combination; realizes automated search for optimal hyperparameters, greatly reduces manual intervention, and improves the efficiency of the search process, which not only optimizes each hyperparameter under parallel training, improves the training efficiency and model performance of the model in a distributed computing environment, but also speeds up the training speed of large models and improves the quality of the final model.

[0120] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor (Processor) 410, a communication interface (Communications Interface) 420, a memory (Memory) 430 and a communication bus (Communications Bus) 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic command in the memory 430 to execute the method described in the above embodiment, for example: Determine a hyperparameter combination for training a target model in a distributed computing system; generate a particle swarm based on the hyperparameter combination; the speed of each particle in the particle swarm is determined based on the value distribution probability of each hyperparameter in the hyperparameter combination; train the target model based on the distributed training execution script generated for each particle, update the speed and position of each particle using the training throughput performance of the distributed computing system as the fitness value of each particle, and determine the optimal solution of the hyperparameter combination; continue to train the target model based on the distributed training execution script generated based on the optimal solution of the hyperparameter combination.

[0121] In addition, the logic commands in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0122] The processor in the electronic device provided in the embodiment of the present invention can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0123] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0124] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0125] An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described above is implemented.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed parallel model training method, characterized in that: include: Determine a combination of hyperparameters for training a target model in a distributed computing system; A particle swarm is generated based on the hyperparameter combination; a speed of each particle in the particle swarm is determined based on a value distribution probability of each hyperparameter in the hyperparameter combination; The target model is trained based on the distributed training execution script generated by each particle, the speed and position of each particle are updated with the training throughput performance of the distributed computing system as the fitness value of each particle, and the optimal solution of the hyperparameter combination is determined; Based on the distributed training execution script generated by the optimal solution of the hyperparameter combination, the target model continues to be trained.

2. The model distributed parallel training method according to claim 1, characterized in that: Determining a hyperparameter combination for training a target model in a distributed computing system includes: Determining multiple parallel strategies for training the target model; The hyperparameter combination is determined based on the data segmentation dimensions and model segmentation dimensions corresponding to each parallel strategy.

3. The model distributed parallel training method according to claim 1, characterized in that: The generating a particle swarm based on the hyperparameter combination comprises: Determine a value set of each hyperparameter in the hyperparameter combination; Based on the value set, determining the value distribution probability of each hyperparameter; The particle swarm is generated by taking the value distribution probability of each hyperparameter as the speed of each particle.

4. The model distributed parallel training method according to claim 1, characterized in that: The target model is trained based on the distributed training execution script generated by each particle, the speed and position of each particle are updated with the training throughput performance of the distributed computing system as the fitness value of each particle, and the optimal solution of the hyperparameter combination is determined, including: Initialize the speed and position of each particle based on expert experience settings and / or random sampling generation; Generate a distributed training execution script based on each initialized particle, train the target model, determine the training throughput performance of the distributed computing system as the fitness value of each particle, and search for the individual optimal position and the global optimal position of each particle during the training process; When the search stop condition is not met, the individual optimal position and the global optimal position of each particle are updated based on the fitness value of each particle, and the speed of each particle is updated based on the individual optimal position and the global optimal position; When the search stop condition is met, the optimal solution of the hyperparameter combination is determined based on the position of each particle.

5. The model distributed parallel training method according to claim 4, characterized in that: The updating of the speed of each particle based on the individual optimal position and the global optimal position comprises: Based on the value distribution probability of each hyperparameter of each particle in the current search process, the value distribution probability of each hyperparameter of each particle at the individual optimal position, and the value distribution probability of each hyperparameter of each particle at the global optimal position, the speed of each particle, the speed of each particle at the individual optimal position, and the speed of each particle at the global optimal position are determined respectively; Determine a first velocity update amount for each particle based on the velocity of each particle and the velocity of each particle at the individual optimal position; Determine a second speed update amount of each particle based on the speed of each particle and the speed of each particle at the global optimal position; Determine the speed of each particle in the next search process based on the speed of each particle in the current search process, the first speed update amount, and the second speed update amount; Based on the speed of each particle in the next search process, the position of each particle in the next search process is updated.

6. The model distributed parallel training method according to claim 5, characterized in that: The updating of the position of each particle in the next search process based on the speed of each particle in the next search process comprises: In the case where the positions of the particles are updated, determining the current training state of the target model based on the current model parameters of the target model, the training progress, and the state parameters of the optimizer in the distributed computing system; Save the current training state of the target model.

7. The model distributed parallel training method according to claim 6, characterized in that: After saving the current training state of the target model, the method further includes: Determine the value of each hyperparameter based on the updated position of each particle; Based on the values ​​of each hyperparameter, determining the data segmentation dimension and the model segmentation dimension in the next search process; Redistributing the training data of the target model based on the data segmentation dimension; The weights and layers of the target model are redistributed based on the model segmentation dimension.

8. A model distributed parallel training device, characterized in that: include: A parameter determination module, used to determine a hyperparameter combination for training a target model in a distributed computing system; A particle swarm generation module, used to generate a particle swarm based on the hyperparameter combination; the speed of each particle in the particle swarm is determined based on the value distribution probability of each hyperparameter in the hyperparameter combination; A particle swarm optimization module is used to train the target model based on the distributed training execution script generated by each particle, update the speed and position of each particle with the training throughput performance of the distributed computing system as the fitness value of each particle, and determine the optimal solution of the hyperparameter combination; The model training module is used to continue training the target model based on the distributed training execution script generated based on the optimal solution of the hyperparameter combination.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model distributed parallel training method described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model distributed parallel training method described in any one of claims 1 to 7 is implemented.

Images