Large language model data parallel training method and system based on computing power of computing equipment

Abstract

The invention discloses a large language model data parallel training method and system based on computing power of computing devices, and the method comprises the steps: S1, carrying out the statistics of the computing power values of all computing devices in a training cluster, and forming a statistical set D: D = (C1: m1, C2: m2,..., Cn: mn); ci represents the computing power value of the ith computing device, mi represents the number of the ith computing device, and n is the total number of computing device types; s2, calculating the computing power proportion Pi: # imgabs0 # of the computing power of each type of computing equipment in the total computing power, wherein j is a summation index and is used for traversing all computing equipment types; s3, according to the calculation power proportion Pi, calculating the number Ni of training data samples allocated to each type of calculation equipment: # imgabs1 # # imgabs2 #, and N is the total number of training data samples; s4, calculating the number nij: # imgabs3 # of training data sample subsets allocated to each computing device; and S5, allocating the training data to each computing device for parallel training according to the calculated nij. According to the method, the utilization efficiency of the computing equipment in data parallel training can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a method and system for parallel training of large language model data based on the computing power of a computing device, so as to improve the parallel training efficiency of the large language model. Background Art

[0002] With the rapid growth of the number of parameters and the amount of training data required for large language models, the limited resources of a single computing device can no longer meet the needs of large language model training. A common solution is to use a distributed training system to decompose the training task of a large language model into multiple subtasks, and assign these subtasks to multiple computing devices for synchronous or asynchronous training, and finally merge the training results of each computing device to obtain the same computing effect as a single device, thereby effectively solving the resource bottleneck and accelerating the training process of large language models.

[0003] In distributed training systems, data parallelism is a common method that assigns a subset of training data to each computing device and performs forward computation training of a large language model based on the subset.

[0004] In common data parallel training strategies, it is usually assumed that all computing devices have the same computing power and memory capacity, and the total number of training data samples N is evenly distributed to M computing devices, so that each device obtains N / M data samples for processing. However, in the actual training process, the computing devices in the cluster often have differences in computing power and memory size. If the training data is evenly distributed to all devices indiscriminately, it may cause the computing of low-computing power devices to be overloaded, while the resources of high-computing power devices are not fully utilized. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data parallel training method and system for a large language model based on the computing power of a computing device, so as to allocate training data subsets of corresponding sizes to devices of different computing power levels, thereby significantly improving the utilization efficiency of the computing device in data parallel training.

[0006] To achieve the above objectives, in a first aspect, a large language model data parallel training method based on computing power of a computing device is provided, which comprises the following steps:

[0007] S1: Count the computing power values ​​of all computing devices in the training cluster to form a statistical set D:

[0008] D=(C1: m1, C2: m2,..., Cn: mn); C i

[0009] Represents the computing power value of the i-th computing device, mi represents the number of computing devices of type i, and n is the total number of computing device types;

[0010] S2: Calculate the proportion of computing power of each type of computing device to the total computing power Pi:

[0011] Where j is the sum index, and the value range of j is 1 to n, which is used to traverse all computing device types;

[0012] S3: Calculate the number of training data samples N allocated to each type of computing device according to the computing power ratio Pi i :

[0013]

[0014] Where N is the total number of training data samples;

[0015] S4: Calculate the number of training data sample subsets n assigned to each computing device ij :

[0016]

[0017] S5: The number of training data sample subsets n allocated to each computing device according to the calculation ij , distribute the training data to each computing device for parallel training.

[0018] In some possible implementations, the method further includes:

[0019] When a new computing device is added to the training cluster, the statistical set D is dynamically updated to add the computing power value and quantity information of the newly added computing device, and the above steps S1 to S4 are re-executed to redistribute the training data.

[0020] In some possible implementations, the method further includes:

[0021] During the training process, the computing power values ​​of the computing devices are periodically recalculated and the training data is redistributed according to the updated computing power values.

[0022] In some possible implementations, the method further includes:

[0023] Setting a computing power threshold to exclude computing devices below the computing power threshold from the training cluster;

[0024] The number of allocated training data samples is adjusted according to the video memory capacity of the computing device to ensure that the amount of allocated training data does not exceed the processing capacity of the computing device.

[0025] In some possible implementations, the method further includes:

[0026] Monitor the training progress of each computing device in real time; calculate the deviation between the actual training speed and the expected speed of each device; trigger the redistribution of training data when the deviation between the actual training speed and the expected speed of any computing device exceeds the preset threshold; recalculate the training data allocation ratio according to the actual training speed of each computing device; dynamically adjust the training data allocation to balance the load of each computing device, so as to

[0027] Optimize overall training efficiency;

[0028] Detect failures of computing devices during training and automatically redistribute training data from failed computing devices to other available computing devices.

[0029] In a second aspect, a large language model data parallel training system based on computing power of a computing device is provided, which includes:

[0030] The computing power statistics module is used to count the computing power values ​​of all computing devices in the training cluster to form a statistical set D: D = (C1: m1, C2: m2, ..., Cn: mn); Ci represents the computing power value of the i-th computing device, mi represents the number of the i-th computing device, and n is the total number of computing device types;

[0031] The ratio calculation module is used to calculate the ratio of the computing power of each type of computing device to the total computing power Pi:

[0032]

[0033] Where j is the sum index, and the value range of j is 1 to n, which is used to traverse all computing device types;

[0034] The sample allocation module is used to calculate the number of training data samples N allocated to each type of computing device according to the computing power ratio Pi. i :

[0035]

[0036] Where N is the total number of training data samples;

[0037] The subset allocation module is used to calculate the number of training data sample subsets n allocated to each computing device ij :

[0038]

[0039] A data distribution module, used for distributing the training data to each computing device according to the calculated number nij of training data sample subsets allocated to each computing device;

[0040] The parallel training module is used to coordinate various computing devices to perform parallel training.

[0041] In some possible implementations, the system further includes:

[0042] The dynamic update module is used to dynamically update the statistical set D when a new computing device is added to the training cluster, add the computing power value and quantity information of the newly added computing device, and trigger other modules to re-execute data allocation;

[0043] The periodic re-statistics module is used to periodically re-statistic the computing power value of the computing device during the training process and trigger other modules to redistribute training data according to the updated computing power value;

[0044] A computing power screening module, used to set a computing power threshold and exclude computing devices with computing power below the computing power threshold from the training cluster;

[0045] The video memory adaptation module is used to adjust the number of allocated training data samples according to the video memory capacity of the computing device to ensure that the amount of allocated training data does not exceed the processing capacity of the computing device.

[0046] In some possible implementations, the system further includes:

[0047] The load balancing module is used to monitor the training progress of each computing device in real time; calculate the deviation between the actual training speed and the expected speed of each computing device; trigger the redistribution of training data when the deviation between the actual training speed and the expected speed of any computing device exceeds a preset threshold; recalculate the training data distribution ratio according to the actual training speed of each computing device; dynamically adjust the training data distribution to balance the load of each computing device, thereby optimizing the overall training efficiency;

[0048] The fault recovery module is used to detect the failure of computing devices during the training process and automatically redistribute the training data of the failed computing devices to other available computing devices.

[0049] According to a third aspect, a computer device is provided, comprising:

[0050] one or more processors;

[0051] a storage device for storing one or more programs,

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above methods.

[0053] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the methods described above is implemented.

[0054] The above technical solution has the following beneficial technical effects:

[0055] The present invention pre-constructs a computing power map of each computing device in the full training network and allocates training data subsets of corresponding sizes to devices with different computing powers, thereby significantly improving the utilization efficiency of computing devices in data parallel training.

[0056] This approach implements a truly "computing power aware" data allocation strategy. Traditional data parallel methods often use uniform allocation or simple proportional allocation, ignoring the performance differences between computing devices. This new approach ensures that data allocation is accurately matched to actual computing power by accurately calculating the computing power share of each device. This not only significantly improves the overall training efficiency, but also minimizes the "straggler" effect, that is, avoiding the situation where high-performance devices wait for low-performance devices.

[0057] In large-scale heterogeneous computing environments, adding or removing computing devices often leads to load balancing issues. However, this computing power-based approach can seamlessly adapt to changes in cluster size and composition. Whether adding new high-performance GPUs or integrating old CPU nodes, the system can quickly adjust data distribution strategies to maintain optimal load balance. This dynamic adaptability not only improves the robustness of the system, but also greatly reduces the complexity of cluster management and maintenance, allowing large language model training to be carried out efficiently in various complex computing environments.

[0058] This approach also brings unexpected energy efficiency improvements. By accurately matching the amount of data and the computing power of the device, each computing unit operates at its optimal efficiency point, avoiding the two inefficient states of underload and overload. This not only optimizes the use of computing resources, but also significantly reduces energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0060] Figure 1 This is a schematic diagram of the large language model training process;

[0061] Figure 2 It is the forward and reverse calculation flow chart of the neural network calculation graph;

[0062] Figure 3 is a flow chart of a method for parallel training of large language model data based on computing power of a computing device according to an embodiment of the present invention;

[0063] Figure 4 It is a functional block diagram of a large language model data parallel training system based on computing power of a computing device according to an embodiment of the present invention;

[0064] Figure 5It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0066] like Figure 1 As shown in the figure, for a large language model, the training process is the process of updating the parameters of the neural network model using the training data and loss function and the optimization algorithm. Training is usually performed in multiple data batches. The training system uses these data batches to generate gradients based on the loss function and the optimization algorithm to correct the model parameters. In the execution process of a multi-layer neural network, each interconnected operator implements a different neural network layer, and the parameters represent the weights updated during training.

[0067] like Figure 2 As shown in the figure, the execution process of the computational graph can be divided into two stages: forward computation and reverse computation. In the forward computation stage, data is input into the first operator, the corresponding output is calculated, and this process is repeated until the last operator is completed. In the reverse computation stage, the gradient of each operator is calculated in turn according to the optimization function and loss, and these gradients are used to update the local parameters. After the reverse computation is completed, the computation of the data mini-batch is completed, and the system will read the next data mini-batch to continue updating the model parameters.

[0068] In conventional data parallel methods, each computing device has a complete copy (Model Replica) of the entire neural network model. During the iteration process, each computing device is only allocated a subset of a batch of data samples, and performs forward calculations on the network model based on this subset. Assuming that the number of training samples in a batch is N, and M computing devices are used for parallel calculations, each computing device will obtain N / M samples. After the forward calculation is completed, each computing device will calculate the loss error based on the local sample, obtain the gradient Gi (where i is the computing device number), and broadcast the local gradient Gi. All computing devices then aggregate the gradient values ​​provided by other devices, use the average gradient (ΣNi=1Gi) / N to update the model, and complete the training of the batch.

[0069] In common data parallel training strategies, it is usually assumed that all computing devices have the same computing power and memory capacity, and the total number of training data samples N is evenly distributed to M computing devices, so that each device obtains N / M data samples for processing. However, in the actual training process, the computing devices in the cluster often have differences in computing power and memory size. If the training data is evenly distributed to all devices indiscriminately, it may cause the computing of low-computing power devices to be overloaded, while the resources of high-computing power devices are not fully utilized.

[0070] The embodiment of the present invention constructs a computing power graph of each computing device in the cluster and allocates training data subsets of corresponding sizes to devices with different computing power levels, thereby significantly improving the utilization efficiency of computing devices in data parallel training.

[0071] Embodiment 1

[0072] like Figure 3 As shown, the embodiment of the present invention provides an optimization method for the problems existing in the existing large language model data parallel training method, and the specific optimization steps are as follows:

[0073] Assume that the total number of training data samples is N and the number of computing devices is M. According to the computing power of the computing devices, the training data samples are reasonably distributed to each computing device to achieve data parallel training. The specific steps are as follows:

[0074] Step S1: Count the computing power values ​​of all computing devices in the cluster (i.e., cluster computing power statistics) to form a statistical set D, which is defined as:

[0075] D=(C1:m1, C2:m2,..., Cn:mn)

[0076] Among them C i represents the computing power of the i-th computing device (e.g., 100 TFLOPS), and m i Indicates the number of computing devices of this type, n indicates the total number of computing device types. i indicates the computing device type, and the value range is 1 to n.

[0077] Step S2: Calculate the proportion Pi of the computing power of each type of computing device to the total computing power.

[0078] Specifically, this step calculates the proportion of the computing power of type Ci devices to the total computing power P i :

[0079]

[0080] Where j is the sum index, and the value range of j is 1 to n, which is used to traverse all computing device types;

[0081] Step S3: Calculate the number of training data samples N allocated to each type of computing device according to the computing power ratio Pi i .

[0082] Specifically, this step calculates the number of training data samples N that are reasonably allocated to the Ci type computing device i :

[0083]

[0084] Step S4: Calculate the number n of training data sample subsets allocated to each computing device ij .

[0085] This step calculates the number of training data sample subsets that are reasonably allocated to each Ci type device. If the i-th type of computing device has m i The number of training data samples that should be allocated to each computing device is:

[0086]

[0087] S5: The number of training data sample subsets n allocated to each computing device according to the calculation ij , distribute the training data to each computing device for parallel training.

[0088] In this embodiment, the distribution of training data is calculated based on the computing power ratio, which ensures the precise match between the computing load and the computing device capabilities, and is conducive to improving training efficiency.

[0089] Furthermore, step S6, a dynamic update step, may also be included. In this step, if a new computing device is added to the training cluster, the statistical set D is dynamically updated, the computing power value and quantity information of the new device are added, and the calculations of steps S1 to S4 are repeated. As the cluster configuration changes (for example, adding or removing computing devices), the statistical set D and the corresponding computing power ratio Pi can be easily updated, thereby maintaining the real-time accuracy of the computing power map.

[0090] The embodiment of the present invention optimizes the method of data parallel training of large language models by constructing a statistical set of computing power of each computing device in the cluster during the early data allocation, allocating training data subsets of corresponding sizes to devices of different computing power levels, avoiding ignoring the differences in device computing power, and evenly allocating data subsets, thereby significantly improving the utilization efficiency of computing devices in data parallel training.

[0091] Furthermore, the method further includes: periodically recounting the computing power value of the computing device during the training process, and reallocating the training data according to the updated computing power value.

[0092] Specifically, first, perform an initial computing power assessment on all computing devices in the training cluster, and distribute training data in proportion to the computing power of each device. This initial allocation ensures that each device can process the appropriate amount of data according to its own capabilities. Set a regular re-evaluation interval, such as re-evaluating every 1,000 training batches or every 2 hours. This interval can be flexibly adjusted according to the specific training task and hardware environment.

[0093] When the preset re-evaluation time point is reached, perform the following steps: Use system tools or custom performance test programs to remeasure the current computing power of each computing device. This step takes into account changes in computing power that may be caused by temperature changes, other background processes, or hardware fluctuations. The newly measured computing power data is updated to the device information table. Based on the updated computing power information, recalculate the proportion of data that should be allocated to each device. Devices with higher computing power will receive a larger proportion of data. Based on the new allocation ratio, calculate the number of new data samples for each device.

[0094] According to the new allocation plan, adjust the training data on each device: for devices with increased computing power, increase their training data volume. Allocate additional samples from the global data pool.

[0095] For devices with reduced computing power, the amount of training data is reduced. The excess samples are returned to the global data pool. In this process, it is ensured that the same data samples are not repeatedly allocated and all samples are reasonably allocated.

[0096] Continue the training process using the new data allocation. Repeat this process of evaluation and adjustment periodically throughout the training process to continuously optimize the data allocation.

[0097] This dynamic adjustment mechanism is particularly suitable for long-term training tasks or in environments where performance fluctuations may occur, such as when using cloud computing resources. By adjusting data allocation in a timely manner, it is possible to maximize the use of available computing resources, improve training efficiency, and maintain overall training quality when the performance of some computing nodes degrades.

[0098] By implementing this method, more efficient use of computing resources can be achieved, improving the overall efficiency and adaptability of large language model training. This dynamic adjustment strategy can make the training process more flexible and better cope with complex and changing computing environments.

[0099] Furthermore, the method also includes: setting a computing power threshold to exclude computing devices below the computing power threshold from the training cluster; adjusting the number of allocated training data samples according to the video memory capacity of the computing device to ensure that the amount of allocated training data does not exceed the processing capacity of the computing device.

[0100] Specifically, first, you need to set a computing power threshold. This computing power threshold represents the minimum computing power level required for computing devices participating in training. The purpose of setting this computing power threshold is to ensure the efficiency of the training process and prevent devices with too low performance from slowing down the overall training speed. For example, the computing power threshold can be set to 30% of the highest computing power device in the cluster. This ratio can be adjusted according to actual needs.

[0101] Before starting training, evaluate the computing power of all computing devices in the cluster: use a standardized performance test procedure to measure the computing power of each computing device, and compare the computing power of each computing device with the set computing power threshold. If the computing power of a computing device is lower than the computing power threshold, it will be excluded from the training cluster. For example, if there are 10 computing devices and the computing power of 2 computing devices is lower than the computing power threshold, these 2 computing devices will be excluded and only the remaining 8 devices will be used for training.

[0102] After determining the computing devices involved in the training, it is also necessary to consider the video memory capacity of each computing device to adjust the data allocation. This step ensures that the amount of data allocated to each computing device does not exceed its processing capacity. First, obtain the video memory capacity of each computing device involved in the training, and estimate the amount of video memory required to process a single sample. Then, calculate the maximum number of samples that each computing device can process. If the number of samples allocated based on computing power exceeds the maximum number of samples that the computing device can process, the number of allocated samples will be reduced accordingly, and the excess samples will be reallocated to other computing devices.

[0103] During the training process, the computing power and available video memory of the computing devices are regularly re-evaluated. If the computing power of a computing device drops below the computing power threshold, it will be removed from the training cluster and the data it is responsible for will be reallocated to other computing devices. If it is found that the available video memory of a computing device has decreased, the number of samples allocated to the computing device will be reduced and the data will be reallocated to ensure that the training process remains stable.

[0104] Through the above steps, we can ensure that the training cluster only contains computing devices with sufficient performance, thereby improving the overall training efficiency. At the same time, the consideration of video memory capacity makes data allocation more reasonable, avoiding memory overflow or performance bottlenecks. The entire training process can dynamically adapt to changes in device performance to ensure efficient and stable progress.

[0105] This approach is particularly suitable for heterogeneous computing environments, which may contain devices of different performance levels. By setting the computing power threshold and considering the memory capacity, training efficiency can be improved, resource utilization can be optimized, training stability can be enhanced, and the system can automatically adjust resource allocation when the cluster is expanded without manual intervention.

[0106] Furthermore, the method further comprises:

[0107] Monitor the training progress of each computing device in real time; calculate the deviation between the actual training speed and the expected speed of each device; trigger the redistribution of training data when the deviation between the actual training speed and the expected speed of any computing device exceeds a preset threshold; recalculate the training data allocation ratio based on the actual training speed of each computing device; dynamically adjust the training data allocation to balance the load of each computing device, thereby optimizing the overall training efficiency;

[0108] Detect failures of computing devices during training and automatically redistribute training data from failed computing devices to other available computing devices.

[0109] Specifically, this embodiment sets a monitoring module for each computing device, which continuously tracks the training progress of the computing device. This includes recording the number of samples processed per hour, the number of training rounds (epochs) completed, and model performance indicators (such as loss function values). These data are collected in real time and sent to the central control system for global analysis and decision-making.

[0110] Based on the collected data, this embodiment calculates the deviation between the actual training speed of each device and the expected speed. The expected speed can be set based on the theoretical performance of the device or the initial test results. This embodiment uses the following formula to calculate the deviation: Deviation = (actual speed-expected speed) / expected speed*100%. This calculation is performed every fixed time interval (e.g., every 15 minutes) to capture the dynamic changes in performance.

[0111] When the actual training speed of any computing device deviates from the expected speed by more than a preset threshold, the system automatically triggers the redistribution process of the training data. For example, in this embodiment, the preset threshold can be set to 20%. If the actual speed of a device is more than 20% slower or faster than expected, redistribution will be triggered. This ensures that the system can respond to performance changes in a timely manner, whether it is a performance degradation or an unexpected performance improvement.

[0112] After the reallocation is triggered, the system will recalculate the allocation ratio of training data based on the actual training speed of each computing device. The new allocation ratio directly reflects the actual processing power of each computing device. For example, if the actual speed of computing device A is twice that of computing device B, then in the new allocation scheme, computing device A will receive approximately twice the amount of data as computing device B. This dynamic adjustment based on actual performance ensures optimal utilization of resources.

[0113] In order to balance the load of each computing device and optimize the overall training efficiency, the system dynamically adjusts the distribution of training data. This process includes removing some data from computing devices with reduced performance and reallocating this data to computing devices with improved performance. The adjustment process adopts a gradual approach to avoid the impact of drastic changes on training stability. At the same time, the system takes into account the continuity and integrity of the data to ensure that each computing device has access to a variety of training samples.

[0114] The method also includes a fault detection and recovery mechanism. The system continuously monitors the status of each computing device, including heartbeat signals, error logs, and performance indicators. When a computing device failure is detected (such as a continuous sharp drop in performance, no response, or frequent errors), the system automatically marks the device as failed and immediately starts the data redistribution process. The training data of the failed computing device will be automatically redistributed to other available computing devices. This process is seamless, ensuring that training can continue even in the face of hardware failures, minimizing interruptions and data loss. Among them, the heartbeat signal is used to periodically check whether the device is still online and responsive, the error log is used to analyze the error information generated by the device and identify potential problems, and the performance indicators are used to track key indicators such as computing speed, memory usage, and temperature of the computing device.

[0115] By implementing this dynamic monitoring and adjustment method, this embodiment can significantly improve the efficiency and reliability of large-scale language model training. This method can adapt to complex and changing computing environments, ensure that computing resources are optimally utilized, and improve the training process's resistance to hardware fluctuations and failures. This not only speeds up the training process, but also improves the quality and consistency of the results, providing strong support for the development of large language models.

[0116] Embodiment 2

[0117] like Figure 4 As shown, this embodiment provides a large language model data parallel training system 200 based on computing power of a computing device, which includes:

[0118] The computing power statistics module 210 is used to count the computing power values ​​of all computing devices in the training cluster to form a statistical set D: D = (C1:m1C2:m2, .., Cn:mn); Ci represents the computing power value of the i-th computing device, m i represents the number of computing devices of type i, and n is the total number of computing device types;

[0119] The ratio calculation module 220 is used to calculate the ratio Pi of the computing power of each type of computing device to the total computing power:

[0120]

[0121] Where j is the sum index, and the value range of j is 1 to n, which is used to traverse all computing device types;

[0122] The sample allocation module 230 is used to calculate the number of training data samples N allocated to each type of computing device according to the computing power ratio Pi. i :

[0123]

[0124] Where N is the total number of training data samples;

[0125] The subset allocation module 240 is used to calculate the number n of training data sample subsets allocated to each computing device. ij :

[0126]

[0127] A data distribution module 250 is used to distribute the training data to each computing device according to the calculated number nij of training data sample subsets allocated to each computing device;

[0128] The parallel training module 260 is used to coordinate various computing devices to perform parallel training.

[0129] In some embodiments, the system further comprises:

[0130] The dynamic update module is used to dynamically update the statistical set D when a new computing device is added to the training cluster, add the computing power value and quantity information of the newly added computing device, and trigger other modules to re-execute data allocation;

[0131] The periodic re-statistics module is used to periodically re-statistic the computing power value of the computing device during the training process and trigger other modules to redistribute training data according to the updated computing power value;

[0132] A computing power screening module, used to set a computing power threshold and exclude computing devices with computing power below the computing power threshold from the training cluster;

[0133] The video memory adaptation module is used to adjust the number of allocated training data samples according to the video memory capacity of the computing device to ensure that the amount of allocated training data does not exceed the processing capacity of the computing device.

[0134] In some embodiments, the system further comprises:

[0135] The load balancing module is used to monitor the training progress of each computing device in real time; calculate the deviation between the actual training speed and the expected speed of each computing device; trigger the redistribution of training data when the deviation between the actual training speed and the expected speed of any computing device exceeds a preset threshold; recalculate the training data distribution ratio according to the actual training speed of each computing device; dynamically adjust the training data distribution to balance the load of each computing device, thereby optimizing the overall training efficiency;

[0136] The fault recovery module is used to detect the failure of computing devices during the training process and automatically redistribute the training data of the failed computing devices to other available computing devices.

[0137] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware.

[0138] Now, it can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the above method embodiment, which will not be repeated here.

[0139] Embodiment 3

[0140] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the above methods is implemented.

[0141] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. Of course, there are other ways of readable storage media, such as quantum memory, graphene memory, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0142] Embodiment 4

[0143] The present invention also provides a computer device. The computer device of the embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the present invention.

[0144] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer device 300 suitable for implementing an embodiment of the present invention. Figure 5 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0145] like Figure 5As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 303 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the computer system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0146] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 305 as needed.

[0147] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, the embodiments of the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the main step diagram. In the above embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit 301, the above functions defined in the system of the present invention are executed.

[0148] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be arranged in a processor, and the names of these units do not constitute limitations on the units themselves in some cases.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data parallel training method for a large language model based on computing power of a computing device, characterized in that: The following steps are involved: S1: Count the computing power values ​​of all computing devices in the training cluster to form a statistical set D: D = (C1: m1, C2: m2, ..., Cn: mn); Ci represents the computing power value of the i-th computing device, mi represents the number of the i-th computing device, and n is the total number of computing device types; S2: Calculate the proportion of computing power of each type of computing device to the total computing power P i : Where j is the sum index, and the value range of j is 1 to n, which is used to traverse all computing device types; S3: Calculate the number of training data samples N allocated to each type of computing device according to the computing power ratio Pi i : Where N is the total number of training data samples; S4: Calculate the number of training data sample subsets n assigned to each computing device ij : S5: The number of training data sample subsets n allocated to each computing device according to the calculation ij , distribute the training data to each computing device for parallel training.

2. The method according to claim 1, characterized in that: Also includes: When a new computing device is added to the training cluster, the statistical set D is dynamically updated to add the computing power value and quantity information of the newly added computing device, and the above steps S1 to S4 are re-executed to redistribute the training data.

3. The method according to claim 1, characterized in that Also includes: During the training process, the computing power values ​​of the computing devices are periodically recalculated and the training data is redistributed according to the updated computing power values.

4. The method according to claim 1, characterized in that: Also includes: Setting a computing power threshold to exclude computing devices below the computing power threshold from the training cluster; The number of allocated training data samples is adjusted according to the video memory capacity of the computing device to ensure that the amount of allocated training data does not exceed the processing capacity of the computing device.

5. The method according to claim 1, characterized in that: Also includes: Monitor the training progress of each computing device in real time; Calculate the deviation between the actual training speed and the expected speed of each device; When the deviation between the actual training speed of any computing device and the expected speed exceeds a preset threshold, the training data redistribution is triggered; Recalculate the training data allocation ratio based on the actual training speed of each computing device; Dynamically adjust the distribution of training data to balance the load of each computing device, thereby optimizing the overall training efficiency; Detect failures of computing devices during training and automatically redistribute training data from failed computing devices to other available computing devices.

6. A large language model data parallel training system based on computing power of computing devices, characterized in that: include: The computing power statistics module is used to count the computing power values ​​of all computing devices in the training cluster to form a statistical set D: D = (C1: m1, C2: m2, ..., Cn: mn); Ci represents the computing power value of the i-th computing device, mi represents the number of the i-th computing device, and n is the total number of computing device types; The proportion calculation module is used to calculate the proportion of the computing power of each type of computing device to the total computing power P i : Where j is the sum index, and the value range of j is 1 to n, which is used to traverse all computing device types; The sample allocation module is used to calculate the number of training data samples N allocated to each type of computing device according to the computing power ratio Pi. i : Where N is the total number of training data samples; The subset allocation module is used to calculate the number of training data sample subsets n allocated to each computing device ij : A data distribution module, used for distributing the training data to each computing device according to the calculated number nij of training data sample subsets allocated to each computing device; The parallel training module is used to coordinate various computing devices to perform parallel training.

7. The system according to claim 6, characterized in that Also includes: The dynamic update module is used to dynamically update the statistical set D when a new computing device is added to the training cluster, add the computing power value and quantity information of the newly added computing device, and trigger other modules to re-execute data allocation; Periodic re-statistics module, used in the training process The computing power value of the computing device is periodically recalculated and other modules are triggered to redistribute training data according to the updated computing power value; A computing power screening module, used to set a computing power threshold and exclude computing devices with computing power below the computing power threshold from the training cluster; The video memory adaptation module is used to adjust the number of allocated training data samples according to the video memory capacity of the computing device to ensure that the amount of allocated training data does not exceed the processing capacity of the computing device.

8. The system according to claim 6, characterized in that Also includes: Load balancing module, used to monitor the training progress of each computing device in real time; Calculate the deviation between the actual training speed and the expected speed of each computing device; When the deviation between the actual training speed of any computing device and the expected speed exceeds a preset threshold, the training data redistribution is triggered; Recalculate the training data allocation ratio based on the actual training speed of each computing device; Dynamically adjust the distribution of training data to balance the load of each computing device, thereby optimizing the overall training efficiency; The fault recovery module is used to detect the failure of computing devices during the training process and automatically redistribute the training data of the failed computing devices to other available computing devices.

9. A computer device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images