Gradient-aware parameter freezing based large model pipeline parallel training method and system

Abstract

The application discloses a large model pipeline parallel training method and system based on gradient perception parameter freezing, and the method steps comprise the following steps: step S01, inserting a matrix low-rank decomposition node at each matrix multiplication operator in a model; step S02, performing intra-node parameter freezing and inter-node parameter freezing during model iterative training, when the intra-node parameter freezing is performed, freezing parameters in a decomposed matrix according to parameter gradients, when the inter-node parameter freezing is performed, judging whether each matrix low-rank decomposition node converges or not to perform freezing; step S03, constructing a calculation graph and an optimization target, and generating a pipeline parallel strategy by solving; and step S04, inserting pipeline re-partitioning at a model parameter refreshing point during model training, so as to re-partition active partitions and distribute the active partitions to corresponding devices. The application has the advantages of simple implementation method, small model training overhead, high training precision and efficiency and the like.

Description

Technical Field

[0001] This invention relates to the field of large model parallel training technology, and in particular to a large model pipeline parallel training method and system based on gradient-aware parameter freezing. Background Technology

[0002] Pipeline parallelism divides the model computation graph into multiple dependent stages and assigns them to different devices for execution. It further subdivides data into micro-batches, feeding them sequentially into the pipeline to achieve parallel execution across devices. Pipeline parallelism combined with parameter freezing is an improvement on pipeline parallelism. In the training process of deep neural network models, the parameters of each layer converge layer by layer from the input layer to the output layer. The idea behind pipeline freezing is to freeze model layers sequentially according to the convergence order, reducing the time overhead of model training and thus improving training efficiency.

[0003] Existing pipeline parallelism strategies mainly include PipeDream (pipeline parallelism), HetPipe (heterogeneous pipeline), GPipe (general pipeline), DAPPLE (data parallel processing pipeline), TeraPipe (terabit-level pipeline), and vPipe (virtual pipeline). Pipeline freezing strategies typically employ FreezePipe (online model layer freezing strategy) and SmartPipe (model layer freezing strategy based on updating gradient values ​​of model layer weights). These strategies usually freeze layers that tend to converge, starting from the input layer and moving to the output layer, based on the magnitude of the back gradient values ​​of the weights of each layer in the model. By freezing the model layer weights layer by layer from front to back, the corresponding nodes can avoid the corresponding back gradient calculation and weight parameter update operations, resulting in the pipeline refresh node of each batch being brought forward, reducing the time overhead of model training.

[0004] However, the above-mentioned pipeline parallelism strategy in the existing technology has the following problems:

[0005] 1. Pipeline freezing granularity generally remains at the model layer, resulting in overly coarse freezing granularity and overly aggressive freezing strategies, which can negatively impact model training convergence and accuracy. Taking the Transformer model, a major component of large language models, as an example, current mainstream pipeline freezing techniques typically employ a layer-by-layer freezing strategy targeting Transformer layers. However, the structure within a Transformer layer is relatively complex, containing multiple branches and residual structures. Following traditional layer-by-layer freezing strategies would also require considering pipeline splitting and operator freezing within the Transformer layer, causing the search space to grow exponentially. Furthermore, it introduces various communication operations, increasing the complexity of forward and backward computational operations.

[0006] 2. The lack of efficient equipment resource reallocation technology after pipeline freezing leads to significant equipment idleness and wasted resources. The core idea of ​​frozen pipeline parallelism is to freeze model layer parameters sequentially according to the convergence order during large model training, eliminating the need for training those parameters. This means omitting the backward gradient calculation and parameter update process for those model layers, thereby gaining a benefit in model execution time. Consequently, due to the elimination of this backward computation, the equipment originally responsible for execution becomes idle. To solve this problem, existing technologies can be mainly divided into two main technical approaches.

[0007] First, to avoid the additional time overhead of pipeline repartitioning and reallocation, the common approach is to directly adjust the pipeline, such as moving the forward operations of later batches forward to fill idle time on the device. However, from the perspective of the entire training process, this method does not reduce the number of idle bubbles on the device, resulting in a significant waste of device resources. Furthermore, while the additional time overhead of pipeline repartitioning and reallocation is avoided, the improvement in training efficiency is limited because device resources are not fully utilized, leaving considerable room for optimization.

[0008] Secondly, considering that pipeline freezes result in idle states on corresponding devices, leading to reduced equipment resource utilization, and that adjusting the pipeline cannot fundamentally reduce these pipeline bubbles, some practitioners have proposed using pipeline repartitioning and redistribution strategies. This involves repartitioning the active layers of the model after each pipeline freeze and re-partitioning the pipeline to fill each device block, thus fully utilizing equipment resources. However, this type of pipeline repartitioning and redistribution introduces additional computational and communication time overhead during model training. This overhead increases linearly with the frequency of pipeline freezes, ultimately offsetting most of the time gains from pipeline freezes, resulting in insignificant overall training time overhead gains. Summary of the Invention

[0009] The technical problem to be solved by this invention is: In view of the technical problems existing in the prior art, this invention provides a parallel training method and system for large model pipeline based on gradient-aware parameter freezing, which is simple to implement, has low time and memory overhead, high training efficiency and high device utilization. It can freeze model parameters layer by layer during model training based on gradient awareness, reduce model training overhead, improve model training efficiency, reduce the impact of parameter freezing on model performance, improve device utilization, and reduce the communication overhead of pipeline repartitioning.

[0010] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0011] A pipelined parallel training method for large models based on gradient-aware parameter freezing, comprising the following steps:

[0012] Rectangular decomposition: Low-rank matrix decomposition nodes are inserted at each matrix multiplication operator in the model to represent the increment matrix ΔW of the weights in the matrix multiplication operators. d1*d2 Perform low-rank decomposition of the matrix to obtain a set of matrices B. d1*R and A R*d2 Where R is the rank of the increment matrix, and A and B represent the two decomposition matrices obtained from the decomposition. Matrix B... d1*R and A R*d2 Further decomposition yields the parameter {B} i A i}, where B i Representative matrix B d1*R The i-th column, A i Representative matrix A R*d2 The i-th row;

[0013] Model parameter freezing: During the iterative training of the model, intra-node parameter freezing and inter-node parameter freezing are performed separately. When freezing intra-node parameters, the parameters {B} are obtained from the low-rank decomposition of each matrix node. i A i The gradient value control of the parameter {B} i A i} Freeze; When freezing the parameters between nodes, check whether each low-rank decomposition node of the matrix converges. If it converges, freeze.

[0014] Pipeline parallelism strategy generation: Construct a computation graph for the model, where nodes represent data or computation nodes, edges represent data flow, and Boolean variables S are set in each edge to control whether the pipeline is split. i The Boolean variable S on each side controls i The computation graph is divided into different sets of operators to form each pipeline stage. The devices in each pipeline stage are divided based on the physical topology of the device cluster. An optimization objective is constructed based on the total cost of the computation graph in the entire pipeline stage. The optimal solution is obtained by solving the optimization objective as the generated pipeline parallel strategy.

[0015] Pipeline parallel reallocation: During model training, pipeline reallocation is inserted at model parameter refresh points to re-divide and allocate active partitions in the model to the corresponding devices until model training is completed.

[0016] Furthermore, during the node parameter freezing process, the parameters {B} obtained from the low-rank decomposition of each matrix node are... i A iThe gradient value of {B} is used to determine the importance of the parameter, and the gradient value of the parameter {B} is used to control the gradient of the parameter {B}. i A i The steps for freezing include:

[0017] Based on parameter B in the decomposition matrix i A i Given the gradient values ​​of the parameters, calculate the 1-norm of the gradient value vector, and sum the values ​​to calculate the parameter {B}. i A i The absolute value of the gradient of the parameters};

[0018] Based on the parameter {B} in the two iterations i A i The gradient value of the parameter is used to calculate the rate of change of the parameter gradient.

[0019] Use parameter {B i A i Historical gradient information for parameter {B} i A i The absolute value of the parameter gradient and the rate of change of the parameter gradient are used to obtain the absolute value of the parameter gradient and the rate of change of the parameter gradient after sliding equilibrium.

[0020] The importance value Γ of the parameter is calculated based on the absolute value of the parameter gradient and the rate of change of the parameter gradient after the sliding equilibrium is reached.

[0021] The importance value Γ of the parameter is determined. If it is less than a preset threshold, then parameter {B} is... i A i The result is frozen and merged into the model weights, and the rank of the decomposition matrix is ​​reduced by one.

[0022] Furthermore, the parameter gradient value ψ(B) of the parameters {Bi,Ai} is calculated. i A i The expression for ) is:

[0023]

[0024] Wherein, ψ(A) i ), ψ(B i ) are parameters A i B i The absolute value of the gradient, They represent parameter A respectively i B i The 1-norm value of the gradient value vector;

[0025] For parameter {B i A i The expression for applying the gradient values ​​of the parameters of} to the sliding balance is:

[0026] Ψ t (B i A i )=αψ t (B i A i )+(1-α)Ψ t-1 (B i A i )

[0027] Where, ψ t (B i A i ) represents the parameter {B} in the t-th iteration. i A i The absolute value of the gradient of Ψ, where t represents the number of iterations in training. t-1 (B i A i ) represents the absolute value of the parameter gradient after sliding equilibrium in the (t-1)th iteration, α represents the hyperparameter controlling the sliding equilibrium ratio, and Ψ t (B i A i () represents the absolute value of the parameter gradient after sliding equilibrium in the t-th iteration;

[0028] The rate of change of the parameter gradient is calculated by dividing the difference between the norm 1 values ​​of the gradients of two consecutive iterations by the norm 1 value of the previous gradient. The calculation expression is as follows:

[0029]

[0030] φ t (B i A i )=φ t (B i )+φ t (A i ),

[0031] Φ t (B i A i )=αφ t (B i A i )+(1-α)Φ t-1 (B i A i )

[0032] Where φ t (A i ), φ t (B i ) represent the parameters A in the t-th iteration. i B i The gradient rate of change, φt (B i A i ) represents the parameter {B} in the t-th iteration. i A i The parameter gradient rate of change is calculated, and using the same sliding equilibrium concept, historical gradient information is used to calculate the parameter gradient rate of change Φ after sliding equilibrium in the t-th iteration. t (B i A i );

[0033] The parameter importance value Γ is calculated based on the absolute value of the parameter gradient and the rate of change of the parameter gradient after the sliding equilibrium is reached. t The calculation expression is:

[0034]

[0035] Where r represents the rank of the nodes in the low-rank decomposition of the matrix.

[0036] Furthermore, the step of freezing the parameters between nodes, and determining whether each low-rank decomposition node of the matrix converges on a node-by-node basis, and freezing the nodes if they converge, includes:

[0037] During model training, all active nodes are topologically sorted {node1, node2, node3, ..., node...} n}, where n is the total number of active nodes, and the gradient values ​​of the parameters for each active node in the model are calculated.

[0038] Based on the topological sequence of active nodes, determine whether a node needs to be frozen from front to back according to the parameter gradient value of each active node. When multiple active nodes need to be frozen at the same time, determine the maximum number of nodes that can be frozen in the current iteration based on the number of active nodes and the freezing rate in the current iteration.

[0039] Furthermore, the absolute value γ of the parameter gradient of the active node is calculated. t The expression is:

[0040] γ t =∑ i Ψ t (B i A i )

[0041] Among them, Ψ t (B i A i ) represents the absolute value of the parameter gradient of parameters {Bi, Ai};

[0042] Based on the absolute value of the parameter gradient γ of the active node tDetermine whether an active node needs to be frozen using the following formula:

[0043]

[0044] Where μ represents the preset threshold;

[0045] And calculate the maximum number of nodes allowed to be frozen in the t-th iteration according to the following formula.

[0046]

[0047] in, γ represents the number of active nodes in the t-th iteration, and γ is the freezing rate.

[0048] Furthermore, the Boolean variable S on each control edge i The steps to achieve dynamic partitioning of the computation graph include:

[0049] When the computation graph is a linear computation graph, the total number of pipelines S and each Boolean variable S are... i The following constraints exist:

[0050]

[0051] When there are branches in the computation graph, the total number of pipeline stages that are split is calculated according to the following formula:

[0052]

[0053] And the configuration meets the following constraints:

[0054] Sl i ≥ControlVar

[0055] Sl i ≤M*ControlVar

[0056] Among them, S in S represents the total number of pipeline stages accumulated at the current branch entry point. out This represents the total number of pipeline stages after the current branch structure, ControlVar represents the binary constraint variable, and S i_j This represents a Boolean variable used to control whether to split the i-th branch on the j-th edge.

[0057] Furthermore, the computational and storage overhead during model training are calculated using the following formulas:

[0058] CompO i =CompW i +CompB i +CompAi ,

[0059] CompW i =ProfileForward(w i ,op i d i ),

[0060] CompB i =Profile(b i ,op i d i ),

[0061] CompA i =Profile(a i ,op i d i ),

[0062] MemO i =w i +MemB i +MemA i ,

[0063] MemB i =wb i +ab i +gb i +optb i ,

[0064] MemA i =wa i +aa i +ga i +opta i .

[0065] Among them, CompO i For computation operator O in computation graph i Computational overhead, CompW i CompB i CompA i Representing Wi and B respectively i A i The computational overhead is represented by ProfileForward, where Profile represents the forward computation overhead, and Profile represents the sum of the forward and backward computational overheads. i For operator type, d i For device type, MemO i For operator O i The memory overhead is determined by the memory overhead w of the original parameter matrix W. i Memory overhead MemB for matrix factorization modules A and B i and MemAi It consists of three parts, of which w i wb i and wa i The memory overheads for the original parameter matrix W and the matrices A and B introduced by matrix decomposition are aa. i ab i The memory overhead of the forward activation values ​​of matrices A and B, respectively. i gb i Opta represents the memory overhead of the back gradients of matrices A and B, respectively. i optb i The memory overhead of saving the optimizer state required to update the parameters of matrices A and B respectively.

[0066] Furthermore, based on the computational graph, the total overhead throughout the entire pipeline stage is constructed according to the following formula:

[0067]

[0068] Where, ∑ k CostStage k PPComm represents the total time cost required to execute a single microbatch in the pipeline, where B is the maximum number of microbatches in the pipeline. The time cost of each pipeline stage is obtained by summing the computation time of each operator and the communication time between operators within that stage. ij It calculates the communication overhead between operators Oi and Oj.

[0069] Furthermore, the step of inserting pipeline repartitioning at the model parameter refresh point to repartition and allocate the active partitions in the model to the corresponding devices includes: moving the forward computation operation of the (t+1)th batch-size data in the frozen part forward and inserting it into the device for the backward gradient computation of the tth batch-size data; during model training, recording the load of each device, and when the load difference between devices is less than a preset threshold, adopting a lightweight pipeline parallel partitioning strategy with equal operator division; otherwise, calling a pipeline parallel strategy search algorithm to repartition the active area of ​​the model; inserting a preprocessing operation after each batch-size data is computed, and using the preprocessing operation to predict the upper limit of the number of nodes with frozen parameters to estimate the potential operators that need to be transmitted, and calling the communication operator in advance during the backward computation of the model to adjust the distribution of model weight parameters.

[0070] A computer system includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0071] Compared with the prior art, the advantages of the present invention are as follows:

[0072] 1. This invention employs a hierarchical parameter freezing strategy, introducing matrix factorization nodes for all matrix multiplication operators in the model. This decouples model parameters from gradient updates and treats each matrix factorization module as a node. The parameter freezing process is divided into intra-node parameter freezing and inter-node parameter freezing strategies. Intra-node parameter freezing reduces the rank of the matrix factorization node for each matrix multiplication operator, improving model training efficiency while minimizing the impact of parameter freezing on model performance. Furthermore, inter-node parameter freezing further refines parameter freezing during model training into node-by-node freezing. This more refined parameter freezing strategy reduces the possibility of premature parameter freezing, enabling operator-level, fine-grained parameter freezing. This significantly reduces the time and memory overhead required for model training, improves training efficiency, ensures training accuracy, and enhances equipment utilization.

[0073] 2. This invention, by configuring corresponding pipeline repartitioning and redistribution algorithms based on the specific structure formed after introducing matrix factorization nodes, can further reduce the corresponding time overhead, thereby maximizing the benefits of pipeline freezing while maintaining model training accuracy.

[0074] 3. Furthermore, by inserting preprocessing operations during the pipeline reverse computation process and transmitting model parameters in advance, this invention can significantly reduce the communication overhead of pipeline repartitioning. By designing a lightweight pipeline repartitioning algorithm based on the number of operators, it can further reduce the computational overhead caused by the model frequently calling the pipeline parallel strategy search algorithm. Attached Figure Description

[0075] Figure 1 This is a flowchart illustrating the LoRA fine-tuning method.

[0076] Figure 2 This is a schematic diagram of the frozen pipeline execution process.

[0077] Figure 3 It is a frozen pipeline parallel timing diagram.

[0078] Figure 4 This is a schematic diagram of the parallel timing of the GPipe pipeline used in this embodiment.

[0079] Figure 5 This is a schematic diagram illustrating the implementation process of the large model pipeline parallel training method based on gradient-aware parameter freezing in this embodiment.

[0080] Figure 6 This is a schematic diagram illustrating the principle of low-rank matrix decomposition in this embodiment.

[0081] Figure 7This is a schematic diagram illustrating the auxiliary principle of parameter freezing used in this embodiment.

[0082] Figure 8 This is a schematic diagram illustrating the principle of the layer-by-layer freezing strategy used in this embodiment.

[0083] Figure 9 This is a schematic diagram illustrating the principle of linear computation graph segmentation used in this embodiment.

[0084] Figure 10 This is a schematic diagram illustrating the principle of nonlinear computation graph segmentation used in this embodiment.

[0085] Figure 11 This is a timing diagram of the pipelined parallel redistribution algorithm used in this embodiment. Detailed Implementation

[0086] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0087] LoRA (Low-Rank Adaptation) is a low-resource fine-tuning method for large models. It only fine-tunes a subset of the model's parameters, effectively reducing memory overhead during model fine-tuning based on low-rank matrix factorization. Its design is primarily based on the following characteristics: 1) The pre-trained model possesses a very small intrinsic dimensionality, meaning there exists a parameter with extremely low dimensionality, and fine-tuning this parameter achieves the same effect as fine-tuning in the full parameter space; 2) Larger models tend to have smaller intrinsic dimensionality, which to some extent explains why large models can still perform well after training with a small number of samples. Improvements are made to the parameter update process in conventional model training:

[0088] W0 + ΔW = W0 + BA B∈R d*r ,

[0089] A∈R r*k , r << min(d, k) (1)

[0090] As shown in formula (1), LoRA divides the parameter update during model fine-tuning into two parts, W0 and ΔW. The first part, W0, refers to the pre-trained weight parameters loaded into the large model, while the second part, ΔW, is the incremental part of weight parameter update during the fine-tuning process of downstream tasks.

[0091] LoRA then performs a low-rank decomposition on the incremental part ΔW, further dividing ΔW into the product of two low-rank matrices A and B. The ranks r of matrices A and B are much smaller than the length and width (d, k) of the original weight matrix, making the memory overhead of storing matrices A and B significantly lower than that of storing matrix W. Throughout the LoRA training process, the parameter matrix W0 is frozen, ensuring that only the parameters in A and B are trained. Figure 1 As shown, the updated forward computation process becomes:

[0092] h=W0x+ΔWx=W0x+BAx (2)

[0093] LoRA transforms the update of all model parameters during model fine-tuning into an update of two smaller matrices A and B by performing low-rank decomposition on the weight increment matrix. When the rank r of matrices A and B is much smaller than the original weight matrix, the training parameters are only one ten-thousandth of the total parameters, which can greatly reduce the memory overhead and GPU memory usage during model training, including forward activation values ​​and backward gradients.

[0094] Pipeline parallelism divides the entire model into different computational blocks based on model layers, and then distributes these blocks sequentially to different devices. Each device is responsible for several layers of the model, considered a pipeline stage, and thus a Virtual Worker. The input data for each batch is divided into multiple mini-batches and sequentially input into the network for computation. Therefore, although there are data dependencies within each mini-batch, there are no data dependencies between mini-batches. Each computation node can immediately proceed to the next batch of data after completing the computation task of its current mini-batch. When the pipeline is full, devices on each node can execute simultaneously, thus utilizing computational resources more efficiently. In traditional mainstream pipeline parallelism strategies, the division and distribution of the pipeline remain fixed throughout the model training process. However, model training has convergence characteristics; the convergence speed of the weight parameters in each layer of the model differs, generally exhibiting a convergence pattern from the input layer to the output layer. During the entire model training process, the shallower layer weight parameters converge earlier, and continuing training after convergence yields little benefit, resulting in wasted computational resources. By using pipeline freezing technology, a strategy of freezing the model weight parameters in a timely manner after they converge layer by layer, computational resources can be released to concentrate on training deep parameters, thereby maintaining model accuracy while reducing the overall training time of the model.

[0095] Pipeline freezing strategies selectively freeze the weight parameters of specific model layers during training, eliminating the need for backward gradient calculations and weight parameter updates, thus reducing training time. Since the weight parameters of each layer converge progressively from the input to the output layer during deep neural network training, freezing them promptly after convergence can save significant training time and improve efficiency. Traditional pipeline freezing strategies, such as FreezePipe and SmartPipe, typically freeze layers that tend to converge, starting from the input and working backward gradients of the weights. Figure 2As shown, by freezing the model layer weights layer by layer from front to back, the corresponding nodes can skip the back gradient calculation and weight parameter update operations, thus advancing the pipeline refresh nodes for each batch and reducing the time overhead of model training. Figure 3 As shown, compared with the ordinary model training process, pipeline freezing technology has two main challenges: the model layer freezing strategy, i.e., determining a suitable evaluation criterion to determine which layers to freeze or keep active during model training; and a suitable lightweight pipeline tuning method, i.e., minimizing the additional time overhead and maximizing the time cost benefit of model freezing. However, traditional pipeline freezing technologies SmartPipe and FreezePipe have the following shortcomings in these two aspects: 1. The model freezing granularity is coarse, only stopping at the model layer; 2. The selection of historical gradient versions is relatively extreme, which is prone to producing incorrect model layer freezing strategies; 3. In terms of pipeline tuning, SmartPipe wastes a lot of device resources, and the actual training time cost benefit is very limited. FreezePipe's pipeline repartitioning and redistribution process has too much time overhead, and there is still room for further optimization.

[0096] This invention combines gradient-aware pipeline parallelism and freezing methods based on matrix factorization to further refine the parameter freezing process during model training. At the same time, by adopting a pipeline repartitioning and redistribution strategy with overlapping communication parts, this additional time overhead is minimized, thereby maximizing the benefits of pipeline freezing while maintaining model training accuracy.

[0097] This embodiment uses synchronous pipeline technology, represented by the GPipe algorithm, to achieve parallel training of large models in a pipelined manner. The GPipe pipeline parallel timing is as follows: Figure 4 As shown, after each iteration, there is a unified Refresh Point for uniformly updating the model parameters, i.e., "pipeline synchronous refresh". Figure 5 As shown, the detailed steps of the large model pipeline parallel training method based on gradient-aware parameter freezing in this embodiment include:

[0098] Step S01. Rectangular Decomposition: Insert low-rank matrix decomposition nodes at each matrix multiplication operator in the model to convert the increment matrix ΔW of the weights in the matrix multiplication operators. d1*d2 Perform low-rank decomposition of the matrix to obtain a set of matrices B. d1*R and A R*d2 Where R is the rank of the increment matrix, d1 and d2 represent the row and column dimensions of the increment matrix, respectively, and A and B represent the two decomposition matrices obtained from the decomposition. Matrix B... d1*R and A R*d2 Further decomposition yields parameters {Bi,Ai}, where B iThe \(i\)-th column of matrix \(B\) d1*R , and \(A\) i represents the \(i\)-th row of matrix \(A\). R*d2

[0099] Consider the following two issues: (1) Model parameter freezing and matrix low-rank factorization are essentially consistent and have a unified optimization goal, concentrating limited computing and storage resources on updating important parameters in the model, improving model training efficiency while maintaining model performance; (2) The adjustment of the rank in matrix low-rank factorization can be regarded as a special operator internal parameter freezing process. In this embodiment, a matrix factorization module is additionally introduced, and matrix low-rank factorization nodes are inserted at each matrix multiplication operator in the model for matrix low-rank factorization, and the rank \(R\) is uniformly initialized.

[0100] Suppose the incremental matrix \(\Delta W\) of the weights of a matrix multiplication operator d1*d2 can be obtained by full-rank factorization to get a set of matrices \(B\) d1*R and \(A\) R*d2 such that:

[0101] \(\Delta W\) d1*d2 = \(B\) d1*R * \(A\) R*d2 (1)

[0102] Further decomposition gives:

[0103]

[0104] where (abbreviated as \(B\) i ) represents the \(i\)-th column of matrix \(B\) d1*R , (abbreviated as \(A\) i ) represents the \(i\)-th row of matrix \(A\) R*d2 , that is, the product of two matrices \(B\) and \(A\) is equivalent to the sum of the products of the \(i\)-th column of matrix \(B\) and the \(i\)-th row of matrix \(A\).

[0105] In the low-rank factorization strategy of LoRA, the rank \(R\) needs to be reduced until \(r\) is much smaller than \(\min(d1, d2)\). This rank reduction process can be achieved by simultaneously freezing the \(i\)-th column of matrix \(B\) and the \(i\)-th row of matrix \(A\). As Figure 6 shown, the LoRA low-rank factorization strategy is equivalent to freezing the parameters corresponding to \(\{Bi, Ai\}\) (\(0\leq i < R - r\)) in the weight matrix during the operator initialization process, and concentrating the training on adjusting the remaining parameters.

[0106] ​LoRA matrix low-rank decomposition is a special type of operator intrinsic parameter freezing. This embodiment introduces a matrix decomposition module at each matrix multiplication operator in the original model structure, which can further refine the granularity of model parameter freezing. At the same time, it decouples the large number of parameters of the original model from the training update process, providing conditions for more efficient pipeline parallelism and communication methods.

[0107] Step S02. Model Parameter Freezing: During the iterative training of the model, intra-node parameter freezing and inter-node parameter freezing are performed separately. When freezing intra-node parameters, the parameters {B} obtained from the low-rank decomposition of each matrix node are used. i A i The gradient value control of the parameter {B} i A i} Freeze; when freezing parameters between nodes, check whether the low-rank decomposition nodes of each matrix converge. If they converge, freeze.

[0108] In model training, the matrix multiplication operator is the core operator and also the main source of overhead. This embodiment inserts a low-rank matrix factorization node into the matrix multiplication operator in the model, while keeping the parameters of other operators frozen. For example... Figure 6 As shown, for each matrix multiplication operator of the inserted matrix factorization module in the model, it is regarded as a node. For this special model structure, a gradient-aware hierarchical parameter freezing strategy is adopted. In this embodiment, the intra-node parameter freezing and inter-node parameter freezing are performed according to the granularity of the model parameter freezing.

[0109] (1) Freezing parameters within nodes

[0110] Intranode parameter freezing refers to the process of reducing the rank of each matrix decomposition node in the model. For example... Figure 6 As shown, low-rank matrix decomposition can essentially be viewed as a process of freezing operator intrinsic parameters. Considering the varying importance of parameters across different modules in the model, parameters with higher importance have a greater impact on model performance and require concentrated training resources. Conversely, parameters with lower importance have a smaller impact on model performance and can be frozen to save training resources. Therefore, how to measure the importance of model parameters is a crucial basis for freezing operator intrinsic parameters, significantly influencing the final model performance.

[0111] Assuming that during model training, the gradient value of parameter a is small, meaning parameter a tends to converge, this proves that parameter a has a limited role in learning from the samples, and updating parameter a has a small impact on model performance. Conversely, if the gradient value of parameter b is large, this proves that the model needs to continuously adjust parameter b based on sample information during training, thereby making the model better fit the feature distribution of the samples. Updating parameter b has a greater impact on model performance. From the above analysis, it can be seen that the magnitude of the parameter gradient value is one of the important metrics for parameter importance. In this embodiment, parameter B is used. i A i The absolute value of the gradient vector with 1 norm is used to calculate the gradient of the parameters, which measures the magnitude of the gradient of the model parameters.

[0112] Parameter B i The absolute value of the gradient ψ(B) i It can be calculated using the following formula:

[0113]

[0114] Among them B i Let B represent the i-th column of the decomposition matrix. Equation (3) represents the parameter B. i The magnitude of a gradient can be measured using the 1-norm of its corresponding gradient value. This represents the gradient value of parameter Bi. Similarly, A... i The absolute value of the gradient ψ(A) i The same principle can also be used to calculate, ultimately resulting in ψ(B) i ), ψ(A i The absolute value of the parameter gradient ψ(B) of the parameters {Bi, Ai} can then be calculated. i A i ):

[0115]

[0116] Equation (4) above indicates that the absolute value of the parameter gradient of {Bi, Ai} in the decomposition matrix can be obtained by summing the two corresponding absolute values.

[0117] This embodiment describes {B} in each matrix factorization module. i A i To maintain the absolute value of the gradient of a parameter, when {B} i A i When the absolute value of the gradient of} is below a certain threshold, it indicates that the update of this set of parameters has little impact on the model performance and can be frozen. Considering that other special cases may occur in actual model training, this embodiment further optimizes equation (4):

[0118] First, since the gradient transformation during model training is random, the magnitude of the gradient value in a single iteration is insufficient to measure the global importance of the parameter. Therefore, this embodiment further introduces the sliding balance concept for equation (4), and adjusts the parameter {B}. i A i The gradient values ​​of the parameters are processed by sliding equilibrium to obtain the gradient values ​​of the parameters after sliding equilibrium, i.e.:

[0119] Ψ t (B i A i )=αψ t (B i A i )+(1-α)Ψ t-1 (B i A i (5)

[0120] Where, ψ t (B i A i ) represents the parameter {B} in the t-th iteration. i A i The absolute value of the gradient of Ψ, where t represents the number of iterations in training. t-1 (B i A i ) represents the absolute value of the parameter gradient after sliding equilibrium in the (t-1)th iteration, α represents the hyperparameter controlling the sliding equilibrium ratio, and Ψ t (B i A i ) represents the absolute value of the parameter gradient after sliding equilibrium in the t-th iteration.

[0121] As shown in equation (5), by adopting the concept of sliding equilibrium, the parameter pair {B} can be preserved. i A i The gradient value information from previous iterations is used to avoid premature parameter freezing caused by a single minimal gradient value.

[0122] Secondly, besides gradient randomness, during model training, the gradient value may repeatedly iterate between positive and negative values. This manifests as the model parameters oscillating around a certain value. This embodiment considers this a waste of computational resources and requires a freezing strategy. To address this, this embodiment further introduces the calculation of the gradient value change rate, based on the parameter {B} in the two iterations. i A i The gradient value of the parameter is used to calculate the rate of change of the parameter gradient.

[0123] In a specific application embodiment, the expression for calculating the rate of change of the parameter gradient based on the parameter gradient values ​​of {Bi, Ai} in two iterations is as follows:

[0124]

[0125] φ t (B i A i )=φ t (B i )+φ t (A i (7)

[0126] Φ t (B i A i )=αφ t (B i A i )+(1-α)Φ t-1 (B i A i (8)

[0127] As shown in Equation (6), the difference between the norm of the gradient values ​​of the parameter in two consecutive iterations is divided by the norm of the gradient value of the previous iteration to measure the rate of change of the parameter gradient, so as to keep it in sync with Equation (5). As can be seen from Equation (5), the smaller the parameter gradient value, the smaller the value of Equation (5), and the less important the parameter is. When the value is less than a certain threshold, the parameter is frozen. Therefore, the optimization direction of Equation (5) is to approach 0. In Equation (6), the gradient values ​​of the two consecutive iterations are not directly divided, but a subtraction is performed first, so that the final convergence target is changed from 1 to 0, which is consistent with Equation (5). In addition, considering the influence of randomness in the actual training of the model, a moving average is also used for the rate of change of the gradient, as shown in Equation (8), which can retain the information of previous iterations to a certain extent.

[0128] After obtaining the absolute value of the parameter gradient and the rate of change of the parameter after sliding equilibrium, the importance value Γ of the parameter is then calculated based on these parameters. t This embodiment takes into account both equations (5) and (8) to calculate the parameter importance value Γ. t The expression is:

[0129]

[0130] Where r represents the rank of the matrix decomposition module.

[0131] As shown in equation (9), the weighted average of the values ​​of equations (5) and (8) is used as the basis for finally measuring the importance of the operator's intrinsic parameters. When this value is less than a certain threshold, the importance of the parameter to {B} is reduced. i A i} Freeze the matrix, reduce the rank of the matrix, and merge the frozen result into the original weight matrix. The two coefficients in equation (9) mainly consider the change law of gradient value during model training. When the model is first trained, the gradient value changes rapidly and the rate of change is large. In the early stage of training, the importance of the parameter is measured more by the magnitude of the parameter gradient value. Based on this, the parameter is frozen and r gradually decreases. As r gradually decreases, the weight of the second term, the gradient change rate, will gradually increase. When the training gradually stabilizes, the parameter gradually converges and stabilizes, or appears to oscillate around a certain value. The gradient change rate can be used more to freeze the parameter as early as possible to save computing resources.

[0132] It is understandable that the absolute value of the parameter gradient, the rate of change of the parameter gradient, the parameter importance value, etc., can also be calculated using other methods according to actual needs.

[0133] Specifically, when freezing the intranode parameters, during the model iterative training process, first calculate {B} in the decomposition matrix according to equation (4). i A i The absolute value of the parameter gradient is obtained by calculating the absolute value of the parameter gradient and then performing sliding equilibrium processing on the calculated absolute value of the parameter gradient according to equation (5) to obtain the absolute value of the parameter gradient Ψ after sliding equilibrium. t (B i A i The rate of change Φ of the parameter gradient after sliding equilibrium is calculated by dividing the difference between the norm of the gradient values ​​of the two iterations before and after the previous iteration by the norm of the previous gradient value, according to equation (8). t (B i A i The gradient change rate of the combined equation (8) and the absolute value of the parameter gradient of equation (5) are calculated according to formula (9) for each module {B i A i The importance value of the parameter Γ t Then, determine the importance value Γ of the parameter. t The size of Γ t When the values ​​are below a certain threshold, {Bi, Ai} in the decomposition matrix are frozen, and the results are merged into the model weights, reducing the rank of the decomposition matrix by one. If the model parameters oscillate around a certain value, the parameters are also frozen.

[0134] (2) Parameter freezing between nodes

[0135] Inter-node parameter freezing is a freezing strategy applied to all matrix factorization modules in the model. During model training, it's necessary to promptly detect and freeze the convergence of parameters in each module to save unnecessary overhead. This embodiment inserts the matrix multiplication operator into the matrix factorization module (low-rank matrix factorization node) while keeping other parameters frozen during training; only the matrix multiplication operator in the model is updated. For example... Figure 7 As shown, in this embodiment, each low-rank decomposition node of the matrix is ​​treated as a separate node. Similar to the intra-node parameter freezing strategy, this embodiment uses model gradient information as the main basis for judging parameter convergence. When the gradient value of the parameter tends to 0, the parameter converges. Then, the absolute value γ of the parameter gradient of the active node is calculated first. t :

[0136] γ t =∑ i Ψ t (B i A i (10)

[0137] Among them, Ψ t (B i A i ) represents parameter {B i A i The absolute value of the gradient of the parameter {Bi, Ai} can be obtained by formula (5), that is, by summing the 1-norm of the gradient values ​​of the parameter pair {Bi, Ai} in the module.

[0138] Based on the absolute value of the parameter gradient γ of the active node t Determine whether an active node needs to be frozen using the following formula:

[0139]

[0140] Where μ represents a preset threshold. That is, when γ t When the value is below a certain threshold, it means that the parameters of that node can be frozen.

[0141] Regarding the selection of the freezing order among different nodes, this embodiment adopts a node-by-node parameter freezing strategy to avoid the "false freezing" phenomenon during the model parameter freezing process. "False freezing" refers to the situation where, after the module parameters are frozen, reverse calculation is still required. For example... Figure 8 As shown, when the parameters of Stage 4 in the model are frozen while Stage 3 remains active, although the parameters in Stage 4 are frozen, due to the chain rule, the backward gradient still needs to be calculated for updating the parameters in Stage 3. To avoid this situation of false parameter freezing and to effectively utilize device computing resources, this embodiment further refines the layer-by-layer freezing algorithm into a node-by-node freezing algorithm.

[0142] This embodiment introduces the matrix factorization concept of LoRA to further improve the algorithm for adjusting the matrix rank in matrix factorization. It further unifies this process into a parameter freezing process. Through the special model structure of matrix factorization, the granularity of model parameter freezing is refined to the operator level. At the same time, the original model parameters are coupled with the gradient update process, which can create new opportunities for computation and communication overlap during model training.

[0143] In this embodiment, the detailed steps for freezing parameters between nodes include:

[0144] Perform topological sorting of all active nodes during model training: {node1, node2, node3, ..., node...} n}, where n is the total number of active nodes, and the gradient values ​​of the parameters for each node in the model are calculated. Specifically, the calculation can be performed according to formula (10);

[0145] According to the topological sequence of nodes, nodes are frozen sequentially from front to back based on the absolute value of their parameter gradients to prevent "false freezing". When multiple nodes have parameter gradient values ​​less than a preset threshold and need to be frozen, i.e., when equation (11) is satisfied simultaneously, in order to prevent the parameters from freezing too quickly and affecting the model performance, this embodiment further introduces a freezing rate γ. The maximum number of nodes allowed to be frozen in the current iteration is determined based on the number of active nodes and the freezing rate in the current iteration.

[0146]

[0147] in, This represents the number of active nodes in the t-th iteration. This represents the maximum number of nodes that can be frozen in the t-th iteration.

[0148] This embodiment adopts a gradient-aware parameter freezing strategy, which refines the model parameter freezing process through a layered freezing method. This can save resource consumption while reducing the impact of parameter freezing on model performance.

[0149] To avoid potential parameter overfitting during actual training, this embodiment further employs a method such as k-fold cross-validation to reliably address parameter overfitting.

[0150] Step S03. Pipeline Parallel Strategy Generation: Construct a computation graph for the model, where nodes represent data or computation nodes, edges represent data flow, and Boolean variables S are set in each edge to control whether the pipeline is split. i The Boolean variable S on each side controls iThe computation graph is divided into different sets of operators to form various pipeline stages. The devices in each pipeline stage are divided based on the physical topology of the device cluster. An optimization objective is constructed based on the total cost of the computation graph in the entire pipeline stage. The optimal solution is obtained by solving the optimization objective as the generated pipeline parallel strategy.

[0151] This embodiment introduces a matrix factorization module into the original model structure, making the model more complex. Different parameter freezing strategies will change the model's execution flow, thus requiring a redesign of the pipeline parallel strategy and a reconfiguration of the reassignment algorithm. This embodiment designs corresponding pipeline parallel strategy generation algorithms and pipeline reassignment algorithms for the complex structure with the matrix factorization module.

[0152] In this embodiment, during the generation of the pipelined parallel strategy, the computation graph is first dynamically partitioned. The pipelined parallel strategy aims to partition the model computation graph and the device cluster. While searching for suitable inter-layer partitioning methods on the computation graph, based on the physical topology of the device cluster, the device cluster is divided into feasible device cluster blocks, and these blocks are then compared with the various computational sub-modules obtained from the pipelined partitioning. Figure 1 One-to-one correspondence.

[0153] Taking the simplest linear computation graph as an example, if the computation graph consists of a set of operators O1, O2, O3, ..., O... that follow a linear topological sequence... k Composition, such as Figure 9 As shown, in a typical computation graph, nodes represent data or computation nodes, and edges represent data flow. Based on this, a Boolean variable S controlling the pipeline partitioning is further added to each edge of the computation graph. i An extended graph is formed to represent the partitioning process of the model computation graph in the pipelined parallel strategy. For example, when S1 = 1, it indicates that pipeline partitioning occurs between operators O1 and O2; otherwise, no partitioning occurs. Based on this extended graph, the Boolean variables S on each edge are controlled... i This allows for dynamic segmentation of the computation graph.

[0154] Specifically, the total number of pipelines S and each variable S i The following constraints exist:

[0155]

[0156] As shown in equation (13), in the linear computation graph, the total number of pipelines S is equal to the Boolean variable S of each control segment. i The sum plus one.

[0157] However, taking the Transformer layer as an example, because the layer contains residual connections and classic branching structures such as Q, K, V calculations, the constraints in the linear computation graph alone are insufficient to handle graph partitioning in the Transformer model. Based on the constraints of the linear computation graph, this embodiment further addresses the dynamic partitioning of the computation graph in nonlinear computation graphs with branching conditions, such as... Figure 10 As shown, taking a two-branch computation graph as an example, assuming there are branches Sl1 and Sl2 in the graph, the splitting control variables on the operator edges of each branch are respectively represented by S... i_j This represents a Boolean variable used to control whether to split the branch on the j-th edge of the i-th branch. Figure 9 Different Figure 10 The two ends are not Input and Output, but S in and S out To represent the nestability of branch structures, S in S refers to the total number of pipeline stages accumulated at the branch entry point. out This represents the total number of pipeline stages after the branching structure. The calculation formulas and related constraints between the above variables in this embodiment are as follows:

[0158]

[0159] Sl i ≥ControlVar (16)

[0160] Sl i ≤M*ControlVar (17)

[0161] As shown in equation (14), this embodiment simplifies the pipeline partitioning calculation method in the branch computation graph to S. in By adding the maximum value of the segmentation in each path, this segmentation method can ensure that each stage of the entire model is executed sequentially according to the pipeline sequence. However, an invalid partitioning situation may occur in the branch computation graph, that is, among many branches, only some branches are pipelined. After this segmentation method, the original computation graph cannot be divided into independent subgraphs, which is an invalid partitioning situation. Therefore, this embodiment introduces an additional binary constraint variable ControlVar, which controls whether each branch in the branch computation graph is segmented at the same time or not segmented at the same time through constraint formulas (16) and (17).

[0162] In the pipeline parallel strategy, the device cluster partitioning in this embodiment defaults to a homogeneous experimental environment, i.e., using communication links with the same bandwidth, such as a single machine with multiple GPUs as the experimental platform. Furthermore, since the device cluster is homogeneous, it can be evenly partitioned to maintain the same computing power for each device in each pipeline stage; therefore, this embodiment adopts an equal-partitioning strategy.

[0163] This embodiment further constructs an overall cost model by calculating the computation, communication, and storage overhead at the operator granularity level in the computation graph. The cost model can be mainly divided into two parts: First, execution overhead, which is the time overhead required during the actual execution of the model, mainly including the computational overhead of each operator and the communication overhead between operators. Second, storage overhead, which is the amount of data that needs to be stored during model execution. To ensure the normal execution of the model, the peak memory overhead during execution cannot exceed the storage limit of the hardware device. In a typical computation graph, the calculation formulas for the three types of overhead are as follows:

[0164] CompO i =Profile(w i ,op i d i (18)

[0165] PPComm ij =v ij / b ij +ε ij (19)

[0166] MemO i =w i +a i +g i +opt i (20)

[0167] As shown in equation (18), in order to evaluate operator O as accurately as possible... i Computational overhead CompO i In the specific implementation, the profiling method is used to measure the actual execution time cost of the operator, and the main factors considered include: the size of the operator weight parameter w. i Operator type op i and device type d i Equation (19) is used to calculate operator O. i With O j Communication overhead between PPComm ij This embodiment only considers the pipeline parallel strategy. Communication between operators mainly uses a pipeline-to-peer (P2P) communication method. The communication cost is calculated using the α-β model, and mainly includes two parts: the first part is v ij Operator O i With O j The amount of communication between them, b ij This represents the bandwidth of the underlying P2P communication link. The communication time overhead is obtained by dividing the communication volume by the communication bandwidth. However, to account for potential contention in the link during actual execution, an additional disturbance term ε is added. ijFinally, formula (20) calculates operator O. i Memory overhead MemO i It mainly consists of four parts: model weight parameters w i Activation value a i gradient g i and optimizer state opt i .

[0168] This embodiment freezes the original model parameters and introduces a matrix decomposition module for all matrix multiplication operators in the model, thus changing the original model's computational flow. For example... Figure 7 As shown, the pre-trained weights W of the model on the left are kept frozen. During the entire training process, only forward operations are needed. The parameters of the right module are adjusted during model training. In this embodiment, the corresponding overhead is calculated according to the following formula:

[0169] CompO i =CompW i +CompB i +CompA i (twenty one)

[0170] CompW i =ProfileForward(w i ,op i d i ) (twenty two)

[0171] CompB i =Profile(b i ,op i d i ) (twenty three)

[0172] CompA i =Profile(a i ,op i d i ) (twenty four)

[0173] MemO i =w i +MemB i +MemA i (25)

[0174] MemB i =wb i +ab i +gb i +optb i (26)

[0175] MemA i =wa i +aai +ga i +opta i (27)

[0176] Among them, CompO i For computation operator O in computation graph i Computational overhead, CompW i CompB i CompA i W i B i A i The computational overhead is represented by ProfileForward, where Profile represents the forward computation overhead, and Profile represents the sum of the forward and backward computational overheads. i For operator type, d i For device type. MemO i For operator O i The memory overhead is determined by the memory overhead w of the original parameter matrix W. i Memory overhead MemB for matrix factorization modules A and B i and MemA i It consists of three parts, of which w i wb i and wa i The memory overheads for the original parameter matrix W and the matrices A and B introduced by matrix decomposition are aa. i ab i The memory overhead of the forward activation values ​​of matrices A and B, respectively. i gb i Opta represents the memory overhead of the back gradients of matrices A and B, respectively. i optb i The memory overhead of saving the optimizer state required to update the parameters of matrices A and B respectively.

[0177] Formulas (21)-(24) are used to calculate the computational overhead of the operator. For the special structure of the matrix factorization module, the computational overhead is divided into two branches, left and right. The left branch only performs the forward process and does not need to update the weights w in the reverse direction. iTherefore, only the forward computation cost needs to be profiled; the right branch requires the entire computation cost of the forward and backward processes. The storage cost is calculated using equations (25)-(27). Similarly, since the left branch does not need to update parameters during the entire training process, only the operator weight parameters need to be stored, while the right branch is calculated with normal storage cost. Finally, the communication cost is not affected. After analysis, the pipeline split point cannot be located within the adapter structure. Assuming the split point is located within the LoRA adapter, compared with the original communication, not only is it necessary to transmit the output scale of the original matrix multiplication operator, but also to transmit the results of the calculation of operators B and A, which increases the communication volume. Therefore, this embodiment will exclude the case of pipeline splitting within the adapter structure.

[0178] This embodiment implements the dynamic partitioning of the computation graph and the construction of the operator-level cost model according to the above steps. For any model, the model computation graph can first be divided into different sets of operators using a dynamic partitioning method, thus forming various pipeline stages. Then, the overhead of the operators in each stage is summed to obtain the total overhead of the entire pipeline stage. As can be seen from the pipeline execution flow, the overall time overhead of the pipeline is limited by the stage with the largest time overhead in the pipeline. The execution overhead of each stage of the pipeline is obtained by summing the computation and communication overhead. In this embodiment, the specific expression for calculating the total overhead of the entire pipeline stage is as follows:

[0179]

[0180] Based on each Stage k Based on the corresponding computational and storage overhead, this embodiment constructs the final optimization objective:

[0181] Min Object = ∑ k CostStage k +(B-1)MAX{CostStage k} (29)

[0182] Where, ∑ k CostStage k This refers to the total time cost required to execute a single microbatch in the pipeline, where B is the maximum number of microbatches in the pipeline, and (B-1)MAX{CostStage k The term} represents the time cost of executing the remaining (B-1) microbatch.

[0183] The overall pipeline time overhead consists of two main parts. The first part is the total time overhead required to execute a single microbatch in the pipeline, and the second part is the time overhead required to execute the remaining B-1 microbatch. This time overhead is mainly determined by the characteristics of the pipeline and is limited by the stage with the highest time overhead in the pipeline. The execution overhead of each stage of the pipeline is obtained by simply summing the computation and communication overhead.

[0184] Specifically, by first extracting the model computation graph and constructing the computation graph, then evaluating it through profiling and calculating the cost of each operator, and finally using a solver to search for the global optimal solution, the required pipeline parallel strategy can be generated.

[0185] Step S04. Pipeline parallel reallocation: During model training, pipeline reallocation is inserted at model parameter refresh points to re-divide and allocate active partitions in the model to the corresponding devices until model training is completed.

[0186] In traditional pipeline freezing techniques, to reduce equipment idle time and improve equipment resource utilization, after model parameters are frozen, an attempt is made to re-search for pipeline parallel strategies, repartition the model, and allocate it to different devices. However, this method causes pipeline interruptions, generating significant additional time overhead. This embodiment addresses these issues by introducing matrix factorization nodes to decouple the originally large-scale model parameters from the update process. A novel pipeline parallel reallocation algorithm is then employed. Through overlapping computation and communication, the interruption time overhead caused by pipeline adjustments is reduced, improving model training efficiency and maximizing the benefits of model parameter freezing.

[0187] The new timing diagram based on the 1F1B GPipe pipeline parallel algorithm in this embodiment is as follows: Figure 11 As shown, compared to the ordinary 1F1BGPipe pipeline algorithm, this embodiment introduces an additional parameter freezing method. During model training, the first active node is used as the split point, and based on the parameter state, the model can be divided into two main parts: frozen and active. The frozen partition only requires forward computation, while the active partition requires backward gradient computation to update the parameters of the nodes in that partition. Figure 3 As shown, a large number of devices will be idle after the parameters are frozen. In order to eliminate the idle devices, this embodiment adopts a pipeline repartitioning strategy. Pipeline repartitioning is inserted at the model parameter refresh point to repartition the active partitions in the model and allocate them to the corresponding devices.

[0188] For the special model structure and training process that introduces matrix factorization nodes, this embodiment adopts the following pipeline repartitioning strategy:

[0189] First, the forward computation operation of the (t+1)th batch-size data in the frozen part is moved forward and inserted into the device bubble of the backward gradient calculation of the tth batch-size data, making full use of the device's idle time and preparing in advance for the calculation of subsequent batch-size data.

[0190] Second, due to the significant time overhead of the pipelined parallel strategy generation algorithm, this embodiment further configures the algorithm to reduce its invocation frequency. The goal of pipelined parallelism is to maintain load balancing among devices. Taking the Transformer model as an example, the model contains numerous repetitive structures and operators. This embodiment uses an equal division of operators as a lightweight pipelined parallelism strategy. During model training, the load status of each device is recorded in real time. When the load difference between devices is less than a certain threshold, the lightweight pipelined parallelism strategy with equal operator division is adopted; otherwise, the aforementioned pipelined parallelism strategy is invoked to rigorously re-divide the active region of the model.

[0191] Third, to further reduce the additional overhead of pipeline parallel repartitioning, this embodiment inserts a preprocessing stage after each batch-size data calculation is completed. Specifically, formula (12) is used to predict the upper limit of the number of nodes where parameters are frozen, thereby estimating the operators that may need to be transmitted. During the backward computation of the model, the communication operator is called in advance to adjust the distribution of model weight parameters. In this embodiment, due to the introduction of matrix factorization nodes, the original weight parameters of the model are coupled with the backward parameter update process, and the communication of model weight parameters and the gradient calculation and parameter update process of matrix factorization module parameters are overlapped. In this way, during pipeline redistribution, only a small number of matrix factorization module parameters need to be transmitted, which greatly reduces some additional communication and reduces the additional communication overhead caused by pipeline parallel repartitioning and redistribution.

[0192] This embodiment, designed for a special model structure and computational process that incorporates a parameter freezing algorithm, can fully exploit potential opportunities for model computation and communication overlap, reduce device idle time, and minimize the time overhead of pipeline repartitioning and reallocation, thereby maximizing the benefits of model parameter freezing and improving model training efficiency.

[0193] In summary, traditional pipelined parameter freezing techniques freeze model parameters at the model-level granularity. Overly coarse granularity can lead to overly aggressive parameter freezing strategies, reducing model performance. This invention employs a layered parameter freezing strategy, introducing matrix factorization modules to all matrix multiplication operators in the model and treating each module as a node. The parameter freezing process is divided into intra-node parameter freezing and inter-node parameter freezing strategies. Intra-node parameter freezing primarily reduces the rank of the matrix factorization module for each matrix multiplication operator. Different ranks are set based on the importance of operators at different positions in the model, improving model training efficiency while reducing the impact of parameter freezing on model performance. Furthermore, inter-node parameter freezing further refines the process into node-by-node freezing, reducing the possibility of premature parameter freezing and achieving operator-level, fine-grained parameter freezing.

[0194] In traditional pipeline freezing techniques, a significant amount of equipment becomes idle after model parameters are frozen. While traditional pipeline repartitioning and redistribution strategies can improve equipment utilization to some extent, they interrupt the original pipeline, introducing additional and substantial time overhead. This offsets some of the time benefits of model parameter freezing, reducing the practical application value of pipeline freezing techniques. This invention introduces matrix factorization nodes to decouple model parameters from gradient updates and reconfigures the corresponding computation and communication overlap methods. By inserting preprocessing operations during pipeline reverse computation and transmitting model parameters in advance, the communication overhead of pipeline repartitioning can be significantly reduced. Furthermore, the lightweight pipeline repartitioning algorithm designed based on the idea of ​​equal operator division further reduces the computational overhead caused by frequent model calls to pipeline repartitioning.

[0195] This embodiment also provides a computer system, including a processor and a memory, wherein the memory is used to store a computer program and the processor is used to execute the computer program to perform the method described above.

[0196] It is understood that the method described in this embodiment can be executed by a single device, such as a computer or server, or it can be applied to a distributed scenario where multiple devices cooperate to complete the task. In a distributed scenario, one of the multiple devices may execute only one or more steps of the method described in this embodiment, and the multiple devices interact to complete the method. The processor can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the method described in this embodiment. The memory can be implemented using read-only memory (ROM), random access memory (RAM), static storage devices, and dynamic storage devices. The memory can store the operating system and other applications. When the method described in this embodiment is implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0197] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A pipelined parallel training method for large models based on gradient-aware parameter freezing, characterized by the following steps: include: Rectangular decomposition: Low-rank matrix decomposition nodes are inserted at each matrix multiplication operator in the model to represent the increment matrix ΔW of the weights in the matrix multiplication operators. d1*d2 Perform low-rank decomposition of the matrix to obtain a set of matrices B. d1*R and A R*d2 Where R is the rank of the increment matrix, d1 and d2 represent the row and column dimensions of the increment matrix, respectively, and A and B represent the two decomposition matrices obtained from the decomposition. Matrix B... d1*R and A R*d2 Further decomposition yields the parameter {B} i A i }, where B i Representative matrix B d1*R The i-th column, A i Representative matrix A R *d2 The i-th row; Model parameter freezing: During the iterative training of the model, intra-node parameter freezing and inter-node parameter freezing are performed separately. When freezing intra-node parameters, the parameters {B} are obtained from the low-rank decomposition of each matrix node. i A i The gradient value control of the parameter {B} i A i } Freeze; When freezing the parameters between nodes, check whether each low-rank decomposition node of the matrix converges. If it converges, freeze. Pipeline parallelism strategy generation: Construct a computation graph for the model, where nodes represent data or computation nodes, edges represent data flow, and Boolean variables S are set in each edge to control whether the pipeline is split. i The Boolean variable S on each side controls i The computation graph is divided into different sets of operators to form each pipeline stage. The devices in each pipeline stage are divided based on the physical topology of the device cluster. An optimization objective is constructed based on the total cost of the computation graph in the entire pipeline stage. The optimal solution is obtained by solving the optimization objective as the generated pipeline parallel strategy. Pipeline parallel reallocation: During model training, pipeline reallocation is inserted at model parameter refresh points to re-divide and allocate active partitions in the model to the corresponding devices until model training is completed.

2. The large-scale model pipeline parallel training method based on gradient-aware parameter freezing according to claim 1, characterized in that, When freezing the intra-node parameters, the parameters {B} obtained from the low-rank decomposition of each matrix node are used as follows: i A i The gradient value of {B} is used to determine the importance of the parameter, and the gradient value of the parameter {B} is used to control the gradient of the parameter {B}. i A i The steps for freezing include: Based on parameter B in the decomposition matrix i A i Given the gradient values ​​of the parameters, calculate the 1-norm of the gradient value vector, and sum the values ​​to calculate the parameter {B}. i A i The absolute value of the gradient of the parameters}; Based on the parameter {B} in the two iterations i A i The gradient value of the parameter is used to calculate the rate of change of the parameter gradient. Use parameter {B i A i Historical gradient information for parameter {B} i A i The absolute value of the parameter gradient and the rate of change of the parameter gradient are subjected to sliding equilibrium processing to obtain the absolute value of the parameter gradient and the rate of change of the parameter gradient after sliding equilibrium. The importance value of the parameter is calculated based on the absolute value of the parameter gradient and the rate of change of the parameter gradient after the sliding equilibrium is reached. ; Determine the importance value of the parameter If the value of {B} is less than a preset threshold, then the parameter {B} is... i A i The result is frozen and merged into the model weights, and the rank of the decomposition matrix is ​​reduced by one.

3. The large model pipeline parallel training method based on gradient-aware parameter freezing according to claim 2, characterized in that, Calculate parameter {B i A i absolute value of the gradient of the parameter} The expression is: in, , Parameters , The absolute value of the gradient, , Representing parameters respectively , The 1-norm value of the gradient value vector; For parameter {B i A i The expression for applying the absolute value of the gradient of the parameter} to the sliding balance is: in, Denotes the parameter {B} in the t-th iteration. i A i The absolute value of the gradient of}, where t represents the number of iterations in training. This represents the absolute value of the parameter gradient after sliding equilibrium in the (t-1)th iteration. The hyperparameter representing the proportion controlling the sliding balance. This represents the absolute value of the parameter gradient after the sliding equilibrium is reached in the t-th iteration; The rate of change of the parameter gradient is calculated by dividing the difference between the norm 1 values ​​of the gradients of two consecutive iterations by the norm 1 value of the previous gradient. The calculation expression is as follows: in Let represent the parameters in the t-th iteration. The gradient rate of change, Denotes the parameter {B} in the t-th iteration. i A i The parameter gradient rate of change is calculated, and using the same sliding equilibrium concept as above, the parameter gradient rate of change after sliding equilibrium in the t-th iteration is obtained by utilizing historical gradient information. ; The importance value of the parameter is calculated based on the absolute value of the parameter gradient and the rate of change of the parameter gradient after the sliding equilibrium is reached. The calculation expression is: Where r represents the rank of the nodes in the low-rank decomposition of the matrix.

4. The large-scale pipelined parallel training method based on gradient-aware parameter freezing according to claim 1, characterized in that, When freezing the parameters between nodes, the step of determining whether each low-rank decomposition node of the matrix has converged, and freezing it if it has converged, includes: During model training, all active nodes are topologically sorted: {node1, node2, node3, ..., node...} n }, where n is the total number of active nodes, and the magnitude of the parameter gradient value of each active node in the model is calculated. }; Based on the topological sequence of active nodes, determine whether a node needs to be frozen from front to back according to the parameter gradient value of each active node. When multiple active nodes need to be frozen at the same time, determine the maximum number of nodes that can be frozen in the current iteration based on the number of active nodes and the freezing rate in the current iteration.

5. The large-model pipeline parallel training method based on gradient-aware parameter freezing according to claim 4, characterized in that, Calculate the absolute value of the parameter gradient of active nodes. The expression is: in, Indicates parameter {B i A i The absolute value of the gradient of the parameters}; Based on the absolute value of the parameter gradient of the active node Determine whether an active node needs to be frozen using the following formula: in, Indicates a preset threshold; And calculate the maximum number of nodes allowed to be frozen in the t-th iteration according to the following formula. : in, This represents the number of active nodes in the t-th iteration. This refers to the freeze rate.

6. The method for parallel training of large models based on gradient-aware parameter freezing according to any one of claims 1 to 5, characterized in that, The Boolean variable S on each control side i The steps to achieve dynamic partitioning of the computation graph include: When the computation graph is a linear computation graph, the total number of pipelines S and each Boolean variable S are... i The following constraints exist: When there are branches in the computation graph, the total number of pipeline stages that are split is calculated according to the following formula: And the configuration meets the following constraints: Among them, S in S represents the total number of pipeline stages accumulated at the current branch entry point. out This indicates the total number of pipeline stages after the current branch structure. Represents binary constraint variables. This represents a Boolean variable used to control whether to split the i-th branch on the j-th edge.

7. The method for parallel training of large models based on gradient-aware parameter freezing according to any one of claims 1 to 5, characterized in that, The computational and storage overhead during model training are calculated using the following formulas: in, For computation operator O in computation graph i The computational overhead, They represent , The computational overhead, The forward computation cost is represented by `Op`, the backward computation cost by `Profile`, and the forward and backward computation costs by `Op`. i For operator type, d i For equipment type, For operator O i The memory overhead is determined by the memory overhead of the original parameter matrix W. Memory overhead of matrix factorization modules A and B It consists of three parts, among which The memory overheads are the original parameter matrix W and the matrices A and B introduced by matrix decomposition, respectively. These represent the memory overhead of the forward activation values ​​of matrices A and B, respectively. These are the memory overheads of the backward gradients of matrices A and B, respectively. The memory overhead of saving the optimizer state required to update the parameters of matrices A and B respectively.

8. The large model pipeline parallel training method based on gradient-aware parameter freezing according to claim 7, characterized in that, The optimization objective is constructed based on the total overhead throughout the entire pipeline stage according to the computational graph, using the following formula: in, This represents the total time cost required to execute a single microbatch in the pipeline, where B is the maximum number of microbatches in the pipeline. The time cost of each pipeline stage is obtained by summing the computational cost of each operator and the communication cost between operators within that stage. It calculates the communication overhead between operators Oi and Oj.

9. The method for parallel training of large models based on gradient-aware parameter freezing according to any one of claims 1 to 5, characterized in that, The step of inserting pipeline repartitioning at the model parameter refresh point to repartition and allocate active partitions in the model to corresponding devices includes: moving the forward computation operation of the (t+1)th batch-size data in the frozen part forward and inserting it into the device bubble for the backward gradient calculation of the tth batch-size data; during model training, recording the load of each device, and when the load difference between devices is less than a preset threshold, adopting a lightweight pipeline parallel partitioning strategy with equal operator number; otherwise, calling a pipeline parallel strategy search algorithm to repartition the active area of ​​the model; inserting a preprocessing operation after each batch-size data is calculated, and using the preprocessing operation to predict the upper limit of the number of nodes with frozen parameters to estimate the potential operators that need to be transmitted; and calling the communication operator in advance during the backward computation of the model to adjust the distribution of model weight parameters.

10. A computer system comprising a processor and a memory, the memory being used to store computer programs, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 9.

Images