Model prediction-based accelerated AI large model distributed training method and device
By introducing model prediction methods and convolution + channel attention mechanisms into the distributed training of large AI models, the problems of high communication overhead and accuracy loss in gradient compression technology are solved, achieving efficient model training and accuracy preservation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU UNIVERSITY
- Filing Date
- 2023-09-01
- Publication Date
- 2026-04-17
AI Technical Summary
Existing gradient compression techniques suffer from high communication overhead and loss of model accuracy in distributed training of large AI models, making it difficult to maintain model accuracy while reducing communication overhead.
We adopt a model-based prediction approach, utilizing a prediction model structure based on a PS parameter server and a convolutional + channel attention mechanism. By constructing a hierarchical parameter dataset, we train the prediction model and deploy it in the PS parameter server to predict the remaining layer parameters of the target model, thereby reducing communication volume and improving model prediction efficiency and robustness.
It significantly reduces communication volume, improves model accuracy and training efficiency, while maintaining model precision and generalization.
Smart Images
Figure CN117273120B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of distributed training of large AI models, specifically relating to a method and apparatus for accelerating distributed training of large AI models based on model prediction. Background Technology
[0002] Distributed training is a popular and effective method for training large AI models (deep neural networks). However, communication bandwidth has become a bottleneck limiting training speed because a large number of gradient parameters need to be transmitted frequently in distributed training. To address this issue, researchers have extensively explored gradient compression techniques (e.g., gradient sparsity and quantization). The basic idea of this technique is to compress the gradients before sending them, which can effectively reduce the amount of gradient data that needs to be transmitted, thereby reducing communication overhead.
[0003] On the one hand, some gradient compression techniques reduce communication overhead by discarding some gradient information. However, because too much original gradient information is lost, these methods can impair model accuracy. Achieving better model accuracy naturally requires more training iterations, thus increasing training time. On the other hand, some gradient compression techniques reduce communication overhead by increasing computational overhead. For example, gradient sparsity requires introducing computational steps such as top-k gradient selection, gradient matrix recovery, and aggregation, resulting in significant increased computational cost.
[0004] Therefore, how to develop a distributed training method that maintains model accuracy while reducing communication overhead is an urgent problem to be solved. Summary of the Invention
[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method and apparatus for accelerating distributed training of large AI models based on model prediction. This invention is based on the architecture of "PS (Parameter server) + data parallelism" and utilizes a prediction model structure based on "convolution + channel attention mechanism", decoder, and evaluation mechanism based on prediction results to significantly reduce the communication volume of the model and improve the model prediction efficiency and robustness.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides a method for accelerating distributed training of large AI models based on model prediction, comprising the following steps:
[0008] Construct a hierarchical parameter dataset; using the hierarchical parameter dataset, train a prediction model for predicting the model parameters of the target model through an evaluation method based on the prediction results, and deploy it to the PS parameter server; in each sample of the hierarchical parameter dataset, the parameters of each layer are arranged in chronological order.
[0009] In one iteration of the distributed training of the target model, the computing node worker updates the model parameters of the first L2 layers in the total Q layers of the target model, and then pushes the model parameters of the first L2 layers to the PS parameter server; the PS aggregates the received model parameters, and the aggregated model parameters are shaped into multi-dimensional hierarchical parameter data; the target model is a neural network with Q layers;
[0010] Using multidimensional hierarchical parameter data, the prediction model performs an inference process to predict the model parameters of the target model's remaining layers except for the L2 layer. The predicted model parameters and the aggregated parameter data are concatenated to obtain a complete model parameter set, which is then pushed to all compute node workers by the PS parameter server. In the compute node workers, the received complete model parameters are used to iteratively replace the local model parameters.
[0011] Distributed training terminates after multiple iterations until the target model converges.
[0012] As a preferred technical solution, the construction of the hierarchical parameter dataset includes the following steps:
[0013] A portion of the model is sampled during the normal distributed training process of the target model; each round of the normal distributed training of the target model has M1 iterations, and a portion of the iterations is randomly sampled from M1; the parameters of the first L2 layers of the target model in each sampled iteration are used as a sample of the hierarchical parameter dataset to obtain an integer one-dimensional vector; the integer one-dimensional vector is appended in the order of the layers to obtain a two-dimensional parameter dataset; the parameters of the two-dimensional parameter dataset are sorted by layer and by time;
[0014] The two-dimensional parameter dataset is aggregated into a four-dimensional matrix to obtain a hierarchical parameter dataset.
[0015] As a preferred technical solution, the step of training the prediction model through an evaluation method based on the prediction results specifically involves:
[0016] Construct an evaluation neural network with the same structure as the target model. The model parameters of the evaluation neural network are derived from the model parameters predicted by the prediction model.
[0017] Given a task training dataset, a portion of the samples are randomly sampled from the total samples to obtain a test dataset for evaluating the neural network;
[0018] The test dataset is input into the evaluation neural network in batches, and an inference process is performed to obtain n predicted values.
[0019] The loss value is calculated by applying the loss function to the predicted value and the corresponding sample label value.
[0020]
[0021] Among them, y i This represents the label corresponding to sample i. Let represent the predicted value corresponding to sample i, and n represent the total number of predicted values;
[0022] The loss value is fed back to the prediction model, triggering a layer-by-layer gradient descent-based backpropagation process; when the loss value is less than a set threshold and / or the number of iterations reaches a set threshold, the prediction model converges, and a trained prediction model is obtained.
[0023] As a preferred technical solution, the prediction model includes a hidden layer and a decoder; the hidden layer includes a convolutional layer, a channel attention layer, and a pooling layer; the inference process of the prediction model is specifically as follows:
[0024] Convolution processing is performed on multidimensional hierarchical parameter data;
[0025] The processing result is nonlinearly activated, and the obtained activation value is input into the channel attention module to obtain a set of weight parameter vectors. This set of weight parameter vectors weights each channel of the input activation value.
[0026] The output values processed by the channel attention module are subjected to max pooling to obtain a set of predicted values.
[0027] As a preferred technical solution, the decoder of the prediction model includes a first fully connected layer and a second fully connected layer; the first fully connected layer is used to output the convolutional layer model parameters, and the second fully connected layer is used to output the fully connected layer model parameters; the decoder outputs the predicted model parameters by performing matrix multiplication with a preset weight matrix.
[0028] The decoder function includes decoding convolutional layer model parameters and decoding fully connected layer model parameters;
[0029] The decoding of the convolutional layer model parameters is achieved by rounding the features extracted from the hidden layer into a two-dimensional matrix, multiplying it with the first fully connected layer, and then performing non-linear activation to obtain activation values; rounding the activation values into a four-dimensional matrix and convolving it with the convolutional kernel to predict the model parameters of the convolutional layer; and then segmenting and rounding the predicted model parameters to obtain various model parameters of the convolutional layer.
[0030] The decoding of the fully connected layer model parameters involves setting a weight matrix with the same number of neurons as the total number of parameters in the fully connected layer to be decoded; multiplying the output activation value of the second fully connected layer with the set weight matrix; and then slicing and dimensionalizing the output value to obtain the predicted model parameters of the fully connected layer.
[0031] As a preferred technical solution, the integer is multi-dimensional hierarchical parameter data, specifically,
[0032] The two-dimensional parameter dataset is aggregated into a four-dimensional matrix (B, W, H, C), where B is the iteration number (i.e., the number of samples), W and H are the width and height of the matrix, and C represents the number of channels. When the model parameters are divisible by 64, the four-dimensional matrix is... A hierarchical parameter dataset is obtained; otherwise, the four-dimensional matrix is (B, 1, 1, W*H*C), thus maintaining the hierarchical characteristics of the dataset.
[0033] Secondly, the present invention also provides an accelerated distributed training system for large AI models based on model prediction, which is applied to the aforementioned accelerated distributed training method for large AI models based on model prediction, including a model building module, a first model training module, a second model training module, and an execution output module.
[0034] The model building module is used to construct a hierarchical parameter dataset; using the hierarchical parameter dataset, a prediction model for predicting the model parameters of the target model is trained through an evaluation method based on the prediction results, and deployed to the PS parameter server; in each sample of the hierarchical parameter dataset, the parameters of each layer are arranged in chronological order.
[0035] The first module of model training is used to update the model parameters of the first L2 layers in the total Q layers of the target model in one iteration of distributed training of the target model, and then push the model parameters of the first L2 layers to the PS parameter server; the PS aggregates the received model parameters, and the aggregated model parameters are shaped into multi-dimensional hierarchical parameter data; the target model is a neural network with Q layers.
[0036] The second module of model training is used to predict the model to perform an inference process using multi-dimensional hierarchical parameter data, and to predict the model parameters of the target model other than the L2 layer. The predicted model parameters and the aggregated parameter data are concatenated to obtain a complete model parameter, and the PS parameter server pushes the complete model parameter to all computing node workers. The local model parameters are iteratively replaced by the received complete model parameters in the computing node workers.
[0037] The execution output module is used to terminate distributed training after multiple iterations until the model converges.
[0038] Thirdly, the present invention provides an electronic device, the electronic device comprising:
[0039] At least one processor; and,
[0040] A memory communicatively connected to the at least one processor; wherein,
[0041] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the model prediction-based accelerated distributed training method for large AI models.
[0042] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the aforementioned method for accelerating distributed training of large AI models based on model prediction.
[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0044] (1) This invention introduces an evaluation mechanism for prediction results, which combines the predicted model parameters and the aggregated parameter data to obtain complete model parameters, thereby reducing the incremental computational overhead and improving the accuracy and efficiency of the model. At the same time, it adopts random sampling to select samples for use in the evaluation mechanism for prediction results, which improves the generalization of the model.
[0045] (2) This invention constructs a model structure and decoder based on “convolution + channel attention mechanism + pooling” to extract features and predict the model parameters of the target model. At the same time, the introduced channel attention mechanism strengthens the important channels and weakens the unimportant channels, which helps to extract features from the parameter dataset, thereby improving the performance of the prediction model while maintaining the accuracy of the model. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a flowchart of an embodiment of the present invention regarding a method for accelerating distributed training of large AI models based on model prediction;
[0048] Figure 2This is a schematic diagram of the overall training framework of the prediction model in an embodiment of the present invention;
[0049] Figure 3 This is a schematic diagram of the structure of the distributed training system for accelerated AI large models based on model prediction, according to an embodiment of the present invention.
[0050] Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0051] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.
[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0053] Please see Figure 1 This embodiment provides a method for accelerating distributed training of large AI models based on model prediction. The large AI model to be accelerated for distributed training is referred to as the target model. The target model is a neural network with Q layers. From left to right, these Q layers are referred to as front layers to back layers; that is, the left side of the multi-layer neural network is called the front layer, and the right side is called the back layer. This embodiment's method is based on a "PS parameter server + data parallelism" architecture, including multiple worker computing nodes and a parameter server (PS). The method specifically includes the following steps:
[0054] S1. Construct a hierarchical parameter dataset, ensuring that parameters of each layer and type in each sample are arranged in order, guaranteeing that the parameters are distinguishable, possessing hierarchical characteristics, and facilitating feature extraction; the construction process of the hierarchical parameter dataset includes the following steps:
[0055] S11. Sample a portion of the model from the normal distributed training process of the target model;
[0056] Normal distributed training of the target model consists of multiple epochs (one epoch traverses the entire dataset once), with M1 iterations per epoch. In this embodiment, L1% of the iterations are randomly sampled from M1. The parameters of the first L2 layers of the target model in each sampled iteration are treated as a sample. Each type of model parameter in each layer is integerized into a 1-dimensional vector, and then appended to the vector in layer order. Ultimately, a two-dimensional parameter dataset is obtained, where the first dimension is the number of iterations (i.e., the number of samples), and the second dimension is the number of model parameters. The parameters in this two-dimensional parameter dataset are sorted by layer and by time.
[0057] This embodiment demonstrates through experiments that randomly sampling 30% (i.e., L1=30) of the model parameters in each epoch can represent the overall training trend. This embodiment uses random sampling to enhance the generalization of the dataset.
[0058] Understandably, normal distributed training refers to fully distributed training without a model prediction mechanism. The main purpose of distributed training is to improve training speed and expand the model's capacity. By dividing the training data into multiple parts, each computed on a different computing device, more data samples can be processed simultaneously, thereby accelerating the training process.
[0059] S12. Aggregate the two-dimensional parameter dataset into a four-dimensional matrix to obtain a hierarchical parameter dataset.
[0060] Understandably, since the prediction model in this embodiment uses a convolutional neural network structure, it is necessary to reshape the two-dimensional parameter dataset collected in step S11 into a four-dimensional matrix to match the input dimension requirements of convolution operations. This four-dimensional matrix can be denoted as (B, W, H, C). Wherein, B is the iteration number (i.e., the number of samples), W and H are the width and height of the matrix, and C represents the number of channels of the matrix.
[0061] Generally speaking, the number of parameters per layer in most neural network models is divisible by 64, such as the classic VGGNet and ResNet. Therefore, as Figure 2 As shown, by reshaping the two-dimensional parameter dataset described in step S11 into (B, 8, 8, W*H*C / 64), a hierarchical parameter dataset can be obtained, in which each type of model parameter occupies a certain proportion of the number of channels and is arranged in the order of the layers. Figure 2 The dataset only shows one sample (i.e., B=1), while the datasets with multiple samples follow the same principle.
[0062] Furthermore, when the number of model parameters in each layer of the neural network model cannot be divided by 64, the two-dimensional parameter dataset described in step S11 is reshaped into (B, 1, 1, W*H*C) to maintain the hierarchical characteristics of the dataset. Then, the input features are extracted using a 1*1 convolutional kernel.
[0063] S2. Using a hierarchical parameter dataset, a deep neural network model (referred to as the prediction model) for predicting the model parameters of the target model is trained using an evaluation method based on the prediction results, and then deployed to the PS parameter server; the PS parameter server includes the prediction model.
[0064] For further details, please refer to the following: Figure 2 The prediction model adopts an architecture of "three hidden layers + one decoder", where the hidden layers are used to extract input features and the decoder is used to output model parameters.
[0065] For example, the first hidden layer can use a 3*3 kernel, and the other two hidden layers can use 3*3 kernels, with 8 kernels in each layer.
[0066] Alternatively, the first hidden layer can use a 1x1 kernel, and the other two hidden layers can use 3x3 kernels, with 8 kernels in each layer.
[0067] S21. The following description uses the calculation process of the first hidden layer as an example:
[0068] S211 uses convolutional layers to extract features from the input. Specifically, the operation involves sliding the convolution kernel to calculate the dot product of each region of the input feature map. Combined with the hierarchical parameter data of the input, the physical meaning of the convolution operation is that it can assign different weights to different types of input parameters in different layers. These weights are learnable and continuously optimized through gradient descent, thereby achieving the purpose of extracting important input parameters.
[0069] S212. After performing non-linear activation on the convolution result, the activation value is input into the channel attention SE (Squeeze and Excitation) module, which is also a neural network. Its output is a set of weight parameter vectors, the number of which is consistent with the number of input activation channels. This set of weight parameter vectors weights each channel of the input activation value, thereby strengthening important channels and weakening unimportant channels.
[0070] S213. The output value after processing by the SE module will be input into the pooling layer for max pooling to remove redundant features.
[0071] As can be seen from the calculation process of the hidden layer described above, the "convolution + channel attention mechanism + pooling" structure of the hidden layer helps to extract features from the parameter dataset, thereby improving the performance of the prediction model. Firstly, the input data is hierarchical parameter data, where each type of model parameter occupies a certain proportion of channels and is arranged in the order of the layers. Since an image is composed of RGB three channels, using convolution to extract features from the hierarchical parameter dataset is equivalent to using convolution to extract image features, and previous work has already demonstrated the effectiveness of convolution in extracting image features. Secondly, the SE module can selectively enhance the feature map after convolution, further improving the feature extraction capability. Finally, the max pooling operation plays a role in feature dimensionality reduction.
[0072] It is understandable that the above calculation process is the calculation process of the first hidden layer. The calculation processes of the second and third hidden layers are the same as above. Through the calculation of the three hidden layers, the input features can be effectively extracted.
[0073] S22. The decoder of the prediction model includes two fully connected layers. The first fully connected layer is used to output the convolutional layer model parameters; the second fully connected layer is used to output the fully connected layer model parameters; the decoder outputs the predicted model parameters by performing matrix multiplication with a preset weight matrix.
[0074] For example, let's denote the two decoders as D1 and D2, where D1 is used to output the convolutional layer model parameters. It has 64 neurons and outputs the model parameters by performing convolution with a 1x1 convolutional kernel. The other decoder, D2, is used to output the fully connected layer model parameters. It has 128 neurons and outputs the fully connected layer model parameters by performing matrix multiplication with a preset weight matrix. The specific process is as follows:
[0075] S221. Decoding the convolutional layer model parameters involves rounding the features extracted from the hidden layer into a two-dimensional matrix, multiplying it by the matrix of the first fully connected layer, and then performing non-linear activation to obtain activation values. The activation values are rounded into a four-dimensional matrix and convolved with the convolutional kernel to predict the model parameters of the convolutional layer. The predicted model parameters are then sliced and dimension-rounded to obtain various model parameters of the convolutional layer.
[0076] Specifically, D1 is multiplied by the output of the last hidden layer (which needs to be rounded to a 2D matrix), and after non-linear activation, the activation value is obtained. This activation value is rounded to a 4D matrix: (B, 8, 8, 1), and then convolved with a 1*1 convolution kernel to predict the model parameters of the convolutional layer. The predicted model parameters are then sliced and rounded to obtain various model parameters of the convolutional layer. For example, if the dimensions of the weights and biases of the convolutional layer to be predicted are (3, 3, 128, 256) and 256 respectively, a convolution kernel with a dimension of (1, 1, 1, 4612) is used to convolve with the above 4D activation value matrix, resulting in a parameter matrix with a dimension of (B, 8, 8, 4612). After converting this matrix to a 2D matrix, slicing it, and then rounding it again, the predicted convolutional layer weights and biases can be obtained. It is worth noting that in this process, it can be found that 8*8*4612 is exactly equal to (3*3*128*256+256). Therefore, the setting of the number of channels in the 1*1 convolutional kernel and the number of neurons in D1 must match the total number of predicted model parameters. Furthermore, the output model parameters will have B samples, while this invention only needs one. Therefore, this invention randomly selects one from the B output samples, which can improve the generalization of the prediction model.
[0077] S222. The decoding fully connected layer model parameters are defined by setting a weight matrix with the same number of neurons as the total number of parameters of the fully connected layer to be decoded; after performing matrix multiplication between the output activation value of the second fully connected layer and the set weight matrix, the output value is sliced and dimension-integrated to obtain the predicted model parameters of the fully connected layer.
[0078] Specifically, a weight matrix with the same number of neurons as the total number of parameters in the fully connected layer to be decoded is set. Then, the output activation values of decoder D2 are multiplied by the set weight matrix, and the output values are sliced and dimension-reshaped to obtain the predicted model parameters of the fully connected layer. Furthermore, if the number of parameters in the fully connected layer to be decoded is very large, the approach used to decode the convolutional layer model parameters can be adopted to output the fully connected layer model parameters, thereby reducing the size of the weight matrix.
[0079] S23. During the training process of the prediction model, this embodiment uses an evaluation network to evaluate the performance of the prediction model and help the prediction model converge. Specifically,
[0080] S231. Construct an evaluation neural network with the same structure as the target model. The model parameters of the evaluation neural network are derived from the model parameters predicted by the prediction model.
[0081] Furthermore, the samples in the hierarchical parameter dataset (the parameters W of the first L2 layer model) inThe input is fed into the prediction model, which performs one inference process (one forward propagation) to predict the model parameters of the subsequent (Q-L2) layers of the target model. like Figure 2 As shown, the present invention will W in and These parameters are then concatenated into a complete model parameter set. This is called an evaluation network.
[0082] S232. Given a task training dataset, randomly sample a portion of the total samples to obtain a test dataset for evaluating the neural network.
[0083] For a given task, such as classification tasks like MNIST and CIFAR-10, to reduce evaluation time, only L3% of the training images of each class in the training dataset of that task are randomly sampled as the training dataset. This training dataset is then input into the evaluation neural network in batches. The evaluation neural network performs one inference process, i.e., one forward propagation, and outputs a batch of predicted values. The batch of predicted values and the corresponding sample label values are used to calculate the loss value using the Loss function.
[0084] Furthermore, the randomly sampled data constitutes 25% of the training dataset for a given task.
[0085] S233. Input a set of samples from the test dataset into the evaluation neural network in batches and perform an inference process to obtain n predicted values.
[0086] S234. Calculate the loss value by combining the predicted value with the corresponding sample label value using the Loss function.
[0087]
[0088] Among them, y i This represents the label corresponding to sample i. Let represent the predicted value corresponding to sample i, and n represent the total number of predicted values;
[0089] S23. The loss value is fed back to the prediction model, triggering a layer-by-layer gradient descent-based backpropagation process; when the loss value is less than a set threshold and / or the number of iterations reaches the set threshold, the prediction model converges, and the trained prediction model is obtained.
[0090] Preferably, the threshold for the loss value is set to 0.001;
[0091] Preferably, the iteration round threshold is set to 30.
[0092] The loss value is fed back to the prediction model, triggering a layer-by-layer gradient descent-based backpropagation process to help optimize the prediction model. The iterative process described above, "forward propagation of the prediction model → forward propagation of the evaluation neural network → calculation of the loss value → backpropagation of the prediction model → forward propagation of the prediction model," is performed for E rounds. When the loss value is less than a set threshold or E reaches the set threshold, the training of the prediction model ends, that is, the prediction model converges.
[0093] S3. In one iteration of the distributed training of the target model, the computing node worker updates the model parameters of the first L2 layers in the total Q layers of the target model, and then pushes the model parameters of the first L2 layers to the PS parameter server; the PS aggregates the received model parameters, and the aggregated model parameters are shaped into multi-dimensional hierarchical parameter data; the target model is a neural network with Q layers.
[0094] Optionally, the number of layers in the model parameters of the first L2 layer in step S3 can be set to 5% to 10% of the total number of layers Q, or it can be set to 5% to 10% of the total number of model parameters.
[0095] Optionally, each compute node worker only sends the model parameters for the first layer, such as... Figure 1 The colored parts in the neural network model;
[0096] Optionally, in step S3, the updated target model parameters of the first L2 layer are aggregated in the PS parameter server. Specifically,
[0097] The average value of the target model parameters in the first L2 layer after the update is calculated in the PS parameter server.
[0098] S4. Using multidimensional hierarchical parameter data, the prediction model performs an inference process to predict the model parameters of the target model in all layers except L2. The predicted model parameters and the aggregated parameter data are concatenated to obtain a complete model parameter, which is then pushed to all compute node workers by the PS parameter server. The received complete model parameters are used to iteratively replace the local model parameters in the compute node workers.
[0099] Optionally, the predicted model parameters are the model parameters for the remaining layers, including various model parameters for convolutional layers and the predicted model parameters for fully connected layers, see [link to relevant documentation]. Figure 1 The white part in the neural network model.
[0100] Optionally, the prediction model in step 4 includes a hidden layer and a decoder; the hidden layer includes a convolutional layer, a channel attention layer, and a pooling layer; the inference process of the prediction model is specifically as follows:
[0101] Step 41. Perform convolution processing on the multidimensional hierarchical parameter data;
[0102] Step 42. Perform nonlinear activation on the processing result, input the obtained activation value into the channel attention module to obtain a set of weight parameter vectors, which weight each channel of the input activation value;
[0103] Step 43. Perform max pooling on the output values processed by the channel attention module to obtain a set of predicted values.
[0104] Step S5: After multiple iterations, distributed training terminates when the model converges.
[0105] Through laboratory testing and training, this invention achieves excellent prediction results. As shown in Table 1, training a downsized ResNet-18 model on the CIFAR-10 dataset demonstrates that this invention can significantly reduce communication overhead without sacrificing model accuracy. Compared to the Baseline, Top-k, and DGC methods, this invention reduces communication overhead by 88.1%, 40.8%, and 21.1%, respectively, and training time by 42.1%, 8.6%, and 14.5%, respectively, while improving model accuracy by 0.8%, 6%, and 5.8%, respectively.
[0106] Table 1 shows the experimental results of training a downsized ResNet-18 model on the CIFAR-10 dataset.
[0107]
[0108] *The hyperparameters of this invention are L1 = 30%, L2 = 6%, and L3 = 25%. The training accuracy of the prediction model of this invention is 73.97%.
[0109] Based on the same idea as the model prediction-based accelerated distributed training method for large AI models in the above embodiments, this invention also provides a model prediction-based accelerated distributed training system for large AI models. This system can be used to execute the above-described model prediction-based accelerated distributed training method for large AI models. For ease of explanation, the structural diagram of the embodiment of the model prediction-based accelerated distributed training system for large AI models only shows the parts related to the embodiments of this invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.
[0110] Please see Figure 3In another embodiment of this application, a method system 10 for accelerating distributed training of large AI models based on model prediction is provided. The system includes a model building module 11, a first model training module 12, a second model training module 13, and an execution output module 14.
[0111] The model building module 11 is used to build a hierarchical parameter dataset; using the hierarchical parameter dataset, a prediction model for predicting the model parameters of the target model is trained through an evaluation method based on the prediction results, and deployed to the PS parameter server; in each sample of the hierarchical parameter dataset, the parameters of each layer are arranged in chronological order.
[0112] The first module 12 of the model training is used to update the model parameters of the first L2 layers in the total Q layers of the target model in one iteration of the distributed training of the target model, and then push the model parameters of the first L2 layers to the PS parameter server; the PS aggregates the received model parameters, and the aggregated model parameters are shaped into multi-dimensional hierarchical parameter data; the target model is a neural network with Q layers.
[0113] The second module 13 of the model training is used to use multi-dimensional hierarchical parameter data to predict the model to perform an inference process, predict the model parameters of the target model other than the L2 layer; the predicted model parameters and the aggregated parameter data are concatenated to obtain a complete model parameter, and the PS parameter server pushes the complete model parameter to all computing node workers; the local model parameters are iteratively replaced by the received complete model parameter in the computing node worker.
[0114] The execution output module 14 is used to terminate distributed training after multiple iterations until the model converges.
[0115] It should be noted that the accelerated AI large model distributed training system based on model prediction of the present invention corresponds one-to-one with the accelerated AI large model distributed training method based on model prediction of the present invention. The technical features and beneficial effects described in the above embodiments of the accelerated AI large model distributed training method based on model prediction are applicable to the embodiments of the accelerated AI large model distributed training method based on model prediction. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.
[0116] Furthermore, in the above embodiments of the accelerated AI large model distributed training system based on model prediction, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the accelerated AI large model distributed training system based on model prediction is divided into different program modules to complete all or part of the functions described above.
[0117] Please see Figure 4 In one embodiment, an electronic device is provided for implementing a model prediction-based accelerated distributed training method for large AI models. The electronic device 20 may include a first processor 21, a first memory 22, and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a model prediction-based accelerated distributed training program for large AI models 23.
[0118] The first memory 22 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as the portable hard drive of the electronic device 20. In other embodiments, the first memory 22 can be an external storage device of the electronic device 20, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 20. Furthermore, the first memory 22 can include both internal storage units and external storage devices of the electronic device 20. The first memory 22 can be used not only to store application software and various types of data installed on the electronic device 20, such as the code of the accelerated AI large model distributed training program 23 based on model prediction, but also to temporarily store data that has been output or will be output.
[0119] In some embodiments, the first processor 21 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 21 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 22 and calls data stored in the first memory 22 to perform various functions of the electronic device 20 and process data.
[0120] Figure 4 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 4 The structure shown does not constitute a limitation on the electronic device 20, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0121] The distributed training program 23 for accelerating large AI models based on model prediction, stored in the first memory 22 of the electronic device 20, is a combination of multiple instructions. When run in the first processor 21, it can achieve the following:
[0122] Construct a hierarchical parameter dataset; using the hierarchical parameter dataset, train a prediction model for predicting the model parameters of the target model through an evaluation method based on the prediction results, and deploy it to the PS parameter server; in each sample of the hierarchical parameter dataset, the parameters of each layer are arranged in chronological order.
[0123] In one iteration of the distributed training of the target model, the computing node worker updates the model parameters of the first L2 layers in the total Q layers of the target model, and then pushes the model parameters of the first L2 layers to the PS parameter server; the PS aggregates the received model parameters, and the aggregated model parameters are shaped into multi-dimensional hierarchical parameter data; the target model is a neural network with Q layers;
[0124] Using multidimensional hierarchical parameter data, the prediction model performs an inference process to predict the model parameters of the target model's remaining layers except for the L2 layer. The predicted model parameters and the aggregated parameter data are concatenated to obtain a complete model parameter set, which is then pushed to all compute node workers by the PS parameter server. In the compute node workers, the received complete model parameters are used to iteratively replace the local model parameters.
[0125] Distributed training terminates after multiple iterations until the model converges.
[0126] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).
[0127] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0129] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for accelerating distributed training of large AI models based on model prediction, characterized in that, Includes the following steps: Construct a hierarchical parameter dataset; using the hierarchical parameter dataset, train a prediction model for predicting the model parameters of the target model through an evaluation method based on the prediction results, and deploy it to the PS parameter server; In each sample of the hierarchical parameter dataset, the parameters of each layer are arranged in chronological order; the dataset is an image dataset. In one iteration of the distributed training of the target model, the compute node worker updates the model parameters of the first L2 layers in the total Q layers of the target model, and then pushes the model parameters of the first L2 layers to the PS parameter server; the PS aggregates the received model parameters, and the aggregated model parameters are shaped into multi-dimensional hierarchical parameter data; The target model is a neural network with a Q layer; Using multidimensional hierarchical parameter data, the prediction model performs an inference process to predict the model parameters of the target model in all layers except the first L2 layer. The predicted model parameters and the aggregated parameter data are concatenated to obtain a complete model parameter, which is then pushed to all computing node workers by the PS parameter server. In the worker computing node, the received complete model parameters are used to iteratively replace the local model parameters; the prediction model includes a hidden layer and a decoder; the hidden layer includes a convolutional layer, a channel attention layer, and a pooling layer; the inference process of the prediction model is specifically as follows: Convolution processing is performed on multidimensional hierarchical parameter data; The processing result is nonlinearly activated, and the obtained activation value is input into the channel attention module to obtain a set of weight parameter vectors. This set of weight parameter vectors weights each channel of the input activation value. The output value processed by the channel attention module is subjected to max pooling to obtain a set of predicted values. Distributed training terminates after multiple iterations until the target model converges.
2. The method for accelerating distributed training of large AI models based on model prediction according to claim 1, characterized in that, The construction of the hierarchical parameter dataset includes the following steps: A portion of the model is sampled from the normal distributed training process of the target model; each round of the normal distributed training of the target model has M1 iterations, and a portion of the iterations is randomly sampled from M1. The parameters of the first L2 layer of the target model in each sampling iteration are treated as a sample of the hierarchical parameter dataset, resulting in an integer one-dimensional vector; the integer one-dimensional vector is appended in the order of the layers to obtain a two-dimensional parameter dataset; the parameters of the two-dimensional parameter dataset are sorted by layer and by time. The two-dimensional parameter dataset is aggregated into a four-dimensional matrix to obtain a hierarchical parameter dataset.
3. The method for accelerating distributed training of large AI models based on model prediction according to claim 1, characterized in that, The training of the prediction model through an evaluation method based on the prediction results specifically involves: Construct an evaluation neural network with the same structure as the target model. The model parameters of the evaluation neural network are derived from the model parameters predicted by the prediction model. Given a task training dataset, a portion of the samples are randomly sampled from the total samples to obtain a test dataset for evaluating the neural network; The test dataset is input into the evaluation neural network in batches, and an inference process is performed to obtain n predicted values. The loss value is calculated by applying the loss function to the predicted value and the corresponding sample label value. Among them, y i This represents the label corresponding to sample i. Let represent the predicted value corresponding to sample i, and n represent the total number of predicted values; The loss value is fed back to the prediction model, triggering a layer-by-layer gradient descent-based backpropagation process; when the loss value is less than a set threshold and / or the number of iterations reaches a set threshold, the prediction model converges, and a trained prediction model is obtained.
4. The method for accelerating distributed training of large AI models based on model prediction according to claim 1, characterized in that, The decoder of the prediction model includes two fully connected layers. The first fully connected layer is used to output the convolutional layer model parameters, and the second fully connected layer is used to output the fully connected layer model parameters. The decoder outputs the predicted model parameters by performing matrix multiplication with a preset weight matrix. The decoder function includes decoding convolutional layer model parameters and decoding fully connected layer model parameters; The decoding of the convolutional layer model parameters is achieved by rounding the features extracted from the hidden layer into a two-dimensional matrix, multiplying it with the first fully connected layer, and then performing non-linear activation to obtain activation values; rounding the activation values into a four-dimensional matrix and convolving it with the convolutional kernel to predict the model parameters of the convolutional layer; and then segmenting and rounding the predicted model parameters to obtain various model parameters of the convolutional layer. The decoding of the fully connected layer model parameters involves setting a weight matrix with the same number of neurons as the total number of parameters in the fully connected layer to be decoded; multiplying the output activation value of the second fully connected layer with the set weight matrix; and then slicing and dimensionalizing the output value to obtain the predicted model parameters of the fully connected layer.
5. The method for accelerating distributed training of large AI models based on model prediction according to claim 1, characterized in that, The integer type is multidimensional, hierarchical parameter data, specifically, The two-dimensional parameter dataset is aggregated into a four-dimensional matrix (B, W, H, C), where B is the iteration number (i.e., the number of samples), W and H are the width and height of the matrix, and C represents the number of channels in the matrix. When the model parameters are divisible by 64, the four-dimensional matrix is (B, 8, 8, ...). If the four-dimensional matrix is (B, 1, 1, W*H*C), then the hierarchical parameter dataset is obtained; otherwise, the four-dimensional matrix is (B, 1, 1, W*H*C), thus maintaining the hierarchical characteristics of the dataset.
6. The method for accelerating distributed training of large AI models based on model prediction according to claim 1, characterized in that, The PS aggregates the received model parameters, specifically, The average value of the received model parameters is calculated in the PS parameter server.
7. A distributed training system for large AI models based on model prediction, characterized in that: The accelerated distributed training method for large AI models based on model prediction, as described in any one of claims 1-6, includes a model building module, a first model training module, a second model training module, and an execution output module. The model building module is used to build a hierarchical parameter dataset; Using a hierarchical parameter dataset, a prediction model for predicting the model parameters of the target model is trained through an evaluation method based on the prediction results, and then deployed to the PS parameter server. In each sample of the hierarchical parameter dataset, the parameters of each layer are arranged in chronological order; the dataset is an image dataset. The first module of model training is used to update the model parameters of the first L2 layers in the total Q layers of the target model in one iteration of distributed training of the target model, and then push the model parameters of the first L2 layers to the PS parameter server; the PS aggregates the received model parameters, and the aggregated model parameters are shaped into multi-dimensional hierarchical parameter data. The target model is a neural network with a Q layer; The second module of model training is used to use multi-dimensional hierarchical parameter data to predict the model to perform an inference process and predict the model parameters of the target model other than the first L2 layer. The predicted model parameters and the aggregated parameter data are concatenated to obtain a complete model parameter, and the PS parameter server pushes the complete model parameter to all computing node workers. In the compute node worker, the received complete model parameters are used to iteratively replace the local model parameters; The execution output module is used to terminate distributed training after multiple iterations until the model converges.
8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, the computer program instructions being executed by the at least one processor to enable the at least one processor to execute the model prediction-based accelerated distributed training method for large AI models as described in any one of claims 1-6.
9. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the accelerated distributed training method for large AI models based on model prediction as described in any one of claims 1-6.