Multi-parameter adaptive heterogeneous parallel computing method for image classification

Abstract

The present invention provides a multi-parameter adaptive heterogeneous parallel computing method for image classification. First, the image data is copied into R copies, which are respectively input into R multi-parameter adaptive modules. The multi-parameter adaptive modules adopt heterogeneous parallelism. The multi-parameter adaptive modules adaptively select the optimal parallel mode according to the data type, data volume, video memory occupied by model parameters, batch size of model input and number of model layers, train the multi-parameter adaptive modules, and pass the prediction results of the trained multi-parameter adaptive modules into the main parameter server for voting, and output the optimal classification result. While maintaining the original deep learning training effect, the present invention can greatly improve the model training efficiency and improve the robustness of image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and specifically relates to a multi-parameter adaptive heterogeneous parallel computing method for image classification. Background Art

[0002] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained during the learning process is of great help in interpreting data such as text, images, and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and sounds.

[0003] In recent years, deep learning has achieved great success in various fields such as computer vision, natural language processing, autonomous driving, and smart medical care. At the same time, the continuous innovation of hardware devices, such as the emergence of hardware accelerators such as GPU, NPU, and TPU, and the open source of some natural image datasets such as ImageNet and CIFAR training datasets, have greatly promoted the rise and development of deep learning. However, with the rapid growth of deep neural networks and dataset sizes, the computing power of the machines used for training has become a bottleneck, and it takes days or weeks to complete the training of a large neural network model; for example, based on the ImageNet dataset, it takes about 50 hours to train ResNet-101 for 100 rounds using 4 P100 GPUs, which makes it impossible to interact in real time during the development of the network model. At the same time, there will be a problem that the number of single-layer network parameters is too large, resulting in the inability of a single card video memory to accommodate the model. In this case, distributed training is favored by people, which greatly improves the training efficiency and improves the efficiency of neural network training.

[0004] Currently, the industry generally adopts data parallel solutions to reduce training time, and model parallel solutions to solve the problem that the model is too large and cannot be trained on a single GPU. Data parallelism means that different GPUs have complete copies of the model; each machine only obtains different parts of the data, and the results of each GPU are combined in some way. Model parallelism means that different machines in a distributed system are responsible for the calculation of different parts of a single network. For example, each layer in a neural network can be assigned to a different GPU. During data parallel training, the parameter server is responsible for parameter exchange between models and updating local parameters. During model parallel training, different layers of the network model are split and deployed on different GPUs to reduce the number of model parameters on a single GPU, thereby avoiding video memory limitations.

[0005] However, the data and model parallel approach is to deploy the same network model on each GPU or deploy different parts of the same model on different GPUs. It does not take into account that a single model cannot handle image data with diverse features, which may lead to poor robustness of the model. In addition, the traditional model parallel solution officially integrated by the general neural network training framework PyTorch and TensorFlow cannot flexibly select the parallel mode, resulting in the model training effect and efficiency not meeting expectations. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a multi-parameter adaptive heterogeneous parallel computing method for image classification. By applying heterogeneous parallel training and adopting different parallel methods according to data characteristics, it can flexibly respond to changeable data and improve the efficiency and accuracy of image classification.

[0007] The present invention achieves the above technical objectives through the following technical means.

[0008] A multi-parameter adaptive heterogeneous parallel computing method for image classification is proposed. Image data is copied into R copies and respectively input into R multi-parameter adaptive modules. Heterogeneous parallelism is adopted between the R multi-parameter adaptive modules. The multi-parameter adaptive modules adaptively select the optimal parallel mode according to the data type, data volume, video memory occupied by model parameters, batch size of model input and number of model layers, train the multi-parameter adaptive modules, transmit the prediction results of the trained multi-parameter adaptive modules to the main parameter server for voting, and output the optimal classification result.

[0009] In the above technical solution, the parallel methods include data parallelism, model parallelism and hybrid parallelism.

[0010] In the above technical solution, the data parallelism adopts a parameter server structure, in which the globally shared network model parameters are stored. Each working node of the parameter server architecture stores a copy of the network model. The node communicates with the parameter server through push / pull operations, and no communication occurs between any nodes.

[0011] In the above technical solution, the data parallel training process is divided into forward propagation and back propagation. In the forward propagation stage, each node first reads the image data, performs forward propagation, obtains the predicted value, and calculates the error; in the back propagation stage, the node performs back propagation according to the error, calculates the gradient of each network layer, and the node pushes the gradient △W to the parameter server while calculating the gradient; after the parameter server receives the gradient △W sent by all nodes, it integrates the gradient using the formula W(t+1)=W(t)-α*1 / n*∑△W to obtain the network model parameter W(t+1) required for the t+1th iteration, and finally the node pulls the aggregated model parameter W(t+1) from the parameter server, updates the network model in the node, and performs t+1 data iterations; wherein α is the learning rate.

[0012] In the above technical solution, the node uses deep gradient compression in the process of pushing the gradient △W to the parameter server: first, the gradient is sparsely operated to retain the first Q vector values ​​with the largest absolute value in the gradient vector, and the other values ​​are set to zero, and then the quantization function quantiz is used to quantize the first Q vector values ​​with the largest absolute value in the gradient vector and convert them into low-precision values; when the gradient is uploaded to the parameter server, the original gradient is reconstructed through the inverse quantization function.

[0013] In the above technical solution, the parallel mode is selected based on the data feature D and the model feature M. The data feature D includes the data type D 1 and the amount of data D 2 The model feature M includes the memory occupied by the model parameters M 1 , batchsizeM of model input 2 and the number of model layers M 3 ;and: where λ i and μ j is the weight, and

[0014] In the above technical solution, the comprehensive indicators of data characteristics and model characteristics are:

[0015]

[0016] where τ 1 , τ 2 is the threshold value, is the indicator function.

[0017] In the above technical solution, when C=1, data parallelism is selected; when C=2, model parallelism is selected; when C=3, hybrid parallelism is selected.

[0018] The beneficial effects of the present invention are:

[0019] (1) The present invention adopts a heterogeneous parallel approach, i.e., multi-model parallelism, and finally obtains the optimal result through a competition mechanism, thus overcoming the problem of single parallelism, poor robustness, and inability to cope with data diversity.

[0020] (2) The multi-parameter adaptive module in the present invention adaptively selects the optimal parallel mode, namely data parallelism, model parallelism and hybrid parallelism, according to multiple parameters such as data type, data volume, video memory occupied by model parameters, model input batch size and number of model layers, thereby avoiding the trouble of manually selecting the parallel mode.

[0021] (3) The data parallel training process of the present invention includes back propagation, which adopts a method of deep gradient compression to overcome the disadvantage that gradient quantization can compress the gradient by a maximum of 32 times. The gradient is sparsely processed and then compressed and transmitted, which greatly reduces the communication time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of the multi-parameter adaptive heterogeneous parallel computing method for image classification according to the present invention;

[0023] Figure 2 This is a schematic diagram of the data parallel structure of the present invention;

[0024] Figure 3 This is a schematic diagram of the parallel structure of the model of the present invention;

[0025] Figure 4 This is a schematic diagram of the hybrid parallel distributed training architecture of the present invention;

[0026] Figure 5 This is a schematic diagram of the structure of the multi-parameter adaptive module of the present invention;

[0027] Figure 6 This is a structural diagram of the multi-parameter adaptive heterogeneous distributed training for image classification described in the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0029] This embodiment takes natural image classification as an example to introduce the content of the present invention in detail, but it is not limited to natural images and is also applicable to images in other fields.

[0030] like Figure 1 As shown, the present invention provides a multi-parameter adaptive heterogeneous parallel computing method for image classification, which specifically includes the following steps:

[0031] Step (1), constructing the network model structure. The network model includes two parts: feature extraction network and classification network. The feature extraction network consists of input layer, convolution layer, pooling layer, and fully connected layer, and the classification network consists of classification layer. The feature extraction network extracts features from the input image, and then inputs it into the classification network for classification to obtain the prediction result.

[0032] In this embodiment, the network model selects an image classification model, the feature extraction network selects ResNet101, SENet50 and SE_ResNeXt50, the loss function of the image classification model adopts multi-classification cross entropy loss, and the cross entropy calculation loss is calculated. The formula of multi-classification cross entropy loss is as follows:

[0033]

[0034] Where: L c represents multi-classification cross entropy loss, N represents the number of samples, K represents the number of sample categories, i represents the sample number, y represents the sample number, ik represents the true value of the i-th sample, Represents the predicted value of the i-th sample.

[0035] Step (2): Design a distributed data parallel training structure

[0036] The distributed data parallel training structure of the present invention adopts a parameter server structure, and the parameter server stores globally shared network model parameters. Each working node of the parameter server structure stores a copy of the network model. The node communicates with the parameter server through push / pull operations, and no communication behavior occurs between any nodes. A node is a GPU. Taking the tth iteration as an example, before training, the data is distributed according to the number of nodes, and each node obtains the same amount of data. The training process is divided into forward propagation and backward propagation; in the forward propagation stage, each node first reads the image data, performs forward propagation, obtains the predicted value, and calculates the error; in the backward propagation stage, the node performs backward propagation according to the error, calculates the gradient of each network layer, and the node calculates the gradient while pushing the gradient △W to the parameter server, so as to achieve a high degree of overlap between the gradient calculation and the communication operation to reduce the communication time. After the parameter server receives the gradient △W sent by all nodes, it integrates the gradient using the formula W(t+1)=W(t)-α*1 / n*∑△W to obtain the network model parameters W(t+1) required for the t+1th iteration. Finally, the node pulls the aggregated network model parameters W(t+1) from the parameter server, updates the network model in the node, and performs t+1 data iterations. In particular, in the process of pushing the gradient △W to the parameter server, a gradient deep compression strategy is adopted. Specifically, the gradient is first sparsely operated to retain the first Q vector values ​​with the largest absolute value in the gradient vector, and the other values ​​are set to zero. Then, the quantization function quantiz is used to quantize the first Q vector values ​​with the largest absolute value in the gradient vector and convert them into low-precision values. After the gradient is uploaded to the parameter server, the original gradient is reconstructed through the inverse quantization function. This strategy greatly reduces the amount of communication parameters, thereby reducing communication overhead and improving training efficiency. Distributed data parallel structures such as Figure 2 shown.

[0037] Step (3): Design a distributed model parallel training structure

[0038] When the image classification model is too large to be stored on one node (GPU), the classification network is split into several parts and deployed on different GPUs. The training is divided into two stages: forward propagation and back propagation. In the forward propagation stage, the GPUs cooperate with each other in sequence to perform forward propagation, obtain the predicted value, and use the loss function to calculate the corresponding error. If the neurons of the same layer in the classification network appear on different GPUs, the broadcast mechanism is used to distribute all input data to each GPU, and σ(W) is performed independently on each GPU. T X+b) calculation, the output feature maps are obtained respectively, and finally the channel addition operation is performed to obtain the output of this layer, where W Tis the weight matrix, b is the bias, X is the input data matrix, and σ() is the activation function; in the back-propagation stage, back-propagation is performed based on the error calculated by the forward propagation. In this process, if the neurons of the same layer are distributed on different GPUs, the gradients calculated by the neurons of this layer distributed on different GPUs are aggregated, and the gradient of this layer is calculated. Then, the gradient of this layer is passed back to the previous layer to continue back-propagation. When the back-propagation is completed, the parameters of the classification network are updated using the stochastic gradient descent method based on the calculated gradient, and the next data iteration is started. Here, each layer of the classification network can be deployed on different GPUs of the same machine or on different machine GPUs. The specific deployment is determined by the number of neurons, which greatly increases the network's ability to process data. Distributed model parallel structures such as Figure 3 shown.

[0039] Step (4): Design a hybrid parallel distributed training structure

[0040] Based on the data parallel structure designed in step (2) and the model parallel structure designed in step (3), a data parallel solution is adopted between different servers, and a network model parallel solution is adopted between different GPUs on the same server. During training, data parallelism is first performed to evenly divide the data and store the data in the server. Then, network model parallel training is performed on different GPUs on each server. The network model training here is exactly the same as step (3). The specific process of data parallelism is exactly the same as step (2). The only difference is that, under the premise of data parallelism, network model parallelism is performed in each node to reduce the pressure on a single GPU to store large network models. The hybrid parallel distributed training structure is as follows: Figure 4 shown.

[0041] Step (5): Design a distributed training structure for multi-parameter adaptive modules

[0042] The data parallel structure designed in step (2), the model parallel structure designed in step (3) and the hybrid parallel structure designed in step (4) form a multi-parameter adaptive module. The three branches of the multi-parameter adaptive module are data parallelism, model parallelism and hybrid parallelism. Data features and model features are important indicators for selecting appropriate branches.

[0043] The factors that affect data features mainly include data type and data size; data types include 2D and 3D images; Pascal VOC2012 is one of the benchmark data and is frequently used in object classification, object detection, image segmentation network comparison experiments and model effect evaluation, so the data size is based on the general Pascal VOC dataset with a size of 2G.

[0044] The factors that affect the model characteristics (i.e. the memory occupied by the model) mainly include the memory occupied by the model parameters, the batchsize of the model input, and the number of model layers; the memory occupied by the model parameters includes the memory occupied by the model's own parameters and the memory occupied by the optimizer parameters. The model's own parameters are mainly the parameters of the layers with parameters. The memory occupied by the layers with parameters is irrelevant to the input and will be occupied after the model is loaded. The layers with parameters mainly include convolutional layers, fully connected layers, BatchNorm layers, Embedding layers, etc. It should be noted that activation layers such as (Sigmoid / ReLU), pooling layers, and Dropout have no parameters; the specific calculation of the number of model parameters:

[0045] Fully connected layer: Linear (G->F): number of parameters: G*F; where G is the input shape and F is the output shape;

[0046] Convolutional layer Conv2d (C in , C out , S): Number of parameters: Cin*Cout*S*S; where C in is the number of input feature channels, C out is the number of feature channels of the output, and S is the size of the convolution kernel;

[0047] BatchNorm(Num): Number of parameters: 2*Num; Num is the length of input data;

[0048] Embedding(Z,H): Number of parameters: Z*H; where Z is the input data length and H is the data dimension;

[0049] Deep learning generally uses 32-bit floating point numbers, which is 4 bytes, so the size of the video memory occupied by the model's own parameters = the number of parameters * 4.

[0050] In addition to calculating the memory occupied by the model's own parameters, the corresponding optimizer parameters must also be considered (the optimizer parameters are gradients). This embodiment uses the SGD optimizer (i.e., multi-classification cross entropy loss). The formula for the SGD optimizer to update parameters is: It can be seen from the formula that in addition to saving the parameter W, the gradient used in the update process must also be saved Therefore, the video memory occupied by the parameters is: the size of the video memory occupied by the model's own parameters * 2 (the video memory occupied by the optimizer parameters is equal to the video memory occupied by the model's own parameters); α is the learning rate.

[0051] The intermediate parameters generated by the model calculation also occupy a large amount of video memory. The calculation formula for the intermediate parameters is as follows:

[0052] Fully connected layer: B*G*F, B is the batch size, G is the input shape, and F is the output shape;

[0053] The computational cost of convolution: B*H*W*C out *C in *S 2 , H*W is the output size, S is the convolution kernel size, C in is the number of input feature channels, C out is the number of feature channels of the output;

[0054] From the above two formulas, we can see that batchsize affects the size of video memory occupied by intermediate parameters.

[0055] ResNet was proposed in 2015 and won the first place in the classification task of the ImageNet competition. Because it is "simple and practical", many methods are built on the basis of ResNet50 or ResNet101, and are widely used in detection, segmentation, recognition and other fields. This embodiment uses the resNet152 layer as the benchmark model to calculate the size of the video memory occupied by the parameters and the number of model layers. The number of parameters of resNet152 is 60344387, and the parameter video memory occupies 230M of video memory, so the model video memory = 2*parameter layer video memory = 460M. In general, the size of a 3*256*256 image is 8M, and the intermediate parameters calculated by resNet152 occupy 27M of video memory. When batchsize=128, the occupied video memory size is 3G.

[0056] To summarize, the memory occupied by the model is expressed as follows: memory occupied by the model = memory occupied by model parameters + batchsize × memory occupied by each sample.

[0057] The thresholds and types of the main influencing factors of the multi-parameter adaptive module are shown in Table 1.

[0058] Table 1 Thresholds and types of main influencing factors of multi-parameter adaptive module

[0059]

[0060]

[0061] Table 1 can be expressed by the following formula:

[0062]

[0063]

[0064]

[0065] Threshold τ 1 Take 0.5, τ 2Take 0.5, is the indicator function. From the above formula, we can see that C is a comprehensive indicator used to measure data characteristics and model characteristics. Its value range is 1, 2, 3. The corresponding branch selection list is shown in Table 2:

[0066] Table 2 Branch selection list

[0067] C Branch type 1 Data Parallelism 2 Model Parallelism 3 Hybrid Parallel

[0068] The multi-parameter adaptive module structure is as follows Figure 5 shown.

[0069] Step (6): Design a multi-parameter adaptive distributed training structure for image classification

[0070] Based on the network model structure of step (1) and the multi-parameter adaptive module of step (5), the multi-parameter adaptive heterogeneous distributed structure includes three multi-parameter adaptive modules. The three model instances (ResNet101, SENet50 and SE_ResNeXt50) in step (1) are deployed to the three multi-parameter adaptive modules respectively. The local devices in each multi-parameter adaptive module include a GeForce GTX 1080Ti graphics card parameter server, three servers with two 8G memory cards and two servers with one 6G memory card. The training is mainly divided into two stages. In the first stage, the image data is first copied into three copies and input into the three multi-parameter adaptive modules respectively. The optimal parallel method is adaptively selected according to the parameters such as data type, data volume, memory occupied by model parameters, batch size of model input and number of model layers. After the training is completed, prediction is performed. In the second stage, the prediction results of each multi-parameter adaptive module are transmitted to the main parameter server for voting and the optimal result is output. In particular, the voting method is used here to obtain the final result. The integration method is applied to the distributed framework, which cleverly makes multi-model parallelization a reality. The multi-parameter adaptive heterogeneous distributed training structure is as follows: Figure 6 shown.

[0071] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A multi-parameter adaptive heterogeneous parallel computing method for image classification, characterized by: The image data is copied into R copies and respectively input into R multi-parameter adaptive modules. The R multi-parameter adaptive modules adopt heterogeneous parallelism. The multi-parameter adaptive modules adaptively select the optimal parallel mode according to the data type, data volume, video memory occupied by model parameters, batch size of model input and number of model layers, train the multi-parameter adaptive modules, and transmit the prediction results of the trained multi-parameter adaptive modules to the main parameter server for voting, and output the optimal classification results. The parallel mode is selected based on data features D and model features M, wherein the data features D include data type D1 and data volume D2, and the model features M include video memory occupied by model parameters M1, batch size M2 of model input, and number of model layers M3; and: where λ i and μ j is the weight, and The comprehensive indicators of data characteristics and model characteristics are: Among them, τ1 and τ2 are thresholds, is the indicator function.

2. The multi-parameter adaptive heterogeneous parallel computing method for image classification according to claim 1, characterized in that: The parallel methods include data parallelism, model parallelism and hybrid parallelism.

3. The multi-parameter adaptive heterogeneous parallel computing method for image classification according to claim 2, characterized in that: The data parallelism adopts a parameter server structure, in which the globally shared network model parameters are stored. Each working node of the parameter server architecture stores a copy of the network model. The node communicates with the parameter server through push / pull operations, and no communication occurs between any nodes.

4. The multi-parameter adaptive heterogeneous parallel computing method for image classification according to claim 3, characterized in that: The data parallel training process is divided into forward propagation and back propagation. In the forward propagation stage, each node first reads the image data, performs forward propagation, obtains the predicted value, and calculates the error. In the back propagation stage, the node performs back propagation according to the error, calculates the gradient of each network layer, and the node pushes the gradient △W to the parameter server while calculating the gradient. After the parameter server receives the gradient △W sent by all nodes, it integrates the gradient using the formula W(t+1)=W(t)-α*1 / n*∑△W to obtain the network model parameter W(t+1) required for the t+1th iteration. Finally, the node pulls the aggregated model parameter W(t+1) from the parameter server, updates the network model in the node, and performs t+1 data iterations. α is the learning rate.

5. The multi-parameter adaptive heterogeneous parallel computing method for image classification according to claim 4, characterized in that: In the process of pushing the gradient △W to the parameter server, the node adopts deep gradient compression: first, the gradient is sparsely operated to retain the first Q vector values ​​with the largest absolute value in the gradient vector, and the other values ​​are set to zero, and then the quantization function quantiz is used to quantize the first Q vector values ​​with the largest absolute value in the gradient vector into low-precision values; when the gradient is uploaded to the parameter server, the original gradient is reconstructed through the inverse quantization function.

6. The multi-parameter adaptive heterogeneous parallel computing method for image classification according to claim 1, characterized in that: When C=1, data parallelism is selected; when C=2, model parallelism is selected; when C=3, hybrid parallelism is selected.

Images