Deep learning compilation optimization method based on active transfer learning

Abstract

The invention provides a deep learning compilation optimization method based on active transfer learning, and belongs to the field of deep learning compilation, and the method comprises the following steps: S1, generating operator scheduling, and extracting features as training data; the operator scheduling is to transform the data operation process of operators on deployment hardware; s2, selectively labeling data by adopting an active transfer learning method and pre-training the performance prediction model; the output of the performance prediction model is the expected time required for the operator to execute on the deployed hardware; s3, inputting an operator to be compiled, and performing automatic compiling by using the pre-trained performance prediction model; according to the method, an active transfer learning method is utilized to optimize the automatic compilation process of deep learning, so that the cross-hardware portability of a deep learning compiler is improved, and the deployment cost of a deep learning model on different hardware platforms is reduced.

Description

Technical Field

[0001] The present application relates to the field of deep learning compilers, and in particular to a deep learning compilation optimization method based on active transfer learning. Background Art

[0002] With the continuous development of deep learning (DL), DL models have been widely used in many fields, including image recognition, natural language processing, smart cities, and autonomous driving. To improve the inference speed of DL models, optimizing the tensor functions within the models (such as convolution and matrix multiplication) and increasing their execution speed are key. However, hand-optimized libraries provided by hardware manufacturers (such as NVIDIA cuDNN and Intel oneDNN) are only targeted at the hardware they produce and rely on manual coding, resulting in long development cycles and slow update iterations.

[0003] The emergence of DL compilers addresses the challenges of manual optimization libraries. Based on heuristic search algorithms and learnable performance prediction models, DL compilers can automatically generate compiled, optimized tensor programs for a variety of hardware platforms. As a key component of DL compilers, performance prediction models influence the quality of the resulting tensor programs. As tensor program databases become increasingly comprehensive, offline performance prediction models based on database pre-training have become mainstream. Because they don't require hardware interaction during compilation, they significantly reduce compilation time.

[0004] However, offline performance prediction models rely on massive amounts of hardware measurement data. Faced with the increasing diversity of deployed devices, collecting this data requires significant time and computing resources. This is especially true for embedded or mobile devices with limited storage and computing resources. Pre-training can take weeks to complete, significantly slowing the deployment of DL models in various application scenarios. Therefore, optimizing performance prediction models and their training methods, improving the portability of DL compilers across different hardware platforms, and reducing the training time of offline performance prediction models have become research hotspots.

[0005] Based on the above background, the present invention proposes a deep learning compilation optimization method based on active transfer learning, which improves the compilation efficiency and deployment performance of deep learning models on different hardware platforms. Summary of the Invention

[0006] Technical problem: The present invention provides a deep learning compilation optimization method based on active transfer learning to solve the problems of long training data collection time and poor cross-hardware portability of DL compilers in the prior art.

[0007] Technical solution: The present invention adopts a deep learning compilation optimization method based on active transfer learning, which includes the following steps:

[0008] Step S1: Generate operator scheduling and extract features as training data; the operator scheduling is to transform the data operation process of the operator on the deployed hardware;

[0009] Step S2: Using active transfer learning methods to selectively label data and pre-train a performance prediction model; the output of the performance prediction model is the expected execution time of the operator on the deployed hardware;

[0010] Step S3: Input the operator to be compiled and automatically compile it using the pre-trained performance prediction model;

[0011] Step S4: Generate an executable program.

[0012] in,

[0013] In step S1, an operator schedule is generated, and the simulated annealing algorithm TVM is used to automatically search and generate the operator schedule in the schedule space. The operator schedule includes schedule information, and each schedule information includes three parts: schedule operation, schedule object and schedule parameter.

[0014] The scheduling operations include loop reordering, loop fusion, loop blocking, loop blocking adjustment, parallel computing, parameter annotation, reconstruction, compilation instructions, cache writing, serial computing, and inline function conversion.

[0015] The scheduling object refers to the object on which the scheduling operation is implemented.

[0016] The scheduling parameters are parameters assigned to the scheduling object by the scheduling operation.

[0017] Extracting features in step S1 uses the following steps to encode the scheduling information to extract features:

[0018] S11. Set the scheduling operation encoding to a one-hot code with a dimension of 1×11;

[0019] S12. Encode the scheduling objects into integers according to their order of appearance;

[0020] S13. Encode the scheduling parameters into integers according to their values;

[0021] S14. The three parts of each scheduling information are sequentially concatenated and padded to the set length;

[0022] S15. Concatenate all scheduling information codes in the operator scheduling in order.

[0023] The process of using active transfer learning method to selectively label data and pre-train the performance prediction model in step S2 is as follows:

[0024] S21: Use labeled data obtained from known hardware to train a performance prediction model as the teacher model used in subsequent knowledge transfer;

[0025] S22: Divide the data to be labeled into multiple subsets, randomly select 10% of the data from the first subset and label it on the target hardware to obtain its performance score;

[0026] S23: Transfer the labeled data from the teacher model to the performance prediction model, and update the performance prediction model parameters;

[0027] S24: Read the next subset of data to be labeled, use the performance prediction model to predict the performance of each data, and select 10% of the data that maximizes the diversity of performance scores for labeling on the target hardware;

[0028] S25: Repeat the above steps until there is no unread subset;

[0029] S26: Save the current performance prediction model.

[0030] In step S3, the operator to be compiled is input, and the process of automatic compilation using the pre-trained performance prediction model is as follows:

[0031] S31: Input the deep learning network model and split it into independent operators;

[0032] S32: Load the performance prediction model in step S2, use the simulated annealing algorithm TVM to search for possible operator scheduling in the operator scheduling space, extract features and input them into the performance prediction model, and use the prediction results output by the model as the objective function of the simulated annealing algorithm;

[0033] S33: Repeat step S32 until the number of searches reaches the set upper limit, and save the operator schedule with the highest performance score currently searched;

[0034] S34: Compile each operator of the deep learning network model according to the operator scheduling with the highest performance score, and use the underlying virtual machine LLVM or the unified computing device architecture CUDA backend compiler to convert the transformed deep learning network model into machine code that can be executed on the central processing unit CPU or graphics processing unit GPU.

[0035] Beneficial effects: The deep learning compilation optimization method based on active transfer learning provided by the present invention extracts operator scheduling as features, selects representative operator scheduling as training data through active learning sampling strategy, and compresses and trains the performance prediction model using the knowledge distillation method, which greatly reduces the amount of training data required by the DL compiler and the number of parameters and calculation amount of the performance prediction model, thereby improving its compilation efficiency; at the same time, the performance prediction model is trained using the transfer learning method between different hardware platforms, thereby improving the cross-hardware portability and compilation speed of the DL compiler, and reducing the deployment cost of the deep learning model on different hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of the deep learning compilation optimization method based on active transfer learning provided by the present invention;

[0037] Figure 2 is the transfer learning training framework provided by the present invention; wherein, the feature F T Output feature map for the teacher model extractor, feature F S Output feature maps for the student model extractor. DETAILED DESCRIPTION

[0038] To further illustrate the technical solutions provided by the present invention, the specific embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] Figure 1 The flowchart of the deep learning compilation optimization method based on active transfer learning provided by the embodiment of the present invention is as follows: Figure 1 As shown, it includes the following steps:

[0040] Step S1: Generate operator schedule and extract features as training data;

[0041] Among them, operator scheduling refers to the transformation of the data operation mode of the tensor program. For example, the matrix multiplication operator is transformed. Assume that the matrix multiplication operator is the multiplication of matrix A and matrix B. The size of matrix A and matrix B is 1024×1024. Before the transformation, the operation mode of the operator is a three-layer nested loop with a loop length of (1024, 1024, 1024); the loop is split and the loop is rearranged in sequence. The operation mode of the operator is changed to a four-layer nested loop with a loop length of (8, 1024, 1024, 128). In the embodiment of the present invention, the simulated annealing algorithm TVM is used to generate operator scheduling. Each step of the operator scheduling transformation is saved as a scheduling information. Each scheduling information contains three parts: scheduling operation, scheduling object and scheduling parameters. Scheduling operations include 11 operations: loop rearrangement, loop fusion, loop blocking, loop blocking adjustment, parallel computing, parameter annotation, reconstruction, compilation instructions, cache writing, serial computing, and inline function conversion. The scheduling object refers to the object on which the scheduling operation is implemented; the scheduling parameters are the parameters assigned to the scheduling object by the scheduling operation. Follow the steps below to encode the scheduling information to extract features:

[0042] S11. Encode the scheduling operation into a one-hot code of length 1×11;

[0043] S12. Encode the scheduling objects into integers according to their order of appearance;

[0044] S13. Encode the scheduling parameters into integers according to their values;

[0045] S14. Concatenate the three parts of each scheduling information code in sequence and fill them to a specific length;

[0046] S15. Concatenate all scheduling information codes in the operator scheduling in order.

[0047] Step S2: Use active transfer learning to selectively label data and train the performance prediction model;

[0048] Performance refers to the runtime of the deep learning model on the deployed hardware platform. The specific implementation of this step includes the following process:

[0049] S21: Use labeled data obtained from known hardware to train a performance prediction model, which will serve as the teacher model used in subsequent knowledge transfer. Read the training data in batches and use the performance prediction model to predict the performance of the training data.

[0050] The performance prediction model is a deep neural network, and its prediction result is the normalized operator runtime, which represents the acceleration effect of operator scheduling on the target hardware.

[0051] S22: Divide the data to be labeled into multiple subsets, randomly select 10% of the data from the first subset and label it on the target hardware to obtain its performance score, thereby maximizing data diversity;

[0052] The data selection algorithm is as follows:

[0053]

[0054]

[0055] In order to maximize data diversity, each data is regarded as a point in the high-dimensional feature space, and the prediction result is regarded as its position in the high-dimensional space. Each time data is selected, the center of the selected data is made to be farthest away from the center of the selected data. This can ensure that the distribution of data in the high-dimensional space covers as many areas as possible, and its corresponding performance range is also wider, ensuring the prediction accuracy of the performance prediction model for different features while greatly reducing the training data.

[0056] S23: Transfer the labeled data from the teacher model to the performance prediction model, and update the performance prediction model parameters;

[0057] Generate an executable program based on the selected data and measure its actual performance on the target hardware. This performance is used as the label for the sub-training set. The actual performance refers to the time required for the executable program to run on the hardware.

[0058] S24: Read the next subset of unlabeled data, use the performance prediction model to predict the performance of each data, select the 10% data that maximizes the diversity of performance scores and label them on the target hardware; the performance prediction model performs transfer learning based on the sub-training set;

[0059] The following describes the specific implementation of transfer learning. The goal of transfer learning is to quickly transfer a performance prediction model trained on a known hardware (Source Device) to a target hardware (Target Device), and ensure that the prediction accuracy of the model on the target hardware matches the prediction accuracy on the known hardware.

[0060] During the implementation of the performance prediction model training, a performance prediction model is first established. The model consists of a multi-layer neural network and is divided into two parts: extractor and predictor. The pre-trained parameters on the known hardware are obtained to initialize the performance prediction model to obtain the teacher model M. T , compress the performance prediction model on the target hardware and randomly initialize it to obtain the student model M S , where model M T The parameters of are frozen and not updated in subsequent steps;

[0061] Then, input N training data in batches:

[0062] [(x0,y0),(x1,y1),…,(x N-1 ,y N-1 )]

[0063] Where x is the sample feature and y is the label. The training data is input into the model M T The feature conversion part and model M S The feature conversion part is to obtain the embedding vector corresponding to the feature [Eb T,0 ,Eb T,1 ,…,Eb T,N-1 ] and [Eb S,0 ,Eb S,1 ,…,Eb S,N-1 Since the known hardware model is relatively complex and pre-trained on a large amount of data, the known hardware model has higher prediction accuracy and contains more valuable information. In order to make the performance of the target hardware model as close as possible to the known hardware model with only a very small amount of training data, the attention transfer method is used to make the attention distribution of the feature map of the student model similar to the attention distribution of the teacher model. The attention distribution is calculated as follows:

[0064]

[0065] where A∈R W×H×C , W×H is the feature map size, and C is the number of feature map channels.

[0066] Make the two distributions similar through attention loss:

[0067]

[0068] In order to make the model M S Learn more transferable knowledge, model M T and Model M S The extracted embedding vector should have more domain-independent parameters, and the domain adversarial loss is further introduced: first, the embedding vector is input into the discriminator fc to predict the domain to which the embedding vector belongs. If the embedding vector belongs to known hardware, the domain label is 1, otherwise it is 0. The adversarial loss function is

[0069]

[0070] The objective function of the extractor should be opposite to that of the domain classifier. Therefore, a gradient reversal layer (GRL) is added between the extractor and the domain classifier so that the gradient of the domain loss function becomes the opposite when it is back-propagated to the extractor. The expression of the gradient reversal layer is as follows:

[0071] -γ*grad=GRL(grad)

[0072] Where γ is a coefficient, and its value range is 0 to 1.

[0073] Input the embedding vector into the model M S The predictor fp S , predict its performance, the loss function of the predictor is

[0074]

[0075] Performance prediction model M T The overall loss function is

[0076] Loss=Domain loss+Attention loss+MSE loss

[0077] S25: Repeat the above steps until there is no unread subset; determine whether there is any unread training data, if there is unread data, jump back to S11, if there is no unread data, continue to the next step;

[0078] S26: Save the current performance prediction model.

[0079] Among them, torch.save is used to save the structure and parameters of the performance prediction model in pth file format.

[0080] Step S3: Automatically compile using the trained performance prediction model;

[0081] The specific implementation of this step includes the following process:

[0082] S31: Input the deep learning network model and split it into independent operators;

[0083] The input deep learning model is a torch or TensorFlow model.

[0084] S32: Load the performance prediction model in step S2, use the simulated annealing algorithm TVM to search for possible operator scheduling in the operator scheduling space, extract features and input them into the performance prediction model, and use the prediction results output by the model as the objective function of the simulated annealing algorithm;

[0085] The simulated annealing algorithm TVM scheduling space search module first converts the deep learning model in the torch / TensorFlow format into a tensor program. It then uses the simulated annealing algorithm to search for the parameters to be optimized in the tensor program and generate an operator schedule. The correspondence between the operator schedule and the performance of the tensor program is fitted using a performance prediction model. Guided by the performance prediction model, the simulated annealing algorithm searches for parameters with higher performance.

[0086] S33: Repeat step S32 until the number of searches reaches the set upper limit, and save the operator schedule with the highest performance score currently searched;

[0087] S34: Compile each operator of the deep learning network model according to the operator scheduling with the highest performance score, and use the underlying virtual machine LLVM or the unified computing device architecture CUDA backend compiler to convert the transformed deep learning network model into machine code that can be executed on the central processing unit CPU or graphics processing unit GPU.

[0088] Save the operator schedule with the best current prediction performance. The operator schedule is saved in JSON file format and an executable program is generated.

[0089] The specific implementation method of this step is to read the operator schedule saved in step S33, perform operator transformation on the input deep learning network model according to the schedule, and use the LLVM / CUDA back-end compiler to convert the transformed deep learning network model into machine code that can be executed on the CPU / GPU.

[0090] The present invention is described in detail through the above embodiments, but the contents are only exemplary embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. The scope of protection of the present invention is defined by the claims. Any use of the technical solution described in the present invention, or any person skilled in the art who, inspired by the technical solution of the present invention, designs a similar technical solution within the essence and scope of protection of the present invention to achieve the above technical effects, or any equivalent changes and improvements made to the scope of application, shall still fall within the scope of protection covered by the present invention.

Claims

1. A deep learning compilation optimization method based on active transfer learning, characterized in that: The following steps are involved: Step S1: Generate operator scheduling and extract features as training data; the operator scheduling is to transform the data operation process of the operator on the deployed hardware; Step S2: Using active transfer learning methods to selectively label data and pre-train a performance prediction model; the output of the performance prediction model is the expected execution time of the operator on the deployed hardware; Step S3: Input the operator to be compiled and automatically compile it using the pre-trained performance prediction model.

2. The deep learning compilation optimization method based on active transfer learning according to claim 1, characterized in that In step S1, an operator schedule is generated, and the simulated annealing algorithm TVM is used to automatically search and generate the operator schedule in the schedule space. The operator schedule includes schedule information, and each schedule information includes three parts: schedule operation, schedule object and schedule parameter.

3. The deep learning compilation optimization method based on active transfer learning according to claim 2, characterized in that: The scheduling operations include loop reordering, loop fusion, loop blocking, loop blocking adjustment, parallel computing, parameter annotation, reconstruction, compilation instructions, cache writing, serial computing, and inline function conversion.

4. The deep learning compilation optimization method based on active transfer learning according to claim 2, characterized in that The scheduling object refers to the object on which the scheduling operation is implemented.

5. The deep learning compilation optimization method based on active transfer learning according to claim 2, characterized in that: The scheduling parameters are parameters assigned to the scheduling object by the scheduling operation.

6. The deep learning compilation optimization method based on active transfer learning according to claim 1 or 2, characterized in that: Extract features in step 1. Use the following steps to encode the scheduling information to extract features: S11. Set the scheduling operation encoding to a one-hot code with a dimension of 1×11; S12. Encode the scheduling objects into integers according to their order of appearance; S13. Encode the scheduling parameters into integers according to their values; S14. The three parts of each scheduling information are sequentially concatenated and padded to the set length; S15. Concatenate all scheduling information codes in the operator scheduling in order.

7. The deep learning compilation optimization method based on active transfer learning according to claim 1, characterized in that The process of using active transfer learning in step 2 to selectively label data and pre-train the performance prediction model is as follows: S21: Use labeled data obtained from known hardware to train a performance prediction model as the teacher model used in subsequent knowledge transfer; S22: Divide the data to be labeled into multiple subsets, randomly select 10% of the data from the first subset and label it on the target hardware to obtain its performance score; S23: Transfer the labeled data from the teacher model to the performance prediction model, and update the performance prediction model parameters; S24: Read the next subset of data to be labeled, use the performance prediction model to predict the performance of each data, and select 10% of the data that maximizes the diversity of performance scores for labeling on the target hardware; S25: Repeat the above steps until there is no unread subset; S26: Save the current performance prediction model.

8. The deep learning compilation optimization method based on active transfer learning according to claim 1, characterized in that: In step S3, the operator to be compiled is input, and the process of automatic compilation using the pre-trained performance prediction model is as follows: S31: Input the deep learning network model and split it into independent operators; S32: Load the performance prediction model in step S2, use the simulated annealing algorithm TVM to search for possible operator scheduling in the operator scheduling space, extract features and input them into the performance prediction model, and use the prediction results output by the model as the objective function of the simulated annealing algorithm; S33: Repeat step S32 until the number of searches reaches the set upper limit, and save the operator schedule with the highest performance score currently searched; S34: Compile each operator of the deep learning network model according to the operator scheduling with the highest performance score, and use the underlying virtual machine LLVM or the unified computing device architecture CUDA backend compiler to convert the transformed deep learning network model into machine code that can be executed on the central processing unit CPU or graphics processing unit GPU.

Images