A compilation optimization method for dynamic shape operators

Abstract

The application provides a compiling optimization method for a dynamic shape operator, comprising the following steps: obtaining a neural network-based model to be compiled, determining the input shape of the dynamic shape operator in the model at the current execution time; inputting the input shape into a multi-classifier to obtain a predicted optimization scheme corresponding to the input shape, wherein the multi-classifier is obtained by pre-training a training set collected for the model, the training set comprises samples collected after simulating and optimizing all input forms of the dynamic shape operator of the model based on GPU hardware information, each sample comprises a determined input shape corresponding to a dynamic shape operator and a final optimization scheme selected according to the actual running situation of the optimization scheme on the CUDA core and the Tensor core of the GPU; and compiling the corresponding dynamic shape operator in the model according to the predicted optimization scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, in particular to the field of optimization of dynamic shape operators in machine learning, and more particularly to a compilation optimization method for dynamic shape operators. BACKGROUND

[0002] In recent years, the size and complexity of machine learning models have been increasing, requiring more powerful hardware to support their training and inference. Different models require different types of hardware accelerators, such as GPUs, TPUs, etc. In order to achieve optimal performance on different hardware, the concept of machine learning compiler has been proposed in the academic field. Machine learning compiler provides automatic optimization and adaptability, enabling models to better utilize the resources of different hardware platforms, thereby accelerating the training and inference process of machine learning models.

[0003] Operator auto-tuning is a core step in the workflow of machine learning compiler, and its purpose is to optimize the computational efficiency of different operators on specific hardware. Automatic optimization of operators with static shapes is relatively mature. However, in natural language processing related models, there are more and more operators with dynamic shapes, such as: the input sequence length of the Bert model, the change of the batch size (the number of samples in a training batch) will cause the change of the input and output dimensions of the query Q, key K and value V calculation in the Transformer structure, resulting in dynamic shape operators. The Transformer structure is the basis of the current mainstream models, and a large number of models contain the Transformer structure. In addition, there may be other forms of dynamic shape operators, so it is necessary to optimize dynamic shape operators, so that machine learning compiler technology can adapt to more advanced models.

[0004] Existing operator auto-tuning algorithms mostly support static shape operators (i.e. operators with fixed input tensor shape), and the optimization results for specific shapes can be applied to all inputs of the operator. However, for dynamic shape operators (i.e. operators with variable input tensor shape), applying such optimization schemes to other possible input shapes often cannot fully utilize device hardware resources.

[0005] Existing static shape operator auto-tuning schemes mainly include heuristic search algorithms and template-based search algorithms. Among them, the search time of heuristic search algorithms is usually hours, and even after a long time of search, it cannot guarantee to get a suitable search result; template-based search algorithms, although the time to search for a good tuning result is shorter, but it needs experts to complete the definition and design of templates, and the design cycle is long and has poor universality.

[0006] The existing solutions capable of completing dynamic shape operator optimization still mainly refer to the static operator optimization idea, including using a template-based search algorithm to obtain an optimization scheme for the maximum possible shape, and applying this optimization scheme to all possible shapes. In addition to inheriting the shortcomings of the template-based search algorithm, since it only selects one shape for optimization, it will have the shortcoming of poor optimization performance for other specific shapes. Another method is based on heuristic search algorithm in static operator optimization, by selecting some representative shapes in possible shapes (as much as possible to cover small and large size shapes) to optimize using heuristic search algorithm, obtaining multiple optimization schemes, and determining the specific optimization scheme by operator input shape at runtime. This scheme cannot efficiently solve the dynamic operator optimization problem due to the long time consumption of heuristic search and limited optimization size selection.

[0007] In addition, previous operator automatic optimization techniques are mainly implemented around the CUDA core of NVIDIA devices. CUDA core is a general-purpose computing core with low computing efficiency. To accelerate deep learning operators, NVIDIA subsequently introduced Tensor Core. Tensor Core utilizes the parallelism in matrix multiplication and the characteristics of tensor operations to achieve efficient computation acceleration through customized hardware design. The subsequent Bolt technology integrates NVIDIA Cutlass into the compiler, which can use Tensor Core in NVIDIA devices. NVIDIA Cutlass is a technology of an open source project, which is used to implement high-performance matrix multiplication and convolution operations inside the GPU. However, not all subgraphs that can be represented by Cutlass can achieve the best performance through Tensor Core. Although Tensor Core is designed specifically to accelerate deep learning and artificial intelligence (AI) workloads such as matrix operations, deep learning cannot be separated from CUDA core.

[0008] First, CUDA core is optimized for various parallel computing tasks and is more suitable for general-purpose parallel computing tasks. Deep learning tasks are not just matrix operations, but also have a lot of parallel computing. If the bottleneck is parallel computing, this type of deep learning task may be more suitable for CUDA core. Of course, Tensor core can also be used for general-purpose parallel computing tasks, but it may not be as efficient as CUDA core.

[0009] Secondly, although the Tensor core is good for General Matrix Multiplication (GEMM) calculation, the input and output data types of the Tensor core need to be half-precision or single-precision, and the dimension of the matrix is preferably a multiple of 8, which limits the use of the Tensor core.

[0010] In summary, the prior art still has the following problems:

[0011] (1) Hardware information is not considered in the tuning algorithm, so the hardware characteristics cannot be efficiently utilized;

[0012] (2) Automatic tuning caused by too long heuristic search time consumes a lot of time;

[0013] (3) Resource scheduling for specific input shapes is not considered.

[0014] It should be noted that the background art is only used to introduce the relevant information of the present application, so as to help understand the technical solutions of the present application, but it does not mean that the relevant information must be the prior art. The relevant information is submitted and disclosed together with the present application scheme, and in the absence of evidence that the relevant information has been disclosed before the filing date of the present application, the relevant information should not be regarded as prior art. SUMMARY

[0015] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a compilation optimization method for dynamic shape operators.

[0016] The purpose of the present application is achieved by the following technical solutions:

[0017] According to the first aspect of the present application, a compilation optimization method for dynamic shape operators is provided, comprising: obtaining a neural network-based model to be compiled, determining the input shape of the dynamic shape operator in the model at the current execution; inputting the input shape into a multi-classifier to obtain a predicted optimization scheme corresponding to the input shape, wherein the multi-classifier is pre-trained using a training set collected for the model, the training set including samples collected after simulating optimization of all dynamic shape operators of the model in multiple input forms based on GPU hardware information, each sample including a determined input shape corresponding to a dynamic shape operator and an optimal final optimization scheme selected according to the actual running situation of the optimization scheme on the CUDA core and the Tensor core of the GPU; and compiling the corresponding dynamic shape operator in the model according to the predicted optimization scheme.

[0018] Optionally, the training set of each model is collected in the following manner: obtaining a neural network-based model to be compiled, hardware information of a GPU used to execute the model, and a preset tuning threshold, splitting the model into a plurality of calculation sub-graphs according to a model calculation graph, each calculation sub-graph representing an operator, wherein the hardware information includes a total cache level, a minimum memory storage unit, a maximum cache capacity of each layer, and a maximum thread number; collecting various dynamic shape operators from the plurality of calculation sub-graphs to form a first set; for each operator in the first set, collecting possible input shapes of the operator to form a second set corresponding to the operator, wherein if a total number of possible input shapes of the operator is less than or equal to the tuning threshold, the second set collects all the input shapes, otherwise the second set collects a plurality of sampled input shapes; combining each input shape of the operator in the second set corresponding to each operator with corresponding calculation logic into a complete operator; optimizing each complete operator to obtain a corresponding final optimization scheme, wherein for optimization of a CUDA core, hierarchical optimization is performed according to the cache level, each layer of optimization selecting a splitting manner from a plurality of splitting manners set based on the minimum memory storage unit to split an axis related to the complete operator stored in a previous layer to obtain a first optimization scheme, and for a Tensor core, a second optimization scheme is generated using a BOLT technology, and from the first optimization scheme and the second optimization scheme, a scheme that minimizes the running time of the complete operator is selected as the final optimization scheme, wherein the first optimization scheme needs to be verified to meet the requirements of the maximum cache capacity of each layer and the maximum thread number; combining each input shape and the corresponding final optimization scheme into a sample to form a training set. This technical solution can at least achieve the following beneficial technical effects: when the number of possible shapes of the operator input is greater than the tuning threshold, sampling optimization is used, and subsequently a classification algorithm can be used to complete mapping of all input shapes and optimization schemes, more comprehensive and balanced samples of dynamic shape operators with a large number of possible input shapes are collected, so as to realize more accurate prediction subsequently; when the number of possible shapes of the operator input is less than the tuning threshold, full-amount optimization is used, thus for small sample optimization schemes, all possible workloads can be covered, and the prediction effect is better; the running performance of the two optimization schemes on different CUDA cores and Tensor cores is actually simulated to determine the core type and optimization scheme that can more efficiently process each input shape operator.

[0019] Optionally, the plurality of input shapes sampled are sampled from the dynamic range corresponding to the dynamic shape operator according to a preset sampling interval, wherein the sampling interval is determined according to the upper limit value of the dynamic range and the tuning threshold. The technical solution can at least achieve the following beneficial technical effects: the sampling interval is determined by the tuning threshold, and the plurality of input shapes are sampled from the dynamic range according to the sampling interval, which can avoid the problem that only one shape is selected for optimization in the prior art, and more comprehensive and balanced samples of the dynamic shape operator with more possible input shapes can be collected, so that more accurate prediction can be realized subsequently, and the poor optimization performance for specific input shapes is solved.

[0020] Optionally, the first optimization scheme for the CUDA core optimization includes an initial splitting form corresponding to the memory and a splitting mode corresponding to each layer cache, wherein: all splitting factors in the initial splitting form corresponding to the memory are set to 1 to split each axis, and the initial splitting form of the memory is used to split and load operator-related data stored in the memory to the cache of the previous layer; the splitting mode of each layer cache is used to split and load operator-related data stored in the current layer cache to the cache of the previous layer, and the splitting mode of each layer cache is selected from different splitting modes, each splitting mode includes splitting factors corresponding to all axes, the i-th axis in the i-th splitting mode is the associated axis, the splitting factor of the associated axis is set to an integer multiple of the minimum storage unit of the memory, and the splitting factors of the other axes except the associated axis are set to 1, and the value of the splitting factor of the associated axis in each splitting mode is increased after each layer cache is optimized. The technical solution can at least achieve the following beneficial technical effects: each splitting mode only adjusts the splitting factor of one associated axis, which can avoid overly complex splitting schemes that are difficult to implement, thereby efficiently comparing multiple splitting modes.

[0021] Optionally, the splitting mode of each layer cache is obtained in the following manner: the axes of the complete operator are processed according to each splitting mode, and the change rate of the data reuse rate of the cache corresponding to each splitting mode after processing is determined; the splitting modes are selected in turn according to the order from large to small of the change rate of the data reuse rate, and it is determined whether the selected splitting mode is reasonable for the current cache level according to a preset verification mechanism, and if the splitting mode is reasonable, the splitting mode is selected as the splitting mode of the cache of the layer, wherein the verification mechanism includes verification 1 and verification 2, verification 1 is used to determine whether the data block size loaded into the next level cache according to the splitting mode is smaller than the maximum capacity of the next level cache, and verification 2 is used to determine whether the number of threads corresponding to the complete operator after splitting according to the splitting mode is smaller than the maximum number of threads supported by the hardware, and if the results of verification 1 and verification 2 are yes, it indicates that the splitting mode is reasonable.

[0022] Optionally, the method further comprises: obtaining the final optimization scheme of all dynamic shape operators, comparing whether the splitting modes of each layer in each two final optimization schemes are completely consistent, and if yes, merging the repeated final optimization schemes.

[0023] Optionally, the plurality of classifiers is trained based on an XGBoost algorithm, and an objective function of the XGBoost algorithm includes a training loss for guiding the model to correctly classify and a regularization term for controlling complexity of the model.

[0024] According to a second aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is configured to store executable instructions; and the one or more processors are configured to implement the steps of the method of the first aspect by executing the executable instructions. BRIEF DESCRIPTION OF DRAWINGS

[0025] The embodiments of the present application will be further described below with reference to the drawings, in which:

[0026] Figure 1 A flowchart of a compiling optimization method for a dynamic shape operator according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0028] As mentioned in the background section, the prior art does not take into account the hardware information in the tuning algorithm, so it cannot efficiently utilize the hardware characteristics, and the automatic tuning consumes a lot of time and does not have resource scheduling for specific input shapes. In order to efficiently automatically tune the dynamic shape operator, the inventors propose a fast compiling optimization method for a dynamic shape operator, which collects a training set composed of samples collected after simulating optimization considering the hardware information of the GPU for various input forms of the dynamic shape operator, each sample including a specific input shape corresponding to a dynamic shape operator and an optimal final optimization scheme selected according to the actual running situation of the optimization scheme on the CUDA core and the Tensor core of the GPU, which can efficiently utilize the hardware information to determine the final optimization scheme for a specific input shape, thereby constructing a training set and training a plurality of classifiers to predict the optimization scheme for a specific input shape, which can speed up the training or inference process of the machine learning model.

[0029] According to an embodiment of the present application, referring to Figure 1 A compiling optimization method for a dynamic shape operator, comprising: S100, S200, S300, S400, S500, S600, S700 and S800. In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0030] S100: obtaining a neural network-based model to be compiled, hardware information of a GPU for executing the model, and a preset tuning threshold, splitting the model into a plurality of calculation sub-graphs according to a model calculation graph, each calculation sub-graph representing an operator, wherein the GPU comprises a CUDA core and a Tensor core, and the hardware information comprises a total cache level.

[0031] According to an embodiment of the present application, the entire model is first input, the model is split into a plurality of calculation sub-graphs according to a model calculation graph, and each calculation sub-graph represents an operator; a model compilation platform (TVM platform is selected in this embodiment) is obtained, and hardware information and a tuning threshold (used to limit the number of shapes for tuning) are configured in the platform. Preferably, the hardware information comprises a total cache level H, a maximum cache capacity of each layer, a minimum memory storage unit (data access amount of a memory transaction), and a maximum thread number. Alternatively, the hardware information comprises one or a combination of the foregoing elements.

[0032] According to an example of the present application, the model Bert_base is taken as a tuning object (i.e., a neural network-based model to be compiled), and the RTX4090 of Nvidia is taken as a deployment device; the tuning threshold n = 64 and the hardware information of the RTX4090 are configured, and the total cache level (H = 2, the second layer cache refers to the second cache, and the first layer cache refers to the first cache), the maximum cache capacity of each layer, the minimum memory storage unit, and the maximum thread number of the RTX4090 are configured according to the actual situation of the RTX4090.

[0033] S200: collecting various dynamic shape operators from the plurality of calculation sub-graphs to form a first set.

[0034] According to an embodiment of the present application, a first set initially empty is first established, dynamic shape operators are collected by traversing the plurality of calculation sub-graphs, and whether an operator is a dynamic shape operator is determined according to whether the shape input by the operator represented by the calculation sub-graph is a fixed value, and if the operator is a dynamic shape operator, the operator is stored in the first set for subsequent tuning.

[0035] S300: for each operator in the first set, collecting possible input shapes of each operator to form a second set corresponding to the operator, wherein if the total number of input shapes of the operator is less than or equal to the tuning threshold, all input shapes are collected in the second set, otherwise a plurality of sampled input shapes are collected in the second set.

[0036] According to one embodiment of the present application, S300 is used to determine the workload that needs to be optimized, wherein each dynamic shape operator obtained in the first set is taken out respectively, the input shape range of each dynamic shape operator is given, if the number of input shapes that the input shape can appear (i.e. the total number of input shapes of the operator) is less than the given optimization threshold, all shapes are optimized, and all possible input shapes are simulated; if the input shape is greater than the given threshold, sampling is performed according to a certain interval to cover all possible workloads, at most a threshold number of shapes are selected to obtain a shape set: the second set [α1, α2, α3, α4, …, α n ]. Preferably, the plurality of input shapes sampled are obtained by sampling from the dynamic range corresponding to the dynamic shape operator according to a preset sampling interval, wherein the sampling interval is determined according to the upper limit value of the dynamic range and the optimization threshold. For example, the sampling interval is equal to the upper limit value of the dynamic range divided by the optimization threshold to obtain the result of the quotient after taking the integer part downward, and the sampling interval is at least 2. This embodiment sets the optimization threshold, when the number of possible shapes of the operator input is greater than the optimization threshold, sampling optimization is used, and subsequent mapping of all input shapes and optimization schemes can still be completed by using the multi-classifier, when the number of possible shapes of the operator input is less than the optimization threshold, full optimization is used, so as to optimize as many different input forms of the same dynamic shape operator as possible.

[0037] S400: The operator of each input shape in the second set of each operator is combined with the calculation logic to form a complete operator.

[0038] According to one embodiment of the present application, step S400 is the process of operator combination, which includes taking out each input shape α n obtained in step S300, combining the input shape with the calculation logic of the operator, and obtaining a complete operator. Illustratively, taking the Matmul operator as an example, the operator calculation expression is:

[0039] Input * weight = output

[0040] Wherein, input is an input tensor, the corresponding input shape is [S, 768], S is a dynamic input size, weight is a weight tensor, the shape is [768, 768], output is an output tensor, and the shape is [768, S]. The input shape range (dynamic change range) of S is 1-128.

[0041] S500: optimizing each complete operator to obtain a corresponding final optimization scheme, wherein, for the CUDA core, each layer of the cache level selects a splitting mode from the preset plurality of splitting modes that can make the change rate of the data multiplexing rate highest to split the data related to the complete operator stored in the previous layer to obtain a first optimization scheme, and a second optimization scheme is generated by calling the BOLT optimization technology for the Tensor core, and the scheme that makes the runtime delay of the complete operator smallest is selected as the final optimization scheme from the first optimization scheme and the second optimization scheme. The first optimization scheme needs to be verified to meet the requirements of the maximum cache capacity and the maximum thread number of each layer.

[0042] According to an embodiment of the present application, the first optimization scheme for optimizing the CUDA core includes an initial splitting form corresponding to the memory and a splitting mode corresponding to each layer of cache, wherein: all the splitting factors in the initial splitting form corresponding to the memory are set to 1 to split each axis, and the initial splitting form of the memory is used to split and load the operator-related data stored in the memory to the cache of the previous layer; the splitting mode of each layer of cache is used to split and load the operator-related data stored in the current layer of cache to the cache of the previous layer, and the splitting mode of each layer of cache is selected from different plurality of splitting modes, each splitting mode includes a splitting factor corresponding to all axes, the i-th axis in the i-th splitting mode is the associated axis, the splitting factor of the associated axis is set to an integer multiple of the minimum storage unit of the memory, and the splitting factors of other axes except the associated axis are set to 1. After optimizing each layer of cache, the value of the splitting factor of the associated axis in each splitting mode is increased.

[0043] According to an embodiment of the present application, for the complete operator of step S400, step S500 optimizes as follows:

[0044] Step S501: extract the spatial axis (axis appearing in the calculation result tensor) and the reduction axis (axis appearing in the calculation process but not appearing in the calculation result) in the operator, and for the global memory to the H layer of cache, the splitting factor of 1 is used to represent the splitting of each axis (i.e. the original tensor is split according to the size of 1 for each dimension, and the split data is sent to the H layer of cache); for example: extract the spatial axis (axis appearing in the calculation result tensor) M, N and the reduction axis (axis appearing in the calculation process but not appearing in the calculation result) K in the operator, and for the memory to the second layer of cache, the splitting factor of 1 is used to represent the splitting of each axis m=1, n=1, k=1 (i.e. split input according to the shape of [m, k], and send data to the second layer of cache, split weight according to the shape of [n, k], and send data to the second layer of cache, split output according to the shape of [m, n], and send data to the second layer of cache.

[0045] Step S502: increase the split factor, for each axis in the ith split mode, ensure that the associated axis A i and other axes A0A1…A i-1 A i+1 …A n-1 A n remain unchanged, increase the split factor Z i of the axis A i by an integer multiple of the memory transaction, to obtain a split mode represented by a set of split factors (f0, f1, …, f i-1 , Z i , f i+1 , …, f n-1 , f n ), wherein Z i represents the split factor corresponding to the associated axis, f0, f1, …, f i-1 , f i+1 , …, f n-1 , f4 represents the split factor corresponding to the other axis; taking the Nvidia RTX4090 as an example, the minimum data quantity of memory read and write is 4 bytes, then m=4, the remaining split factors remain unchanged, to obtain the split combination sf1:m=4, n=1, k=1; increase the split factor n, increase the split factor of the axis by an integer multiple of the memory transaction, to obtain n=4, the remaining split factors remain unchanged, to obtain the split combination sf2:m=1, n=4, k=1; increase the split factor k, increase the split factor of the axis by an integer multiple of the memory transaction, to obtain k=4, the remaining split factors remain unchanged, to obtain the split combination sf3:m=1, n=1, k=4;

[0046] Step S503: calculate the change of data multiplexing rate after the split factor of each associated axis is changed, calculate the change rate R i of the data multiplexing rate after the split factor of any axis is changed, and the calculation method of R i is as follows:

[0047]

[0048] Wherein Q represents the data flow of the split mode before changing the split factor of a certain axis, Q new represents the data flow of the split mode after changing the split factor of a certain axis, F represents the cache usage of the split mode before changing the split factor of a certain axis, and F new represents the cache usage of the split mode after changing the split factor of a certain axis.

[0049] Step S504: sorting, sort the split modes from large to small according to the change rate of the data multiplexing rate, and the subsequent algorithm will preferentially calculate based on the split mode with the highest change rate of the data multiplexing rate;

[0050] Step S505: verifying whether the current splitting mode is reasonable at the current cache level, including verifying whether the data block size loaded to the H-level cache after splitting according to the splitting mode is smaller than the maximum capacity of the H-level cache, and verifying whether the number of threads is smaller than the maximum number of threads supported by hardware; if not, the splitting mode is discarded;

[0051] Step S506: verifying whether the calculation delay of the current splitting mode at the current cache level is greater than the memory delay; if the calculation delay is greater than the memory delay, it indicates that the optimization of the cache is completed; this step is used to verify whether the memory delay is hidden;

[0052] Step S507: optimizing the previous cache layer by entering step S503 until the optimization of all levels of cache is completed (from H-1 layer to 1 layer in turn), that is, obtaining the optimization scheme γ1 of the CUDA core of the input shape α1;

[0053] Step S508: obtaining the CUDA source code of the optimization scheme γ1 using the build interface (Build interface) in the TVM platform, and then obtaining the running time T1 of the optimization scheme γ1 of the shape α1 using the running interface (Run interface);

[0054] Step S509: Tensor core performance data collection, for the input shape α1, generating a Cutlass operator using the Bolt technology, and obtaining the running time T2 of the Cutlass operator using the running interface (Run interface) of the TVM platform;

[0055] Step S510: comparing the running time T1 of the operator of the input shape α1 in the CUDA core with the running time T2 in the Tensor core, and selecting the scheme with smaller running time as the final optimization scheme β1 of the input shape;

[0056] Step S511: repeating steps S501-S507 to obtain the optimization β i for any input shape α i obtained in step S300, and thus the input shape set [α1, α2, α3, α4, …, α n of step S300 has a corresponding final optimization scheme [β1, β2, β3, β4, …, β n ], since there are several input shapes whose optimization schemes are the same, if the splitting factors of each layer are the same, the two optimization schemes are considered to be the same, the optimization schemes are de-duplicated to obtain k different final optimization schemes [β1, β2, β3, β4, …, β k ], wherein k≤n.

[0057] S600: Form a training set by taking the input shape as the independent variable and the corresponding final optimization scheme as the dependent variable, train a multi-classifier using the training set, and obtain the trained multi-classifier.

[0058] According to one embodiment of the present application, the training set of each model is collected in the following manner: obtaining a neural network-based model to be compiled, hardware information of a GPU used to execute the model, and a preset tuning threshold, splitting the model into a plurality of calculation subgraphs according to the model calculation graph, each calculation subgraph representing an operator, wherein the hardware information includes total cache level, memory minimum storage unit, maximum cache capacity of each layer, and maximum thread number; collecting various dynamic shape operators from the plurality of calculation subgraphs to form a first set; for each operator in the first set, collecting possible input shapes of each operator to form a second set corresponding to the operator, wherein if the total number of possible input shapes of the operator is less than or equal to the tuning threshold, the second set collects all input shapes, otherwise the second set collects a plurality of sampled input shapes; combining each input shape operator in the second set corresponding to each operator with the corresponding calculation logic into a complete operator; optimizing each complete operator to obtain a corresponding final optimization scheme, wherein for CUDA core optimization, hierarchical optimization is performed according to cache level, each layer optimization selects a splitting mode that can make the change rate of data reuse rate highest from a plurality of splitting modes based on memory minimum storage unit to split the axis related to the complete operator stored in the previous layer to obtain a first optimization scheme, and for Tensor core, a second optimization scheme is generated using BOLT technology, and the scheme that makes the running time of the complete operator minimum is selected from the first optimization scheme and the second optimization scheme as the final optimization scheme, wherein the first optimization scheme needs to be verified to meet the requirements of the maximum cache capacity and the maximum thread number of each layer; combining each input shape and the corresponding final optimization scheme to form a sample to form a training set. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: when the number of possible input shapes of the operator is greater than the tuning threshold, sampling optimization is used, and subsequent mapping of all input shapes and optimization schemes can be completed using a classification algorithm, more comprehensive and balanced collection of samples of dynamic shape operators with more possible input shapes is achieved, so that more accurate prediction can be realized subsequently; when the number of possible input shapes of the operator is less than the tuning threshold, full-amount optimization is used, thereby, for small sample optimization schemes, all possible workloads can be covered, and the prediction effect is better; in addition, the running performance of the two optimization schemes on different CUDA cores and Tensor cores is actually simulated to determine the core type and optimization scheme that can more efficiently process each input shape operator.

[0059] According to one embodiment of the present application, a multi-class classifier is trained using XGBoost, taking input shapes as independent variables and corresponding optimized combinations / schemes as dependent variables. Optionally, the objective function of XGBoost consists of two parts: a training loss function L and a regularization function Ω.

[0060] Obj(Θ) = L(Θ) + Ω(Θ)

[0061] It should be understood that different weights can also be assigned to L(Θ) and Ω(Θ), and the objective function is the weighted sum of the two.

[0062] Step S601: Calculate the training loss. For a multi-class problem with k classes, for each instance i, the training loss function is as follows:

[0063]

[0064] where y ik is an indicator function, y ik = 1 when instance i (i.e. the input shape in sample i) belongs to class k, otherwise 0; p ik is the probability that the model predicts instance i to belong to class k.

[0065] Step S602: Calculate the prediction probability p ik by the softmax function:

[0066]

[0067] where φ ik represents the logit value (raw output score) of the probability that the model predicts instance i to belong to class k, φ ij represents the logit value of the probability that the model predicts instance i to belong to class j, and K represents the total number of classes.

[0068] Step S603: Calculate the gradient matrix and Hessian matrix. In order to optimize the objective function, the gradient matrix and Hessian matrix of each instance i and class k need to be calculated.

[0069] Gradient matrix:

[0070]

[0071] Hessian matrix:

[0072]

[0073] The derivatives in these matrices are used to build new decision trees to optimize the objective function in each iteration;

[0074] S604: Construct and update the decision tree, use the calculated gradient matrix and Hessian matrix to construct a new decision tree, and update the model parameters Θ, and the leaf node score is updated:

[0075]

[0076] where R j represents the instance set contained in leaf node j, and λ is a regularization parameter;

[0077] S605: Calculate the regularization term, the regularization term Ω(Θ) is used to control the complexity of the model:

[0078]

[0079] where T represents the number of leaf nodes of the tree, and w j represents the weight of leaf node j, and γ and λ represent regularization parameters;

[0080] S606: Update the objective function, which is the sum of the training loss and the regularization term:

[0081] Obj(Θ)=L(Θ)+Ω(Θ)

[0082] S607: Iterative training, repeat steps S601 to S606 until the objective function converges or the training reaches a predetermined number of iterations.

[0083] It should be understood that other forms such as random forest can also be used for multi-classifier.

[0084] S700: Use the trained multi-classifier to predict the actual input shape of the dynamic shape operator to obtain the corresponding predicted optimization scheme.

[0085] At this point, a multi-classifier that selects a suitable optimization scheme according to the input shape of the operator is obtained, and in the future, the corresponding predicted optimization scheme of the operator for each input shape can be predicted according to the multi-classifier, and the mapping from the operator shape to the optimization scheme can be automatically completed, thereby completing the operator optimization task.

[0086] S800: Compile the corresponding dynamic shape operator in the model according to the predicted optimization scheme.

[0087] After obtaining the predicted optimization scheme corresponding to a certain dynamic shape operator for a certain input shape, the dynamic shape operator can be compiled according to the predicted optimization scheme with the help of the TVM platform.

[0088] In general, the compiling optimization method for dynamic shape input operators provided by the application has high efficiency: combined with hardware characteristics, hardware performance can be fully utilized, an operator with dynamic shape input can be automatically selected during model runtime with dynamic shape input, sampling can be automatically performed to cover a variety of dynamic shape workloads, and the optimization work of the operator with dynamic shape input can be efficiently completed; the method has high expandability: hardware adaptation can be quickly completed by only configuring hardware information; and the method also has universality: efficient optimization schemes can be generated for mainstream computing-intensive operators such as Dense and Convolution.

[0089] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above specific order, and in fact, some of the steps can be performed concurrently or even in a changed order, as long as the desired function can be achieved.

[0090] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0091] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magneto-optical or other optical device, a portable storage device, an memory card, a solid-state memory device, or any suitable combination of the foregoing. A computer readable storage medium can be any tangible device or medium that can retain and store instructions for use by an instruction execution device.

[0092] Embodiments of the application have been described above, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements in the market of the embodiments, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A compilation optimization method for dynamic shape operators, characterized in that, include: Obtain the neural network-based model to be compiled and determine the input shape of the dynamic shape operator within it during the current execution. The input shape is input into a multi-classifier to obtain the corresponding prediction optimization scheme. The multi-classifier is pre-trained using a training set collected for the model. The training set includes samples collected after simulating and optimizing various input forms of all dynamic shape operators of the model based on GPU hardware information. Each sample includes a specific input shape corresponding to a dynamic shape operator and the final optimization scheme selected based on the actual running performance of the optimization scheme on the GPU's CUDA cores and Tensor cores. The training set for each model is collected as follows: The neural network-based model to be compiled, the hardware information of the GPU used to execute the model, and the preset tuning threshold are obtained. The model is split into multiple computational subgraphs according to the model's computational graph. Each computational subgraph represents an operator. The hardware information includes the total cache levels, the minimum memory storage unit, the maximum cache capacity per layer, and the maximum number of threads. Various dynamic shape operators are collected from the multiple computational subgraphs to form a first set. For each operator in the first set, the possible input shapes for each operator are collected. A second set corresponding to each operator is formed. If the total number of possible input shapes for an operator is less than or equal to a tuning threshold, the second set collects all input shapes; otherwise, it collects multiple sampled input shapes. The operator for each input shape in the second set corresponding to each operator is combined with the corresponding computational logic to form a complete operator. Each complete operator is optimized to obtain the corresponding final optimization scheme. For CUDA core optimization, layered optimization is performed according to cache levels. For each layer, a partitioning method that maximizes the rate of change in data reuse is selected from multiple partitioning methods based on the minimum memory storage unit setting to partition the axes related to the complete operator stored in the previous layer to obtain a first optimization scheme. For Tensor cores, a second optimization scheme is generated using BOLT technology. The scheme that minimizes the runtime of the complete operator is selected from the first and second optimization schemes as the final optimization scheme. The first optimization scheme needs to be verified to meet the requirements of maximum cache capacity and maximum number of threads per layer. Each input shape and its corresponding final optimization scheme are combined into a sample to form a training set. The corresponding dynamic shape operators in the model are compiled according to the prediction optimization scheme.

2. The method according to claim 1, characterized in that, The multiple input shapes are sampled from the dynamic range corresponding to the dynamic shape operator at preset sampling intervals, wherein the sampling interval is determined based on the upper limit of the dynamic range and the tuning threshold.

3. The method according to claim 1, characterized in that, When optimizing the CUDA core, the first optimization scheme includes the initial memory partitioning and the partitioning method for each cache level, where: All splitting factors in the initial splitting form corresponding to memory are set to 1 to split each axis. The initial splitting form in memory is used to split the operator-related data stored in memory and load it into the upper-level cache. The partitioning method of each cache layer is used to partition the operator-related data stored in the current cache layer and load it into the cache layer above. The partitioning method of each cache layer is selected from a variety of different partitioning methods. Each partitioning method includes the partitioning factor corresponding to all axes. The i-th axis in the i-th partitioning method is the associated axis. The partitioning factor of the associated axis is set to an integer multiple of the minimum storage unit in memory. The partitioning factor of other axes besides the associated axis is set to 1. After each cache layer is optimized, the value of the partitioning factor of the associated axis in each partitioning method is increased.

4. The method according to claim 3, characterized in that, The partitioning method for each cache level is obtained as follows: Process the axes of the complete operator according to each segmentation method, and determine the rate of change of the data reuse rate of the cache after processing according to each segmentation method; The partitioning methods are selected sequentially in descending order of the rate of change of data reuse rate. A preset verification mechanism is used to determine whether the selected partitioning method is reasonable at the current cache level. If reasonable, the partitioning method is adopted as the partitioning method for that cache level. The verification mechanism includes Verification 1 and Verification 2. Verification 1 is used to determine whether the size of the data block loaded into the next level cache according to the partitioning method is less than the maximum capacity of the next level cache. Verification 2 is used to determine whether the number of threads used by the complete operator after partitioning according to the partitioning method is less than the maximum number of threads supported by the hardware. If the results of Verification 1 and Verification 2 are yes, it is considered reasonable.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the final optimized schemes of all dynamic shape operators, compare whether the layer segmentation methods of each of the two final optimized schemes are completely consistent, and if so, merge the duplicate final optimized schemes.

6. The method according to any one of claims 1-4, characterized in that, The multi-classifier is trained based on the XGBoost algorithm, whose objective function is the sum or weighted sum of the training loss function used to guide the model to classify correctly and the regularization function used to control the complexity of the model.

7. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-6.

9. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-6 by executing the executable instructions.

Images