Compilation optimization method for dynamic shape operator

Abstract

The invention provides a compilation optimization method for a dynamic shape operator, which comprises the following steps: acquiring a to-be-compiled model based on a neural network, and determining an input shape of the dynamic shape operator in the model during current execution; the input shape is input into a multi-classifier, a prediction optimization scheme corresponding to the input shape is obtained, the multi-classifier is obtained through pre-training of a training set collected for the model, and a prediction optimization scheme corresponding to the input shape is obtained; the training set comprises samples collected after simulation optimization is carried out on various input forms of all dynamic shape operators of the model based on GPU hardware information; each sample comprises a determined input shape corresponding to a dynamic shape operator and a final optimization scheme selected according to the actual operation condition of the optimization scheme of the input shape on the CUDA core and the Tensor core of the GPU; and compiling a corresponding dynamic shape operator in the model according to the prediction optimization scheme.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, in particular to the field of optimization of dynamic shape operators in machine learning, and more particularly to a compilation optimization method for dynamic shape operators. Background Art

[0002] In recent years, the scale and complexity of machine learning models have continued to increase, requiring hardware with higher computing power to support their training and reasoning. Different models require different types of hardware accelerators, such as GPUs, TPUs, etc. In order to achieve the best performance on different hardware, the academic community has proposed the concept of machine learning compilers. Machine learning compilers enable models to better utilize the resources of different hardware platforms by providing automated optimization and adaptability, thereby accelerating the training and reasoning process of machine learning models.

[0003] Operator automatic tuning is the core link of the machine learning compiler workflow, and its purpose is to optimize the computational efficiency of different operators on specific hardware. Automatic optimization of operators with static shapes is relatively mature. However, in the related models of natural language processing, there are more and more operators with dynamic shapes. For example, the input sequence length of the Bert model and its batch size (the number of samples contained in a training batch) will cause changes in the input and output sizes of the query Q, key K and value V calculations in the Transformer structure, resulting in dynamic shape operators. The Transformer structure is the basis of the current mainstream models, and a large number of models contain the Transformer structure. In addition, there may be other forms of dynamic shape operators, so it is necessary to optimize the dynamic shape operators so that the machine learning compiler technology can adapt to more advanced models.

[0004] Most of the existing automatic operator tuning algorithms only support static shape operators (i.e., operators with fixed input tensor shapes), and the optimization results for a specific shape can be applied to all inputs of the operator. However, for dynamic shape operators (i.e., operators with unfixed input tensor shapes), applying such optimization schemes to other possible input shapes often cannot fully utilize the device hardware resources.

[0005] The existing automatic tuning schemes for static shape operators are mainly divided into heuristic search algorithms and template-based search algorithms. Among them, the search time of heuristic search algorithms is usually hours, and even after a long search, it is not guaranteed to get a suitable search result; the template-based search algorithm, although the search time for a good tuning result is shorter, requires experts to complete the definition and design of the template, has a long design cycle and poor versatility.

[0006] The existing schemes that can complete the optimization of dynamic shape operators still mainly refer to the static operator optimization idea, including using a template-based search algorithm to obtain an optimization scheme for the most likely shape, and applying this optimization scheme to all possible shapes. In addition to inheriting the shortcomings of the template-based search algorithm, this algorithm has the disadvantage of poor optimization performance for other specific shapes because it only selects one shape for optimization. Another method is based on the heuristic search algorithm in static operator optimization. By selecting some representative shapes among the possible shapes (covering small and large shapes as much as possible) and optimizing them using the heuristic search algorithm, a variety of optimization schemes are obtained. At runtime, the specific optimization scheme is determined by the operator input shape. This scheme cannot efficiently solve the dynamic operator optimization problem because the heuristic search takes too long and the optimization size selection is limited.

[0007] In addition, the previous operator automatic optimization technology was mainly implemented around the CUDA core (CUDACore) of NVIDIA devices. As a general computing core, the CUDA core has low computing efficiency. In order to accelerate deep learning operators, NVIDIA subsequently launched the Tensor Core. The Tensor Core uses the characteristics of parallelism and tensor operations in matrix multiplication to achieve efficient computing acceleration through customized hardware design. The subsequent Bolt technology integrates NVIDIA Cutlass into the compiler, which can use the Tensor core in NVIDIA devices. NVIDIA Cutlass is an open source project technology for implementing high-performance matrix multiplication and convolution operations inside the GPU. However, not all subgraphs that can be represented by Cutlass can achieve optimal performance through Tensor Core. Although Tensor Core is specially designed to accelerate deep learning and artificial intelligence (AI) workloads (such as matrix operations), deep learning cannot leave the CUDA core at present.

[0008] First, CUDA cores are optimized for various parallel computing tasks and are more suitable for general parallel computing tasks. Deep learning tasks are not just matrix operations, but also involve a lot of parallel computing. If the bottleneck is in parallel computing, then this type of deep learning task may be more suitable for CUDA cores. Of course, Tensor cores can also be used for general parallel computing tasks, but it may not be as efficient as CUDA cores.

[0009] Secondly, although Tensor Cores work well for General Matrix Multiplication (GEMM) calculations, their input and output data types need to be half-precision or single-precision, and the matrix dimensions should preferably be a multiple of 8, which leads to certain limitations in the use of Tensor Cores.

[0010] In summary, the prior art still has the following problems:

[0011] (1) Hardware information is not taken into account in the tuning algorithm, so hardware features cannot be used efficiently.

[0012] (2) The heuristic search time is too long, which causes huge time consumption in automatic tuning;

[0013] (3) Resource scheduling for specific input shapes is not considered.

[0014] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. The relevant information is submitted and disclosed together with the present invention solution. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0015] Therefore, the purpose of the present invention is to overcome the defects of the prior art and provide a compilation optimization method for dynamic shape operators.

[0016] The objective of the present invention is achieved through the following technical solutions:

[0017] According to a first aspect of the present invention, a compilation optimization method for dynamic shape operators is provided, comprising: obtaining a neural network-based model to be compiled, and determining an input shape of a dynamic shape operator therein during current execution; inputting the input shape into a multi-classifier to obtain a predicted optimization scheme corresponding to the input shape, wherein the multi-classifier is pre-trained using a training set collected for the model, the training set including samples collected after simulation optimization of various input forms of all dynamic shape operators of the model based on hardware information of a GPU, each sample including a determined input shape corresponding to a dynamic shape operator and a final optimization scheme selected based on actual operation of the optimization scheme for the input shape on a CUDA core and a Tensor core of a GPU; and compiling the corresponding dynamic shape operator in the model according to the predicted optimization scheme.

[0018] Optionally, the training set of each model is collected in the following manner: obtain the neural network-based model to be compiled, the hardware information of the GPU used to execute the model, and a preset tuning threshold, split the model into multiple computing subgraphs according to the model calculation graph, each computing subgraph represents an operator, wherein the hardware information includes the total cache level, the minimum memory storage unit, the maximum cache capacity of each layer, and the maximum number of threads; collect various dynamic shape operators from the multiple computing subgraphs to form a first set; for each operator in the first set, collect the possible input shapes of each operator to form a second set corresponding to the operator, wherein if the total number of possible input shapes of the operator is less than or equal to the tuning threshold, the second set collects all the input shapes, otherwise the second set collects the sampled multiple input shapes; each input shape in the second set corresponding to each operator The operators of the shape are combined with the corresponding computing logic into a complete operator; each complete operator is optimized to obtain the corresponding final optimization plan, wherein, hierarchical optimization is performed according to the cache level when optimizing the CUDA core, and each layer of optimization selects a segmentation method that can make the data reuse rate change rate the highest from a variety of segmentation methods set based on the minimum storage unit of the memory to segment the axis related to the complete operator stored in the previous layer to obtain the first optimization plan, and the BOLT technology is used to generate the second optimization plan for the Tensor core, and the plan that minimizes the running time of the complete operator is selected from the first optimization plan and the second optimization plan as the final optimization plan, wherein the first optimization plan needs to be verified to meet the requirements of the maximum cache capacity and the maximum number of threads of each layer; each input shape and the corresponding final optimization plan are combined into a sample to form a training set. This technical solution can at least achieve the following beneficial technical effects: when the number of possible shapes of the operator input is greater than the tuning threshold, sampling optimization is adopted, and the classification algorithm can be used to complete the mapping of all input shapes and optimization schemes subsequently, so as to collect samples of dynamic shape operators with more possible input shapes in a more comprehensive and balanced manner, so as to achieve more accurate prediction in the future; when the possible shape of the operator input is less than the tuning threshold, full optimization is adopted, so that the optimization scheme for small samples can cover all possible workloads and the prediction effect is better; the running performance of the two optimization schemes on different CUDA cores and Tensor cores is actually simulated to determine the core type and optimization scheme for more efficient processing of each input shape operator.

[0019] Optionally, the sampled multiple input shapes are obtained by sampling from the dynamic range corresponding to the dynamic shape operator at a preset sampling interval, wherein the sampling interval is determined according to the upper limit value of the dynamic range and the tuning threshold. This technical solution can at least achieve the following beneficial technical effects: the sampling interval is determined by tuning the threshold, and multiple input shapes are obtained by sampling from the dynamic range at the sampling interval, which can avoid selecting only one shape for optimization as in the prior art, and more comprehensive and balanced collection of samples of dynamic shape operators with more possible input shapes, so as to achieve more accurate prediction later, and solve the disadvantage of poor optimization performance for specific input shapes.

[0020] Optionally, when optimizing the CUDA core, the first optimization scheme includes an initial segmentation form corresponding to the memory and a segmentation method corresponding to each layer of cache, wherein: all segmentation factors in the initial segmentation form corresponding to the memory are set to 1 to segment each axis, and the initial segmentation form of the memory is used to segment the operator-related data stored in the memory and load it into the previous layer of cache; the segmentation method of each layer of cache is used to segment the operator-related data stored in the current layer of cache and load it into the previous layer of cache, and the segmentation method of each layer of cache is selected from a plurality of different segmentation methods, each segmentation method includes segmentation factors corresponding to all axes, the i-th axis in the i-th segmentation method is an associated axis, the segmentation factor of the associated axis is set to an integer multiple of the minimum storage unit of the memory, and the segmentation factors of other axes other than the associated axis are set to 1, and after each layer of cache is optimized, the value of the segmentation factor of the associated axis in each segmentation method is increased. This technical solution can at least achieve the following beneficial technical effects: each segmentation method only adjusts the segmentation factor of one associated axis, which can avoid the segmentation scheme being too complicated and difficult to implement, thereby efficiently comparing multiple segmentation methods.

[0021] Optionally, the splitting method of each layer of cache is obtained in the following manner: the axis of the complete operator is processed according to each splitting method, and the change rate of the data reuse rate corresponding to the cache after processing according to each splitting method is determined; the splitting methods are selected in order of the change rate of the data reuse rate from large to small, and according to a preset verification mechanism, it is determined whether the selected splitting method is reasonable at the current cache level. If it is reasonable, the splitting method is used as the splitting method of the cache at this layer, wherein the verification mechanism includes verification 1 and verification 2, verification 1 is used to determine whether the size of the data block loaded into the next level cache according to the splitting method is less than the maximum capacity of the next level cache, and verification 2 is used to determine whether the number of threads used corresponding to the complete operator after splitting according to the splitting method is less than the maximum number of threads supported by the hardware. If the results of verification 1 and verification 2 are yes, it means it is reasonable.

[0022] Optionally, the method further includes: obtaining final optimization schemes of all dynamic shape operators, comparing whether the segmentation methods of each layer in every two final optimization schemes are completely consistent, and if so, merging repeated final optimization schemes.

[0023] Optionally, the multi-classifier is trained based on the XGBoost algorithm, and the objective function of the XGBoost algorithm includes a training loss for guiding the correct classification of the model and a regularization term for controlling the complexity of the model.

[0024] According to a second aspect of the present invention, there is provided an electronic device comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0026] Figure 1 The figure is a flowchart of a compilation optimization method for a dynamic shape operator according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] As mentioned in the background technology section, the prior art does not incorporate hardware information into the tuning algorithm, so it is impossible to efficiently utilize hardware characteristics, automatic tuning consumes a huge amount of time, and there is no resource scheduling for specific input shapes. In order to efficiently and automatically tune the dynamic shape operator, the inventor proposes a fast compilation optimization method for the dynamic shape operator. This method collects samples collected after simulation optimization considering the hardware information of the GPU for various input forms of the dynamic shape operator to form a training set. Each sample includes a certain input shape corresponding to a dynamic shape operator and a final optimization solution selected based on the actual operation of the optimization solution for the input shape on the CUDA core and Tensor core of the GPU. The hardware information can be efficiently used to determine the final optimization solution for a specific input shape, thereby constructing a training set and training a multi-classifier to predict the optimization solution for a specific input shape, which can accelerate the training or reasoning process of the machine learning model.

[0029] According to one embodiment of the present invention, see Figure 1 , a compilation optimization method for dynamic shape operators, including: S100, S200, S300, S400, S500, S600, S700 and S800. In order to better understand the present invention, each step is described in detail below in conjunction with specific embodiments.

[0030] S100: Obtain a neural network-based model to be compiled, hardware information of a GPU used to execute the model, and a preset tuning threshold, split the model into multiple computational subgraphs according to the model computational graph, each computational subgraph represents an operator, wherein the GPU includes a CUDA core and a Tensor core, and the hardware information includes a total cache level.

[0031] According to one embodiment of the present invention, the entire model is first input, and the model is split into multiple computational subgraphs according to the model computational graph, where a computational subgraph represents an operator; a model compilation platform is obtained (the TVM platform is selected in this embodiment), and hardware information and tuning thresholds (used to limit the number of shapes to be tuned) are configured in the platform. Preferably, the hardware information includes the total cache level H, the maximum cache capacity of each layer, the minimum memory storage unit (the amount of data accessed in one memory transaction), and the maximum number of threads. Alternatively, the hardware information includes one or a combination of the aforementioned elements.

[0032] According to an example of the present invention, the model Bert_base is used as the tuning object (i.e., the neural network-based model to be compiled), and NVIDIA's RTX4090 is used as the deployment device; the tuning threshold n=64 and the hardware information of RTX4090 are configured, and the total cache level (H=2, the second layer cache refers to the second layer cache, and the first layer cache refers to the first layer cache), the maximum cache capacity of each layer, the minimum memory storage unit and the maximum number of threads are configured according to the actual situation of RTX4090.

[0033] S200: Collect various dynamic shape operators from the multiple computation subgraphs to form a first set.

[0034] According to one embodiment of the present invention, a first set which is initially an empty set is first established, and dynamic shape operators are collected by traversing multiple computational subgraphs. Whether the operator is a dynamic shape operator is determined based on whether the shape of the operator input represented by the computational subgraph is a fixed value. If the operator is a dynamic shape operator, the operator is stored in the first set for subsequent tuning.

[0035] S300: For each operator in the first set, collect possible input shapes of each operator to form a second set corresponding to the operator, wherein if the total number of input shapes of the operator is less than or equal to the tuning threshold, the second set collects all input shapes, otherwise the second set collects multiple sampled input shapes.

[0036] According to an embodiment of the present invention, S300 is used to determine the workload that needs to be tuned, wherein each dynamic shape operator obtained in the first set is taken out respectively, and the input shape range of each dynamic shape operator is given. If the number of possible input shapes of the input shape (that is, the total number of input shapes of the operator) is less than a given tuning threshold, all shapes are optimized to simulate all possible input shapes; if the input shape is greater than the given threshold, all possible workloads are sampled at a certain interval, and at most the shapes of the threshold number are selected to obtain a shape set: the second set [α1, α2, α3, α4, …, α n ]. Preferably, the sampled multiple input shapes are sampled from the dynamic range corresponding to the dynamic shape operator at a preset sampling interval, wherein the sampling interval is determined according to the upper limit value of the dynamic range and the tuning threshold. For example, the sampling interval is equal to the upper limit value of the dynamic range divided by the tuning threshold, and the quotient is rounded down, and the sampling interval is at least 2. In this embodiment, by setting a tuning threshold, when the number of possible shapes of the operator input is greater than the tuning threshold, sampling optimization is adopted, and multiple classifiers can still be used to complete the mapping of all input shapes to the optimization scheme. When the possible shape of the operator input is less than the tuning threshold, full optimization is adopted, so as to optimize as many different input forms of the same dynamic shape operator as possible.

[0037] S400: Combining operators of each input shape in the second set of each operator with calculation logic into a complete operator.

[0038] According to one embodiment of the present invention, step S400 is a process of operator combination, including taking out each input shape α obtained in step S300 n , combining the input shape with the operator's calculation logic to obtain a complete operator. Schematically, taking the Matmul operator as an example, the operator calculation expression is:

[0039] Input*weight=output

[0040] Among them, input is the input tensor, the corresponding input shape is [S, 768], S is the dynamic input size, weight is the weight tensor, the shape is [768, 768], output is the output tensor, the shape is [768, S]. The input shape range (dynamic change range) of S is 1-128.

[0041] S500: Optimize each complete operator to obtain the corresponding final optimization solution, wherein, for the CUDA core, select a segmentation method that can make the data reuse rate change rate the highest from the preset multiple segmentation methods for each cache level to segment the data related to the complete operator stored in the front layer to obtain the first optimization solution, and call the BOLT optimization technology for the Tensor core to generate a second optimization solution, and select the solution that minimizes the running delay of the complete operator from the first optimization solution and the second optimization solution as the final optimization solution. Among them, the first optimization solution needs to be verified to meet the requirements of the maximum cache capacity and the maximum number of threads per layer.

[0042] According to one embodiment of the present invention, when optimizing the CUDA core, the first optimization scheme includes an initial segmentation form corresponding to the memory and a segmentation method corresponding to each layer of cache, wherein: all segmentation factors in the initial segmentation form corresponding to the memory are set to 1 to segment each axis, and the initial segmentation form of the memory is used to segment the operator-related data stored in the memory and load it into the previous layer of cache; the segmentation method of each layer of cache is used to segment the operator-related data stored in the current layer of cache and load it into the previous layer of cache, and the segmentation method of each layer of cache is selected from a plurality of different segmentation methods, each segmentation method includes segmentation factors corresponding to all axes, the i-th axis in the i-th segmentation method is an associated axis, the segmentation factor of the associated axis is set to an integer multiple of the minimum storage unit of the memory, and the segmentation factors of other axes other than the associated axis are set to 1, and after each layer of cache is optimized, the value of the segmentation factor of the associated axis in each segmentation method is increased.

[0043] According to one embodiment of the present invention, for the complete operator of step S400, step S500 is optimized according to the following steps:

[0044] Step S501: extract the spatial axis (the axis that appears in the calculation result tensor) and the reduced axis (the axis that appears in the calculation process but does not appear in the calculation result) in the operator. For the global memory to the H-layer cache, the initial split factor of 1 is used to indicate the splitting of each axis (that is, each dimension of the original tensor is split according to the size of 1, and the split data is sent to the H-layer cache); for example: extract the spatial axis (the axis that appears in the calculation result tensor) M, N and the reduced axis (the axis that appears in the calculation process but does not appear in the calculation result) K in the operator. For the memory to the second-layer cache, the initial split factor of 1 is used to indicate the splitting of each axis m=1, n=1, k=1 (that is, the input is split according to the shape of [m, k], and the data is sent to the second-layer cache, the weight is split according to the shape of [n, k], and the data is sent to the second-layer cache, and the output is split according to the shape of [m, n], and the data is sent to the second-layer cache.

[0045] Step S502: Increase the split factor, and for each axis in the i-th split mode, ensure that except for the associated axis A i Other axes A0A1…A i-1 A i+1 …A n-1 A n Remain unchanged and increase the axis A by an integer multiple of the memory transaction i The segmentation factor Z i , and obtain a segmentation method represented by a set of segmentation factors (f0,f1,…f i-1 ,Z i ,f i+1 ,…,f n-1 ,f n ), where Z i Indicates the segmentation factor corresponding to the associated axis, f0,f1,…f i-1 ,f i+1 ,…,f n-1 ,f4 represents the slicing factors corresponding to other axes; taking NVIDIA RTX4090 as an example, the minimum number of data read and written to the memory is 4 bytes, then m=4, and the other slicing factors remain unchanged, and the slicing combination sf1: m=4, n=1, k=1 is obtained; increase the slicing factor n, increase the slicing factor of the axis according to the integer multiple of the memory transaction, and obtain n=4, and the other slicing factors remain unchanged, and obtain the slicing combination sf2: m=1, n=4, k=1; increase the slicing factor k, increase the slicing factor of the axis according to the integer multiple of the memory transaction, and obtain k=4, and the other slicing factors remain unchanged, and obtain the slicing combination sf3: m=1, n=1, k=4;

[0046] Step S503: Calculate the change of the data reuse rate after the segmentation factor of each associated axis is changed, and calculate the change rate R of the data reuse rate after the segmentation factor of any axis is changed. i , R i The calculation method is as follows:

[0047]

[0048] Among them, Q represents the data flow of the segmentation method before changing the segmentation factor of a certain axis, Q new Indicates the data flow of the segmentation method after the segmentation factor of a certain axis is changed. F indicates the cache usage of the segmentation method before the segmentation factor of a certain axis is changed. F new Indicates the cache usage of the split method after the split factor of an axis is changed.

[0049] Step S504: sorting, sorting the segmentation methods from large to small according to the change rate of the data reuse rate, and the subsequent algorithm will give priority to the segmentation method based on the highest change rate of the data reuse rate for calculation;

[0050] Step S505: Verify whether the current segmentation method is reasonable at the current cache level, including verifying whether the size of the data block loaded into the H-layer cache after segmentation according to the segmentation method is less than the maximum capacity of the H-layer cache, and verifying whether the number of threads used is less than the maximum number of threads supported by the hardware; if it is not reasonable, discard the segmentation method;

[0051] Step S506: Verify whether the calculation delay of the current segmentation method at the current cache level is greater than the memory delay. If the calculation delay is greater than the memory delay, it means that the layer cache optimization is completed; this step is used to verify whether the memory delay is hidden;

[0052] Step S507: Enter step S503 for optimization for the previous layer of cache until the optimization of caches at all levels is completed (from layer H-1 to layer 1 in sequence), that is, the optimization solution γ1 of the CUDA core of the input shape α1 is obtained;

[0053] Step S508: For the optimization solution γ1, use the build interface (Build interface) in the TVM platform to obtain the CUDA source code of the optimization solution, and then use the run interface (Run interface) to obtain the running time T1 of the optimization solution γ1 of the shape α1;

[0054] Step S509: Tensor core performance data collection, for the input shape α1, use Bolt technology to generate the Cutlass operator, and use the TVM platform's run interface (Run interface) to obtain the running time T2 of the Cutlass operator;

[0055] Step S510: Compare the running time T1 of the operator of the input shape α1 in the CUDA core and the running time T2 in the Tensor core, and select the solution with the smaller running time as the final optimization solution β1 of the input shape;

[0056] Step S511: For any input shape α obtained in step S300 i , repeat steps S501-S507 to obtain the optimized β i , so far the input shape set of step S300 [α1, α2, α3, α4, …, α n ], there is a corresponding final optimization solution [β1,β2,β3,β4,…,β n ], since there are several inputs with the same optimization scheme, if the split factors of each layer are the same, the two optimization schemes are considered to be the same. The optimization schemes are deduplicated to obtain k different final optimization schemes [β1,β2,β3,β4,…,β k ], where k≤n.

[0057] S600: Taking the input shape as an independent variable and the corresponding final optimization solution as a dependent variable to form a training set, and using the training set to train a multi-classifier to obtain a trained multi-classifier.

[0058] According to one embodiment of the present invention, the training set of each model is collected in the following manner: a neural network-based model to be compiled, hardware information of a GPU used to execute the model, and a preset tuning threshold are obtained, and the model is split into multiple computing subgraphs according to the model calculation graph, each computing subgraph represents an operator, wherein the hardware information includes the total cache level, the minimum memory storage unit, the maximum cache capacity of each layer, and the maximum number of threads; various dynamic shape operators are collected from the multiple computing subgraphs to form a first set; for each operator in the first set, possible input shapes of each operator are collected to form a second set corresponding to the operator, wherein if the total number of possible input shapes of the operator is less than or equal to the tuning threshold, the second set collects all input shapes, otherwise the second set collects multiple sampled input shapes; each operator in the second set corresponding to each operator is collected. An operator with an input shape is combined with the corresponding computing logic into a complete operator; each complete operator is optimized to obtain the corresponding final optimization solution, wherein hierarchical optimization is performed according to the cache level when optimizing the CUDA core, and each layer of optimization selects a segmentation method that can make the data reuse rate change the highest from a variety of segmentation methods set based on the minimum storage unit of the memory to segment the axis related to the complete operator stored in the previous layer to obtain the first optimization solution, and the BOLT technology is used to generate the second optimization solution for the Tensor core, and the solution that minimizes the running time of the complete operator is selected from the first optimization solution and the second optimization solution as the final optimization solution, wherein the first optimization solution needs to be verified to meet the requirements of the maximum cache capacity and the maximum number of threads of each layer; each input shape and the corresponding final optimization solution are combined into a sample to form a training set. The technical solution of this embodiment can at least achieve the following beneficial technical effects: when the number of possible shapes of the operator input is greater than the tuning threshold, sampling optimization is adopted, and the classification algorithm can be used to complete the mapping of all input shapes and optimization schemes subsequently, so as to collect samples of dynamic shape operators with more possible input shapes in a more comprehensive and balanced manner, so as to achieve more accurate prediction subsequently; when the possible shape of the operator input is less than the tuning threshold, full optimization is adopted, so that the optimization scheme for small samples can cover all possible workloads and the prediction effect is better; in addition, the running performance of the two optimization schemes on different CUDA cores and Tensor cores is actually simulated to determine the core type and optimization scheme for more efficient processing of each input shape operator.

[0059] According to an embodiment of the present invention, the input shape is used as an independent variable, the corresponding optimization combination / scheme is used as a dependent variable, and a multi-classification classifier is trained using XGBoost. Optionally, the objective function of XGBoost consists of two parts: a training loss function L and a regularization function Ω.

[0060] Obj(Θ)=L(Θ)+Ω(Θ)

[0061] It should be understood that different weights may be assigned to L(Θ) and Ω(Θ), and the objective function is the weighted sum of the two.

[0062] Step S601: Calculate the training loss. For a multi-classification problem with k categories, for each instance i, the training loss function is as follows:

[0063]

[0064] Among them, y ik is an indicator function that indicates when instance i (i.e., the input shape in sample i) belongs to category k ik is 1, otherwise it is 0; p ik is the probability that the model predicts that instance i belongs to category k.

[0065] Step S602: predict probability p ik Calculated by the softmax function:

[0066]

[0067] Among them, φ ik Represents the logical value (Logit value, which is the original output score) of the model for the probability that instance i belongs to category k, φ ij A logical value representing the model's probability that instance i belongs to category j, where K represents the total number of categories.

[0068] Step S603: Calculate the gradient matrix and the Hessian matrix. In order to optimize the objective function, it is necessary to calculate the gradient matrix and the Hessian matrix of each instance i and category k.

[0069] Gradient matrix:

[0070]

[0071] Hessian matrix:

[0072]

[0073] The derivatives in these matrices are used to build new decision trees, optimizing the objective function at each iteration;

[0074] S604: Build and update the decision tree, use the calculated gradient matrix and Hessian matrix to build a new decision tree, and update the model parameters Θ and the leaf node scores:

[0075]

[0076] Among them, R j represents the set of instances contained in leaf node j, and λ is the regularization parameter;

[0077] S605: Calculate the regularization term, which is used to control the complexity of the model:

[0078]

[0079] Among them, T represents the number of leaf nodes in the tree, w j represents the weight of leaf node j, γ and λ represent regularization parameters;

[0080] S606: Update the objective function, which is the sum of the training loss and the regularization term:

[0081] Obj(Θ)=L(Θ)+Ω(Θ)

[0082] S607: Iterative training, repeating steps S601 to S606 until the objective function converges or the training reaches a predetermined number of iterations.

[0083] It should be understood that the multi-classifier may also adopt other forms such as random forest.

[0084] S700: Using the trained multi-classifier to make predictions based on the actual input shape of the dynamic shape operator to obtain a corresponding prediction optimization solution.

[0085] At this point, a multi-classifier is obtained that selects the appropriate optimization solution according to the input shape of the operator. In the future, the predicted optimization solution corresponding to the operator of each input shape can be predicted based on the multi-classifier, and the mapping of operator shape to optimization solution can be automatically completed, thereby completing the operator optimization task.

[0086] S800: Compile the corresponding dynamic shape operator in the model according to the prediction optimization solution.

[0087] After obtaining the prediction optimization solution corresponding to a certain dynamic shape operator under a certain input shape, the dynamic shape operator can be compiled according to the prediction optimization solution with the help of the TVM platform.

[0088] In general, the compilation optimization method for dynamic shape input operators provided by the present invention is highly efficient: combined with hardware characteristics, it can make full use of hardware performance, can automatically select operators with dynamic shape input when a model with a dynamic shape input operator is running, can automatically sample workloads covering a variety of dynamic shapes, and efficiently complete the tuning of operators with dynamic shape input; the method of the present invention is highly scalable: hardware adaptation only requires configuration of hardware information, and can quickly complete the adaptation of a variety of hardware; the method of the present invention is also universal: efficient optimization solutions can be generated for mainstream computing-intensive operators such as Dense and Convolution.

[0089] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0090] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0091] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0092] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A compilation optimization method for dynamic shape operators, characterized in that: include: Obtaining a neural network-based model to be compiled, and determining an input shape of a dynamic shape operator therein during current execution; Inputting the input shape into a multi-classifier to obtain a prediction optimization solution corresponding to the input shape, wherein the multi-classifier is pre-trained using a training set collected for the model, and the training set includes samples collected after simulation optimization of various input forms of all dynamic shape operators of the model based on GPU hardware information, and each sample includes a determined input shape corresponding to a dynamic shape operator and a final optimization solution selected based on actual running conditions of the optimization solution for the input shape on the CUDA core and the Tensor core of the GPU; Compile the corresponding dynamic shape operators in the model according to the predicted optimization scheme.

2. The method according to claim 1, characterized in that The training set for each model is collected as follows: Obtain the neural network-based model to be compiled, the hardware information of the GPU used to execute the model, and the preset tuning threshold, split the model into multiple computational subgraphs according to the model computation graph, each computational subgraph represents an operator, where the hardware information includes the total cache level, the minimum memory storage unit, the maximum cache capacity of each layer, and the maximum number of threads; Collect various dynamic shape operators from the plurality of computational subgraphs to form a first set; For each operator in the first set, possible input shapes of each operator are collected to form a second set corresponding to the operator, wherein if the total number of possible input shapes of the operator is less than or equal to the tuning threshold, the second set collects all input shapes, otherwise the second set collects multiple sampled input shapes; Combine the operator of each input shape in the second set corresponding to each operator and the corresponding calculation logic into a complete operator; Optimize each complete operator to obtain the corresponding final optimization plan, wherein, for CUDA core optimization, hierarchical optimization is performed according to the cache level, and each layer of optimization selects a segmentation method that can make the data reuse rate change rate the highest from a variety of segmentation methods set based on the minimum storage unit of memory to segment the axis related to the complete operator stored in the previous layer to obtain the first optimization plan, and use BOLT technology to generate a second optimization plan for the Tensor core, and select the plan that minimizes the running time of the complete operator from the first optimization plan and the second optimization plan as the final optimization plan, wherein the first optimization plan needs to be verified to meet the requirements of the maximum cache capacity and the maximum number of threads of each layer; Each input shape and the corresponding final optimization solution are combined into a sample to form a training set.

3. The method according to claim 2, characterized in that The sampled multiple input shapes are sampled from the dynamic range corresponding to the dynamic shape operator at a preset sampling interval, wherein the sampling interval is determined according to an upper limit value of the dynamic range and a tuning threshold.

4. The method according to claim 2, characterized in that: The first optimization scheme for CUDA core optimization includes the initial segmentation form corresponding to the memory and the segmentation method corresponding to each layer of cache, where: All the split factors in the initial split form corresponding to the memory are set to 1 to split each axis. The initial split form of the memory is used to split the operator-related data stored in the memory and load it into the upper-level cache; The splitting method of each layer of cache is used to split the operator-related data stored in the current layer of cache and load it into the previous layer of cache. The splitting method of each layer of cache is selected from a plurality of different splitting methods. Each splitting method includes splitting factors corresponding to all axes. The i-th axis in the i-th splitting method is the associated axis. The splitting factor of the associated axis is set to an integer multiple of the minimum storage unit of the memory. The splitting factors of other axes except the associated axis are set to 1. After each layer of cache is optimized, the value of the splitting factor of the associated axis in each splitting method is increased.

5. The method according to claim 4, characterized in that The partitioning method of each layer of cache is obtained as follows: Process the axis of the complete operator according to each segmentation method, and determine the change rate of the data reuse rate corresponding to the cache after processing according to each segmentation method; The splitting methods are selected in order of the rate of change of the data reuse rate from large to small. The preset verification mechanism is used to determine whether the selected splitting method is reasonable at the current cache level. If it is reasonable, the splitting method is used as the splitting method for the cache at this level. The verification mechanism includes verification 1 and verification 2. Verification 1 is used to determine whether the size of the data block loaded into the next-level cache according to the splitting method is less than the maximum capacity of the next-level cache. Verification 2 is used to determine whether the number of threads used corresponding to the complete operator after splitting according to the splitting method is less than the maximum number of threads supported by the hardware. If the results of verification 1 and verification 2 are yes, it means it is reasonable.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: Get the final optimization solutions of all dynamic shape operators, and compare whether the segmentation methods of each layer in each two final optimization solutions are completely consistent. If so, merge the repeated final optimization solutions.

7. The method according to any one of claims 1 to 5, characterized in that: The multi-classifier is trained based on the XGBoost algorithm, and the objective function of the XGBoost algorithm is the sum or weighted sum of a training loss function for guiding the correct classification of the model and a regularization function for controlling the complexity of the model.

8. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-7 by executing the executable instructions.

Images