Automatic tuning method and device of code generator, storage medium and program product
By designing the automatic tuning method of the code generator, the tensor computing core is used to optimize the performance of large-size matrix multiplication, the problem of FLOPS determining training or inference time in large language models is solved, and an efficient training and inference process is realized.
Patent Information
- Application Number
- CN202411818719.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-04-08
AI Technical Summary
When the scale of large language models increases, the existing technology cannot effectively optimize the performance of large-size matrix multiplication, resulting in the training or inference time being limited by FLOPS, and FlashAttention is no longer applicable.
Design an automatic tuning method for code generators, using the tensor computing core, use the tensor computing core to define the search space, integrate the search algorithm, generate parameter list populations, configure template parameters, compile and measure the performance of the kernel function, use the cost model to filter and prune, and dynamically search for the optimal parameters.
The optimization of the performance of large-size matrix multiplication is achieved, which reduces the latency of accessing shared memory, increases the hit rate of the cache, and improves the training and inference efficiency of large language models.
Smart Images
Figure CN119783775A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of parallel computing technology, and in particular to a method, device, storage medium and program product for automatically tuning a code generator. Background Art
[0002] In the process of training and inference of large language models, there are two main overheads: memory access cost (MAC) and floating-point operations per second (FLOPS). FlashAttention is an optimization method currently used in large language model neural networks. It mainly optimizes MAC to accelerate and save memory. It can reduce the memory access time of operators other than matrix multiplication when the model parameters of the large language model are not very large. When operations such as masking, softmax, and dropout in a neural network take up a lot of time, FlashAttention can significantly reduce the number of memory accesses by decomposing the matrix into blocks, thereby saving the time of neural network training and inference. However, when the scale of large language models continues to increase, MAC will no longer be the dominant factor, and FLOPS will determine the overall training or inference time. FLOPS is mainly occupied by matrix multiplication operations. Therefore, Flash Attention is no longer applicable to large language models with larger scale.
[0003] Since the Volta architecture, NVIDIA has introduced a dedicated matrix multiplication module called Tensor Core, which not only enables mixed-precision calculations, but also significantly improves the calculation speed compared to the original CUDA Core. CUDA Core can only perform one operation per clock cycle, while Tensor Core can perform multiple operations per clock cycle. Therefore, it is necessary to consider designing a code generator that uses tensor cores for automatic tuning to achieve overhead optimization for large-scale large language models. Summary of the invention
[0004] In view of the deficiencies in the prior art, the present invention provides an automatic tuning method, device, storage medium, and program product for a code generator.
[0005] In one aspect of the present invention, there is provided a method for automatically tuning a code generator, comprising:
[0006] A search space is defined according to the characteristics of a given large model inference chip, a given search algorithm is integrated into a software development environment of the large model inference chip, and a parameter list population is generated through multiple cycles using the given search algorithm, wherein the parameter list population includes a plurality of parameter lists;
[0007] According to the parameter list population, the specific template parameters are configured to generate the definition part of the kernel function, the pipeline buffer level, and the process of transferring the result register to the global memory;
[0008] The generated kernel function is compiled by a compiler and executed on the large model inference chip, the performance of the generated kernel function is measured, the kernel function performance index is obtained, and the parameter list population and the corresponding kernel function performance index are added to the performance dictionary as a key-value pair.
[0009] In one embodiment of the present invention, the method further comprises:
[0010] Sorting the parameter list population according to the kernel function performance index to obtain a sorted parameter list population;
[0011] The sorted parameter list population feedback is used to adjust the parameters of the search algorithm.
[0012] In one embodiment of the present invention, the search algorithm adopts a genetic algorithm, and in the first cycle, the parameter list population is randomly generated within a given parameter list range;
[0013] A crossover genetic probability and a mutation genetic probability are set according to the sorted parameter list population, a new round of iteration is performed, and the parameter list population is updated.
[0014] In one embodiment of the present invention, the time spent on the current tuning and the highest performance index are counted;
[0015] If the time spent is less than the tuning threshold, or the highest performance index is less than the allowed performance threshold, then increasing the probability of mutation inheritance to increase the search range;
[0016] If the time spent is greater than or equal to the tuning threshold, or the highest performance index is greater than or equal to the allowable performance threshold, then the mutation inheritance probability is increased, and the mutation inheritance probability is decreased to reduce the search range.
[0017] In one embodiment of the present invention, if the parameter list population currently input does not have a corresponding key-value pair in the performance dictionary, the parameter list population is returned and the kernel function is re-acquired;
[0018] Setting a test environment for the kernel function and using a compiler to statically compile a test file;
[0019] Directly execute the compiled test file to generate the kernel function performance index and write it into an external record document;
[0020] The record document is read to obtain the parameter list population and the corresponding kernel function performance index.
[0021] In one embodiment of the present invention, if the parameter list population currently input has a corresponding key-value pair in the performance dictionary, the performance dictionary is directly read to obtain the kernel function performance index corresponding to the parameter list population.
[0022] In one embodiment of the present invention, after generating a parameter list population by using a given search algorithm through multiple cycles, the method further comprises:
[0023] Pre-screening the parameter list population for multiple rounds to update the parameter list population includes:
[0024] Preliminarily screening the parameter list population to filter out parameter lists that do not meet the large-size matrix multiplication kernel function standard;
[0025] The parameter list population after preliminary screening is passed to the constructed cost model for further screening.
[0026] In one embodiment of the present invention, the parameter list population after preliminary screening is passed to the constructed cost model for further screening, including:
[0027] Constructing a data set from the data output by the performance dictionary, including a parameter list population and corresponding kernel function performance indicators;
[0028] The data set is preprocessed, including:
[0029] Convert each parameter list of the parameter list population into a pure digital list, and discretely correspond it to an integer list increasing from 0;
[0030] Compare the kernel function performance index corresponding to each parameter list with a given performance threshold, assign a value of 0 to a value less than the performance threshold and a value of 1 to a value greater than the performance threshold; and identify kernel functions that cannot run normally and assign a value of 0;
[0031] The cost model is trained with the preprocessed data set, wherein a support vector machine is constructed in the training set, and a recall rate is evaluated in the test set, where the recall rate represents the proportion of the predicted high-performance kernel function to the total high-performance kernel functions;
[0032] The trained cost model is used to perform performance prediction on each parameter list in the parameter list population after preliminary screening, and parameter lists with predicted values of 0 are screened out.
[0033] In another aspect of the present invention, there is provided an automatic tuning device for a code generator, the device comprising:
[0034] An operator search module is used to define a search space according to the characteristics of a given large model inference chip, integrate a given search algorithm into the software development environment of the large model inference chip, and use the given search algorithm to generate a parameter list population after multiple cycles, wherein the parameter list population includes a plurality of parameter lists;
[0035] A code generation module, configured to configure specific template parameters according to the parameter list population, generate a definition part of the kernel function, a pipeline buffer level, and a process of transferring the result register to the global memory;
[0036] An operator measurement module is connected to the code generation module to receive the kernel function, compile the generated kernel function through a compiler, and execute it on the large model inference chip, perform performance measurement on the generated kernel function, obtain the kernel function performance index, and add the parameter list population and the corresponding kernel function performance index as a key-value pair to the performance dictionary.
[0037] In one embodiment of the present invention, the device further comprises: a control core module, comprising:
[0038] A sorter, together with the operator measurement module, is used to sort the parameter list population in descending order according to the kernel function performance index output by the operator measurement module, so as to obtain a sorted parameter list population;
[0039] A control core connected to the sorter and the operator search module, for adjusting the parameters of the search algorithm according to the sorted parameter list population, and feeding back to the operator search module;
[0040] A filter is connected to the operator search module and is used to preliminarily screen the parameter list population and filter out parameter lists that do not meet the large-size matrix multiplication kernel function standard.
[0041] In one embodiment of the present invention, the device further comprises:
[0042] The cost model module is connected to the operator measurement module and the control core module, and is used to receive the data output by the performance dictionary of the operator measurement module to construct a data set and train the cost model, and receive the parameter list population after preliminary screening output by the filter, and use the trained cost model to perform performance prediction on each parameter list in the parameter list population after preliminary screening, filter out the parameter list with a prediction value of 0, and feed back the parameter list population after further screening to the operator measurement module.
[0043] In one embodiment of the present invention, the operator measurement module includes:
[0044] A GPU processor, for combining a compiler and a performance dictionary, performing performance measurement on the kernel function, obtaining a kernel function performance index, adding the parameter list population and the corresponding kernel function performance index as a key-value pair to the performance dictionary, and storing the key-value pair;
[0045] A selector is connected to the code generation module to determine whether the parameter list population currently input has a corresponding key-value pair in the performance dictionary. If not, it returns to the code generation module to re-acquire the kernel function.
[0046] In yet another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method for automatically tuning a code generator are implemented.
[0047] In yet another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the automatic tuning method for a code generator when the computer program is executed by a processor.
[0048] It can be seen from the above scheme that the advantages of the present invention are:
[0049] The present invention discloses an automatic tuning method and device for a code generator suitable for a large model inference chip, which uses a tensor computing core for automatic tuning to optimize the performance of large-size matrix multiplication in the training and inference of a large language model. By determining the appropriate operator tuning method based on the specific size of the input, and dynamically searching for the optimal parameters based on the cost model for fast pruning, the latency of accessing shared memory can be reduced and the cache hit rate can be increased. And by using a performance dictionary, the parameter list that has been implemented can be quickly recorded and searched to avoid redundant compilation and wasting time. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A schematic diagram showing the overall flow of an automatic tuning method for a code generator provided by an embodiment of the present invention is shown;
[0051] Figure 2 Shows Figure 1 Detailed flow diagram of step S3;
[0052] Figure 3 A schematic structural diagram of an automatic tuning device for a code generator provided in an embodiment of the present invention is shown.
[0053] in:
[0054] 300: automatic tuning device of code generator;
[0055] 310: operator search module;
[0056] 3101: search algorithm;
[0057] 3102: parameter list population;
[0058] 320: code generation module;
[0059] 330: operator measurement module;
[0060] 3301: GPU processor;
[0061] 3302: selector;
[0062] 3303: performance dictionary;
[0063] 340: control core module;
[0064] 3401: sorter;
[0065] 3402: control core;
[0066] 3403: filter;
[0067] 350: cost model module;
[0068] 3501: Cost model. DETAILED DESCRIPTION
[0069] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.
[0070] See also Figure 1 , Figure 1 The overall flow chart of the automatic tuning method of the code generator applicable to the large model inference chip provided by one embodiment of the present invention is shown. The automatic tuning method of the code generator comprises the following steps:
[0071] Step S1, define the search space according to the characteristics of a given large model inference chip, integrate a given search algorithm into the software development environment of the large model inference chip, and use the given search algorithm to generate a parameter list population after multiple rounds of cycles, wherein the parameter list population includes several parameter lists.
[0072] Step S2: configuring specific template parameters for the large model inference chip according to the parameter list population, generating the definition part of the kernel function, the pipeline buffer level, and the process of transferring the result register to the global memory.
[0073] Step S3, compile the generated kernel function through the compiler and execute it on the large model inference chip, measure the performance of the generated kernel function, obtain the kernel function performance index, and add the parameter list population and the corresponding kernel function performance index as a key-value pair to the performance dictionary.
[0074] Step S4: sorting the parameter list population in descending order according to the kernel function performance index to obtain a sorted parameter list population; and using the sorted parameter list population as feedback to adjust the parameters of the search algorithm.
[0075] In a specific implementation, the search space is defined according to the characteristics of the large model reasoning chip (such as computing power, memory bandwidth, power consumption limit, etc.), and the search algorithm can be based on the actual configuration. For the search algorithm of traditional machine learning, different algorithms have their own advantages and disadvantages when processing different tasks. In one embodiment, the search algorithm adopts a genetic algorithm. In the first round of cycles, several parameter lists are randomly generated within the range of a given parameter list to obtain a parameter list population. A crossover genetic probability and a mutation genetic probability are set according to the sorted parameter list population, and a new round of iterations is performed to update the parameter list population. However, it should be noted that the search algorithm of the present invention is not limited to genetic algorithms. In addition, for example, the hill climbing method and the simulated annealing algorithm can be set as a given search algorithm according to actual needs, which has a high search flexibility.
[0076] In addition, the default parameters of the search algorithm are automatically adjustable. There are crossover genetic probability and mutation genetic probability in the genetic algorithm, and annealing temperature in the simulated annealing algorithm. These parameters are not static in the search process. They can be fed back to these controllable parameters based on the time spent on the current tuning and the highest performance index, so that the subsequent search is more divergent or convergent, so that the search process is carried out quickly in the direction of higher performance. For example, in one embodiment of the present invention, the time spent on the current tuning and the highest performance index are counted; if the time spent is less than the tuning threshold, or the highest performance index is less than the allowable performance threshold, the mutation genetic probability is increased to increase the search range; if the time spent is greater than or equal to the tuning threshold, or the highest performance index is greater than or equal to the allowable performance threshold, the mutation genetic probability is increased and the mutation genetic probability is reduced to reduce the search range.
[0077] In one embodiment, a set of parameterizable kernel function templates are designed for the inference chip, and these templates should be able to cover common neural network layer types (such as convolutional layers, fully connected layers, pooling layers, etc.). The specific template parameters configured according to the parameter list population include the thread bundle horizontal coordinate index, thread bundle vertical coordinate index, thread block K value index, the amount of data each thread bundle is responsible for processing, the number of pipeline buffer levels, the data arrangement mode, the register allocation mode, the register loading calculation order, the Z value of the grid, etc. The parameter space is large, and the total search space reaches more than 80,000. The horizontal coordinate index of the thread block, the vertical coordinate index of the thread block, and the number of threads in the thread block are abandoned in the template parameters, and are replaced by thread bundles. For the tensor computing core, its most fundamental computing unit is to calculate on a per-thread bundle basis. Using the thread bundle dimension makes it easier to assign tasks and generate template code. If the thread dimension is used, it is easy to cause the final thread bundle to be not an integer, and a large amount of computing resources are wasted.
[0078] In one embodiment, by setting the size of the matrix multiplication to be processed by each thread bundle, as well as the data placement in the shared memory, etc., the definition part of the kernel function, the pipeline buffer level, and the process of transferring the result register to the global memory are configured to ensure that the result of the kernel function calculation can be efficiently transferred to the global memory for subsequent processing. On the NVIDIA platform, for universal large-size matrix multiplication, the thread bundle generally selects 4 or 8 to achieve the best performance. In addition to not changing the original data arrangement, the data arrangement has two data arrangement modes: filling and disorder. The filling mode is to add a 16-byte space after each row of data, which can significantly reduce the number of data conflicts when loading and storing shared data. The disorder mode is essentially the same as the filling mode. By reasonably arranging the position of each data, the same effect as the filling mode is achieved without using additional shared memory. The definition part of the kernel function includes memory address offset calculation, register declaration initialization, etc. For the pipeline buffer level loop, according to the inference chip architecture and parameter configuration, a reasonable pipeline buffer level is designed to reduce data access delay and calculation waiting time. According to the number of pipeline buffer levels, the head, loop body and tail of the loop will be generated in sequence. The pipeline buffer level is a method of adding a buffer level in the matrix multiplication loop process to reduce the number of synchronizations and data conflicts. The matrix multiplication kernel function is essentially a cyclic process of memory access and calculation. If the traditional memory access and calculation process is followed, thread synchronization is required after each memory access calculation. By using the pipeline buffer level to load a piece of data in advance, the memory access and calculation data in the loop body are not related, and one thread synchronization can be reduced. In one embodiment, the effect of using two to three pipeline buffer levels is best. In one embodiment, by rewriting the divergence conditions of the loop body in the pipeline buffer level, the divergence jump instructions are reduced, thereby improving the performance of the matrix multiplication. Although the divergence conditions at the tail of the pipeline buffer level will be deleted, the performance will be improved to a certain extent, indicating that the overhead caused by memory access is less than the performance benefit brought by reducing divergence statements. In one embodiment, by changing the loading and calculation order of the wmma or mma registers and modifying the start and end values of the loop body, the parallelism between registers can be fully utilized to improve the performance of the matrix multiplication kernel function.
[0079] In one embodiment, further reference is made to Figure 3 As shown, Figure 3The detailed flow diagram of step S3 is shown. The kernel function generated by the chip-specific compiler is compiled and executed on the large model inference chip, its performance is measured, and the results are recorded in the performance dictionary. Specifically, first, it is determined whether the parameter list population currently input has a corresponding key-value pair in the performance dictionary (step S31). If the parameter list population currently input does not have a corresponding key-value pair in the performance dictionary, the parameter list population is returned and the kernel function is re-acquired (step S32). And further set a test environment for the kernel function, and use the compiler to statically compile the test file (step S33). The compiled test file is directly executed to generate the kernel function performance indicators (such as execution time, memory access times, power consumption, etc.), and write them to an external record document (step S34). Finally, the record document is read to obtain the parameter list population and the corresponding kernel function performance indicators (step S35). In addition, if the parameter list population currently input has a corresponding key-value pair in the performance dictionary, the performance dictionary is directly read to obtain the kernel function performance indicators corresponding to the parameter list population (step S36). In this embodiment, it is very likely that duplicate parameter lists with exactly the same parameters will be generated during multiple rounds of parameter searches of the search algorithm. If compiled again, it will take too much time. In this regard, by using a performance dictionary, it is possible to quickly record and search for parameter lists that have been implemented, avoiding redundant compilation and wasting time.
[0080] Furthermore, in one embodiment, during the process of the compiler compiling a kernel function, parallel compilation can be selected to compile multiple different kernel functions at the same time, which greatly reduces the time required for compilation.
[0081] In one embodiment, after generating a parameter list population through multiple cycles using a given search algorithm, the method further comprises: pre-screening the parameter list population multiple times to update the parameter list population. Specifically, the parameter list population is first preliminarily screened to filter out parameter lists that do not meet the large-size matrix multiplication kernel function standard, such as some situations where the number of threads is too small to fully utilize the GPU parallelism. Then, the preliminarily screened parameter list population is passed to the constructed cost model for further screening.
[0082] In one embodiment, a cost model is constructed based on a traditional machine learning model of a support vector machine. Specifically, the data output by the performance dictionary includes a parameter list population and a corresponding kernel function performance index to construct a data set. Then, the data set is preprocessed, that is, each parameter list of the parameter list population is converted into a pure digital list, and discretely corresponds to an integer list increasing from 0; the kernel function performance index corresponding to each parameter list is compared with a given performance threshold, and a value less than the performance threshold is assigned to 0, and a value greater than the performance threshold is assigned to 1; and the kernel function that cannot run normally is identified and assigned to 0. The preprocessed data set is used to train the cost model, wherein a support vector machine is constructed in the training set, and a recall rate is evaluated in the test set. The recall rate represents the proportion of the predicted high-performance kernel function to the total high-performance kernel function. During the training process, the cost model needs to maintain a high recall rate, otherwise some high-performance parameter lists will be mistakenly eliminated during the screening process. Finally, the trained cost model is used to perform performance prediction on each parameter list of the parameter list population after preliminary screening, and the parameter list with a predicted value of 0 is screened out.
[0083] By setting the parameters of the support vector machine, try to generate a cost model with a high recall rate in the test set. The higher the recall rate, the lower the possibility of missing a high-performance parameter list, which helps to find the best performing matrix multiplication operator. In addition, it should be noted that in addition to the support vector machine, other traditional machine learning models or even models built by deep learning can be used to construct the cost model. Since the cost model is essentially a binary classification problem, it is highly scalable and different cost models can be used under the premise of providing a data set. In addition, the significance of the cost model is the pruning search process. For large-size matrix multiplication, since the matrix multiplication itself takes a long time to compile and execute, and the parameter space is huge, the search process will be too slow and it will not be possible to quickly generate a high-performance matrix multiplication kernel function. The cost model module supports further pruning of parameters, which greatly reduces the number of parameter spaces.
[0084] In addition, it should be noted that the automatic tuning of the code generator disclosed in the present invention is not limited to the order of the above steps S1-S4 and S31-S36, and the steps can be reordered, added or deleted. For example, the steps recorded in the present invention can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of the present invention can be achieved, the present invention is not limited here.
[0085] In addition, corresponding to the aforementioned embodiment of automatic tuning of the code generator, the present invention also provides an embodiment of an automatic tuning device for a code generator suitable for a large model inference chip.
[0086] See also Figure 3, an automatic tuning device 300 of a code generator provided by an embodiment of the present invention, the device comprises:
[0087] The operator search module 310 is used to define the search space according to the characteristics of a given large model inference chip, integrate a given search algorithm into the software development environment of the large model inference chip, and use the given search algorithm to generate a parameter list population after multiple rounds of cycles, wherein the parameter list population includes several parameter lists.
[0088] The code generation module 320 is used to configure specific template parameters according to the parameter list population, generate the definition part of the kernel function, the pipeline buffer level and the process of transferring the result register to the global memory.
[0089] The operator measurement module 330 is connected to the code generation module 320, and is used to receive the kernel function, compile the generated kernel function through the compiler, and execute it on the large model inference chip, perform performance measurement on the generated kernel function, obtain the kernel function performance index, and add the parameter list population and the corresponding kernel function performance index as a key-value pair to the performance dictionary.
[0090] In one embodiment, the operator measurement module 330 is designed based on a hardware measurement strategy jointly decided by compiler compilation execution and performance dictionary, and specifically includes: a GPU processor 3301, connected to the code generation module 320, used to combine the compiler and the performance dictionary to measure the performance of the kernel function, obtain the kernel function performance index, and add the parameter list population and the corresponding kernel function performance index as a key-value pair to the performance dictionary, and store the key-value pair. A selector 3302, connected to the code generation module 320, is used to determine whether the currently input parameter list population has a corresponding key-value pair in the performance dictionary, and if not, returns to the code generation module to re-acquire the kernel function.
[0091] In one embodiment, the device further includes: a control core module 340, which specifically includes: a sorter 3401, a control core 3402, and a filter 3403. The sorter 3401 is connected to the selector 3302 to sort the parameter list population in descending order according to the received kernel function performance index to obtain the sorted parameter list population. The control core 3402 is connected to the sorter 3401 and the operator search module 310 to adjust the parameters of the search algorithm according to the sorted parameter list population and feed back to the operator search module 310. The filter 3403 is connected to the operator search module 310 to preliminarily screen the parameter list population and filter out the parameter list that does not meet the large-size matrix multiplication kernel function standard.
[0092] In one embodiment, the device further comprises: a cost model module 350 connected to the selector 3302 of the operator measurement module and the filter 3403 of the control core module, for receiving the data output by the performance dictionary of the operator measurement module to construct a data set and train the cost model, and for receiving the parameter list population after preliminary screening output by the filter, and using the trained cost model to perform performance prediction on each parameter list of the parameter list population after preliminary screening, filtering out parameter lists with a prediction value of 0, and feeding back the parameter list population after re-screening to the selector 3302 of the operator measurement module.
[0093] In addition, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each module of the described device can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.
[0094] In summary, the present invention discloses an automatic tuning method and device for a code generator suitable for a large model inference chip, which uses a tensor computing core for automatic tuning to optimize the performance of large-size matrix multiplication in the training and reasoning of a large language model. By determining the appropriate operator tuning method based on the specific size of the input, and dynamically searching for the optimal parameters based on the cost model for rapid pruning, the delay in accessing shared memory can be reduced and the cache hit rate can be increased. For large-size matrix multiplications that take a lot of time to compile and execute, further constraints are imposed on the matrix partitioning and the number of threads for the large size to ensure that the performance of the tensor computing core can be fully utilized, and the time for parameter search can be reduced through multiple screenings. And by using a performance dictionary, it is possible to quickly record and search for a list of parameters that have been implemented, avoiding redundant compilation and wasting time.
[0095] According to an embodiment of the present invention, the present invention also provides a machine-readable medium, which may be a tangible medium, which may contain or store a program for use by an instruction execution system, apparatus or device or for use in conjunction with an instruction execution system, apparatus or device, and the computer program implements the above-mentioned steps of automatic tuning of the code generator when executed by a processor.
[0096] In addition, the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.
[0097] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored in a readable storage medium. When the computer program is executed by a processor, the computer can perform the above-mentioned steps of automatic tuning of the code generator.
[0098] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for automatic tuning of a code generator, characterized in that: Include: A search space is defined according to the characteristics of a given large model inference chip, a given search algorithm is integrated into a software development environment of the large model inference chip, and a parameter list population is generated through multiple cycles using the given search algorithm, wherein the parameter list population includes a plurality of parameter lists; According to the parameter list population, the specific template parameters are configured to generate the definition part of the kernel function, the pipeline buffer level, and the process of transferring the result register to the global memory; The generated kernel function is compiled by a compiler and executed on the large model inference chip, the performance of the generated kernel function is measured, the kernel function performance index is obtained, and the parameter list population and the corresponding kernel function performance index are added to the performance dictionary as a key-value pair.
2. The method according to claim 1, characterized in that Also includes: Sorting the parameter list population according to the kernel function performance index to obtain a sorted parameter list population; The sorted parameter list population feedback is used to adjust the parameters of the search algorithm.
3. The method according to claim 2, characterized in that The search algorithm adopts a genetic algorithm, and in the first cycle, the parameter list population is randomly generated within the given parameter list range; A crossover genetic probability and a mutation genetic probability are set according to the sorted parameter list population, a new round of iteration is performed, and the parameter list population is updated.
4. The method according to claim 3, characterized in that Statistics on the time spent on current tuning and the highest performance index; If the time spent is less than the tuning threshold, or the highest performance index is less than the allowed performance threshold, then increasing the probability of mutation inheritance to increase the search range; If the time spent is greater than or equal to the tuning threshold, or the highest performance index is greater than or equal to the allowable performance threshold, then the mutation inheritance probability is increased, and the mutation inheritance probability is decreased to reduce the search range.
5. The method according to claim 1, characterized in that If the parameter list population currently input does not have a corresponding key-value pair in the performance dictionary, returning the parameter list population and re-obtaining the kernel function; Setting a test environment for the kernel function and using a compiler to statically compile a test file; Directly execute the compiled test file to generate the kernel function performance index and write it into an external record document; The record document is read to obtain the parameter list population and the corresponding kernel function performance index.
6. The method according to claim 5, characterized in that If the parameter list population currently input has a corresponding key-value pair in the performance dictionary, the performance dictionary is directly read to obtain the kernel function performance index corresponding to the parameter list population.
7. The method according to claim 1, characterized in that Using the given search algorithm, after multiple cycles, the parameter list population is generated, which also includes: Pre-screening the parameter list population for multiple rounds to update the parameter list population includes: Preliminarily screening the parameter list population to filter out parameter lists that do not meet the large-size matrix multiplication kernel function standard; The parameter list population after preliminary screening is passed to the constructed cost model for further screening.
8. The method according to claim 7, characterized in that The parameter list population after preliminary screening is passed to the constructed cost model and screened again, including: Constructing a data set from the data output by the performance dictionary, including a parameter list population and corresponding kernel function performance indicators; The data set is preprocessed, including: Convert each parameter list of the parameter list population into a pure digital list, and discretely correspond it to an integer list increasing from 0; Compare the kernel function performance index corresponding to each parameter list with a given performance threshold, assign a value of 0 to a value less than the performance threshold and a value of 1 to a value greater than the performance threshold; and identify kernel functions that cannot run normally and assign a value of 0; The cost model is trained with the preprocessed data set, wherein a support vector machine is constructed in the training set, and a recall rate is evaluated in the test set, where the recall rate represents the proportion of the predicted high-performance kernel function to the total high-performance kernel functions; The trained cost model is used to perform performance prediction on each parameter list in the parameter list population after preliminary screening, and parameter lists with predicted values of 0 are screened out.
9. An automatic tuning device for a code generator, characterized in that: Include: An operator search module is used to define a search space according to the characteristics of a given large model inference chip, integrate a given search algorithm into the software development environment of the large model inference chip, and use the given search algorithm to generate a parameter list population after multiple cycles, wherein the parameter list population includes a plurality of parameter lists; A code generation module, configured to configure specific template parameters according to the parameter list population, generate a definition part of the kernel function, a pipeline buffer level, and a process of transferring result registers to a global memory; An operator measurement module is connected to the code generation module to receive the kernel function, compile the generated kernel function through a compiler, and execute it on the large model inference chip, perform performance measurement on the generated kernel function, obtain the kernel function performance index, and add the parameter list population and the corresponding kernel function performance index as a key-value pair to the performance dictionary.
10. The device according to claim 9, characterized in that Also includes: Control core module, including: A sorter, together with the operator measurement module, is used to sort the parameter list population according to the kernel function performance index received and output by the operator measurement module to obtain a sorted parameter list population; A control core connected to the sorter and the operator search module, for adjusting the parameters of the search algorithm according to the sorted parameter list population, and feeding back to the operator search module; A filter is connected to the operator search module and is used to preliminarily screen the parameter list population and filter out parameter lists that do not meet the large-size matrix multiplication kernel function standard.
11. The device according to claim 10, characterized in that Also includes: The cost model module is connected to the operator measurement module and the control core module, and is used to receive the data output by the performance dictionary of the operator measurement module to construct a data set and train the cost model, and receive the parameter list population after preliminary screening output by the filter, and use the trained cost model to perform performance prediction on each parameter list in the parameter list population after preliminary screening, filter out the parameter list with a prediction value of 0, and feed back the parameter list population after further screening to the operator measurement module.
12. The device according to claim 9, characterized in that The operator measurement module comprises: A GPU processor, for combining a compiler and a performance dictionary, performing performance measurement on the kernel function, obtaining a kernel function performance index, adding the parameter list population and the corresponding kernel function performance index as a key-value pair to the performance dictionary, and storing the key-value pair; A selector is connected to the code generation module to determine whether the parameter list population currently input has a corresponding key-value pair in the performance dictionary. If not, it returns to the code generation module to re-acquire the kernel function.
13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.