Large language model kernel generation method and system and text generation method

Abstract

The invention provides a large language model kernel generation method and system and a text generation method, and the method comprises the steps: firstly obtaining a hardware constraint parameter, and constructing a resource constraint perception rule comprising a degree of parallelism and a cyclic partitioning rule according to the hardware constraint parameter; then, space exploration is performed based on the rules and a sequence decision algorithm to generate a target kernel. The sequence decision algorithm sequentially traverses according to the memory hierarchy and the cycle dimension, the parallelism degree of each dimension and the feasible interval of the block size are determined, memory hierarchy configuration parameters are obtained step by step and finally summarized into target kernel configuration parameters, and a target large language model kernel is generated according to the target kernel configuration parameters. According to the large language model kernel generation method provided by the invention, the resource constraint perception rule is dynamically constructed based on the hardware constraint parameter of the target large language model, so that the high-performance target large language model kernel is quickly generated, the kernel generation time is shortened, and the kernel generation efficiency is improved. And the calculation efficiency and the resource utilization rate of the large language model on special hardware are improved.

Description

A method, system, and text generation method for generating large language model kernels Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method, system, and text generation method for generating a large language model kernel. Background Technology

[0002] With the continuous growth in the scale of large language models, their efficient deployment on dedicated deep learning accelerators faces severe challenges. As the underlying computational program, the performance of the kernel directly determines the execution efficiency of large language models. Traditional kernel generation methods mainly rely on manual tuning or automated search strategies. Manual methods are time-consuming and labor-intensive, and difficult to adapt to diverse hardware and models; while existing automated methods, such as optimization based on integer linear programming or heuristic space exploration, typically face the problem of the configuration space expanding exponentially with the number of variables, and containing a large number of invalid configurations due to violations of hardware resource constraints (such as memory capacity and number of processing units), leading to blind and inefficient searches; secondly, the complex dependencies between kernel variables make the design space highly non-convex, and traditional methods that jointly optimize all variables at once are prone to getting trapped in local optima, making it difficult to obtain globally high-performance configurations. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method, system, and text generation method for generating large language model kernels, thereby solving the aforementioned problems. This method, system, and text generation method dynamically construct resource constraint-aware rules based on the hardware constraint parameters of the target large language model, thereby rapidly generating a high-performance target large language model kernel. While reducing kernel generation time, this improves the computational efficiency and resource utilization of the large language model on dedicated hardware.

[0004] To address the aforementioned technical problems, this invention provides a method for generating a large language model kernel, comprising the following steps: obtaining hardware constraint parameters of the target large language model; constructing resource constraint awareness rules based on the hardware constraint parameters, the resource constraint awareness rules including parallelism rules and cyclic block rules; performing spatial exploration of the target kernel based on the parallelism rules, cyclic block rules, and a preset sequence decision algorithm to generate the target large language model kernel; wherein, the sequence decision algorithm specifically comprises: traversing several memory levels in the target kernel based on a preset hierarchical traversal order; for the currently traversed memory level, performing the following steps: traversing several cyclic dimensions in the memory level based on a preset cyclic dimension traversal order; for each cyclic dimension, obtaining the feasible interval of parallelism and the feasible interval of block size for that cyclic dimension based on the parallelism rules and the cyclic block rules; obtaining the configuration parameters of the currently traversed memory level based on the feasible intervals of parallelism and the feasible intervals of block size corresponding to several cyclic dimensions; repeating the above steps until all memory levels in the target kernel are traversed to obtain the target kernel configuration parameters; and generating the target large language model kernel based on the target kernel configuration parameters.

[0005] In the above scheme, by acquiring hardware constraint parameters and constructing resource constraint-aware rules including parallelism rules and cyclic block rules, the kernel configuration space is effectively pruned based on hardware resource constraints, reducing the search range of invalid configurations and improving search efficiency. Secondly, a pre-defined sequence decision algorithm is used to traverse the memory levels according to a pre-defined hierarchical traversal order, determining the feasible intervals for parallelism and block size layer by layer and dimension by dimension. This avoids the problem of traditional methods easily getting trapped in local optima due to deciding all variables at once. The above scheme can efficiently explore non-convex configuration spaces while meeting the resource constraints of deep learning accelerators, reducing kernel generation time while improving the computational efficiency and resource utilization of large language models on dedicated hardware.

[0006] It should be noted that the hardware constraint parameters may include the cache capacity of each memory level of the deep learning accelerator and the total number of available functional units at each level. The memory levels may include dedicated buffers, local buffers, shared buffers, and global buffers. The preset level traversal order may be from the level closest to the computation to the level furthest from the computation, for example, processing in the order of local buffer, shared buffer, and dedicated buffer. The preset loop dimension traversal order may be the loop order from the outermost to the innermost layer in general matrix multiplication, for example, the M, N, K order. The target kernel configuration parameters may include the actual block size and actual parallelism of each loop dimension under each memory level. The sequence decision algorithm is preferably a reinforcement learning algorithm. The non-convexity of the kernel configuration space stems from the complex interdependencies between kernel variables such as loop blocks and parallelism.

[0007] Further, for each loop dimension, the feasible intervals for parallelism and block size of the loop dimension are obtained based on the parallelism rules and loop block rules. Obtaining the feasible interval for parallelism of the loop dimension based on the parallelism rules includes: obtaining the total number of functional units based on the hardware constraint parameters; for each loop dimension, obtaining the number of occupied functional units based on the preset target kernel sequence; obtaining the maximum parallelism of the loop dimension based on the total number of functional units and the number of occupied functional units; obtaining unblocked loop dimensions based on the currently traversed memory level; obtaining the minimum parallelism of the loop dimension based on the unblocked loop dimensions and the maximum parallelism of the loop dimension; and obtaining the feasible interval for parallelism of the loop dimension at the currently traversed memory level based on the maximum and minimum parallelism.

[0008] In the above scheme, the total number of functional units is obtained based on hardware constraint parameters, and the maximum parallelism of the current loop dimension is determined by combining this with the number of occupied functional units. Simultaneously, based on the un-blocked loop dimensions in the current memory level, the minimum parallelism for that loop dimension that guarantees hardware utilization is derived. Thus, the parallelism search space for each loop dimension converges to a feasible interval defined by the maximum and minimum parallelism, effectively eliminating invalid configurations that exceed hardware capabilities or lead to resource idleness. Through this dimension-by-dimensional, progressive interval calculation, not only are the physical resource constraints of the deep learning accelerator satisfied, but the overall configuration search space is also compressed, improving the efficiency of subsequent space exploration and the resource utilization of the generated kernel.

[0009] It should be noted that the number of occupied functional units (NUFU) is the product of the parallelism of all loop dimensions with determined parallelism at the current memory level, excluding the current decision loop dimension. The un-blocked loop dimension refers to the loop dimension at this memory level that has not yet had its block size decided; the ntile value corresponding to the un-blocked loop dimension is the product of the original dimension size (i.e., the number of loops) of all un-blocked loop dimensions. The minimum parallelism MiP is calculated using the formula MiP=ceil(maximum parallelism MaP / ntile) to ensure that parallel efficiency and hardware resource utilization are maximized even in the case of unblocking.

[0010] Further, for each loop dimension, the feasible intervals for parallelism and block size are obtained based on the parallelism rules and loop block rules. Obtaining the feasible interval for block size based on the loop block rules includes: obtaining the cache capacity of the currently traversed memory level based on the hardware constraint parameters; obtaining the determined and undetermined loop block sizes based on a preset target kernel sequence; obtaining the matrix loop block product based on the determined and undetermined loop block sizes; determining the upper limit of the block size for the loop dimension based on the matrix loop block product and the cache capacity of the currently traversed memory level; and obtaining the feasible interval for block size for the loop dimension based on the upper limit of the block size.

[0011] In the above scheme, based on the cache capacity of the current memory level and considering both the determined and undetermined circular block sizes, the product of the circular blocks in the matrix is ​​calculated to accurately determine the upper limit of the block size for that circular dimension, thus obtaining its feasible block size range. This scheme transforms hardware storage constraints into explicit block limitations for each circular dimension, ensuring that the generated kernel configuration strictly meets cache capacity limits during data storage. By applying constraints hierarchically and dimension by dimension, all invalid block schemes that violate capacity constraints are eliminated, thereby compressing the configuration search space. Under the premise of satisfying hardware resource constraints, this scheme lays a reliable foundation for efficiently exploring block configurations and generating high-performance, resource-matched large language model kernels.

[0012] It should be noted that the calculation of the matrix cyclic block product is related to the specific matrix calculation type (such as GEMM or BMM) and the mapping relationship between the matrix and the memory level. For each matrix, its cyclic block product is equal to the product of the block sizes of each involved cyclic dimension at the current and deeper memory levels. The upper limit of the block size (MaT) is obtained by solving a constraint inequality that requires the sum of the estimated storage usage of all related matrices at that memory level to not exceed the cache capacity (BUF) of that level.

[0013] Further, the memory level configuration parameters for the current traversal are obtained based on the feasible intervals of parallelism and block size corresponding to several cyclic dimensions; including: for each cyclic dimension, inputting the feasible intervals of parallelism and block size corresponding to the cyclic dimension into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size respectively; and obtaining the current memory level configuration parameters based on the target parallelism and target block size corresponding to several cyclic dimensions.

[0014] In the above scheme, a pre-defined policy network is used to take the feasible intervals for parallelism and block size of each loop dimension as input. The policy network then determines the target parallelism and target block size for each loop dimension based on the interval constraints. This scheme directly transforms the resource-constraint-aware search space into the decision-making basis of the policy network, ensuring that the final parameter selection conforms to hardware limitations. By integrating the target values ​​of all loop dimensions, the current memory-level configuration parameters are automatically generated. This scheme utilizes the policy network to perform targeted optimization within the feasible interval, overcoming the inefficiency of traditional methods in searching large spaces, thereby improving the generation quality and decision-making efficiency of kernel configuration.

[0015] It should be noted that the state information received by the policy network is a progressively increasing kernel sequence. Each vector in the kernel sequence contains seven pieces of information: memory level, time step, loop index, maximum block size, actual block size, maximum parallelism, and actual parallelism. Based on the current state, the policy network outputs the probability distribution of the block size and parallelism of the loop dimension of the current decision within their respective feasible intervals, and determines the specific values ​​(i.e., target parallelism and target block size) through sampling.

[0016] Further, in the step of inputting the feasible intervals of parallelism and block size corresponding to the cyclic dimension into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size, the preset policy network performs the following steps: performing feature transformation based on the feasible intervals of parallelism and block size to obtain the corresponding parallelism probability distribution and block size probability distribution; and performing probability sampling based on the parallelism probability distribution and block size probability distribution to determine the target parallelism and target block size of the cyclic dimension.

[0017] In the above scheme, the preset policy network performs feature transformation on the feasible intervals of parallelism and block size, respectively, to obtain the corresponding probability distributions of parallelism and block size. Based on these, probability sampling is performed to determine the target parallelism and target block size for each loop dimension. This scheme transforms discrete hardware constraint intervals into continuous probability distributions, enabling the policy network to perform guided searches more smoothly and reasonably within the feasible space. By sampling from the probability distributions to determine specific parameters, it ensures that the final generated memory-level configuration parameters strictly adhere to resource constraints while effectively guiding towards potentially better performance areas, thereby improving the adaptability and efficiency of parameter decisions and providing effective support for the generation of high-quality kernel configurations.

[0018] It should be noted that the policy network is preferably constructed using attention blocks based on the Transformer architecture. The processing includes: encoding information such as the cycle index, block size, and parallelism in the state input through a learnable embedding layer, and adding it to the positional encoding to obtain the initial hidden state; the initial hidden state undergoes deep feature extraction and relation modeling through multiple concatenated attention blocks; finally, through network layer mapping and the application of the Softmax activation function, probability distributions corresponding to the selectable values ​​of block size and parallelism are output. The agent performs random sampling based on this probability distribution to determine the target block size and target parallelism for the current dimension.

[0019] Further, the feasible intervals of parallelism and block size corresponding to the cyclic dimension are input into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size, respectively. The policy network is then trained and updated using a preset reinforcement learning algorithm. Specifically: the target large language model kernel is input into a preset performance analysis model, so that the target large language model kernel is evaluated to obtain a kernel performance evaluation result; a reward signal is obtained based on the kernel performance evaluation result; and the parameters of the preset policy network are trained and updated based on the reward signal and the preset reinforcement learning reward algorithm.

[0020] In the above scheme, the generated target large language model kernel is input into a preset performance analysis model for performance evaluation to obtain kernel performance evaluation results. Based on these results, a reward signal is obtained, and then the parameters of a preset policy network are trained and updated using a preset reinforcement learning algorithm. This scheme utilizes the reward signal to drive the policy network to continuously learn and optimize, enabling it to adaptively adjust its decisions based on performance feedback. By guiding the update of network parameters through the reward signal, the policy network gradually associates the feasible intervals of parallelism and block size with better kernel performance, thus favoring the generation of high-performance configurations in subsequent decisions. This allows the kernel generation process to learn from iterative evaluations, effectively improving the quality and efficiency of the configurations generated by the policy network, and enhancing its convergence and adaptability when exploring resource-constrained spaces.

[0021] It should be noted that the preset performance analysis model is used to estimate or measure the execution latency or throughput of the generated kernel configuration on the target hardware. The reward signal can be negatively correlated with kernel performance to encourage the policy network to generate low-latency, high-performance kernel configurations. The preset reinforcement learning reward algorithm is preferably a policy gradient algorithm, such as the REINFORCE algorithm, and is used in conjunction with an optimizer to update the parameters of the policy network, continuously optimizing the decision policy by minimizing the expected latency (or maximizing the reward).

[0022] This invention also provides a large language model kernel generation system, comprising: a hardware parameter acquisition module for acquiring hardware constraint parameters of a target large language model; a resource constraint-aware rule construction module for constructing resource constraint-aware rules based on the hardware constraint parameters, the resource constraint-aware rules including parallelism rules and cyclic block rules; and a space exploration module for performing space exploration on the target kernel based on the parallelism rules, cyclic block rules, and a preset sequence decision algorithm to generate the target large language model kernel; wherein, the sequence decision algorithm specifically involves traversing several memory levels in the target kernel according to a preset hierarchical traversal order. For the currently traversed memory level, the following steps are performed: traverse several cyclic dimensions in the memory level according to a preset cyclic dimension traversal order; for each cyclic dimension, obtain the feasible interval of parallelism and the feasible interval of block size for that cyclic dimension based on the parallelism rule and the cyclic block rule; obtain the current memory level configuration parameters based on the feasible intervals of parallelism and the feasible intervals of block size corresponding to several cyclic dimensions; repeat the above steps until all memory levels in the target kernel are traversed to obtain the target kernel configuration parameters; the kernel generation module is used to generate the target large language model kernel based on the target kernel configuration parameters.

[0023] In the above scheme, the hardware parameter acquisition module is used to acquire hardware constraint parameters; the resource constraint awareness rule construction module constructs resource constraint awareness rules including parallelism rules and cyclic block rules based on the hardware constraint parameters; the space exploration module, according to the resource constraint awareness rules and a preset sequence decision algorithm, traverses the memory levels and their cyclic dimensions in a preset order, obtains the feasible intervals of parallelism and block size dimension by dimension, and generates memory level configuration parameters for each memory level in sequence, finally obtaining the target kernel configuration parameters; the kernel generation module generates the target large language model kernel based on this. The above scheme, through modular design and sequential decision-making, systematically transforms hardware constraints into feasible parameter intervals layer by layer and dimension by dimension, thereby compressing the configuration search space, improving the automation and reliability of the generation process, and thus efficiently generating a large language model kernel that conforms to hardware constraints and has optimized performance.

[0024] Further, in the space exploration module, for each loop dimension, the feasible intervals of parallelism and block size for that loop dimension are obtained based on the parallelism rules and loop block rules. Obtaining the feasible intervals of parallelism for that loop dimension based on the parallelism rules includes: obtaining the total number of functional units based on the hardware constraint parameters; for each loop dimension, obtaining the number of occupied functional units based on a preset target kernel sequence; obtaining the maximum parallelism of that loop dimension based on the total number of functional units and the number of occupied functional units; obtaining unblocked loop dimensions based on the currently traversed memory level; obtaining the minimum parallelism of that loop dimension based on the unblocked loop dimensions and the maximum parallelism of that loop dimension; and obtaining the feasible intervals of parallelism for that loop dimension at the currently traversed memory level based on the maximum and minimum parallelism.

[0025] In the above scheme, within the space exploration module, the total number of functional units is obtained based on hardware constraint parameters, and combined with the number of occupied functional units, the maximum parallelism of the current loop dimension is determined. Simultaneously, based on the unpartitioned loop dimensions in the current memory level, the minimum parallelism for that loop dimension that guarantees hardware utilization is derived. Thus, the parallelism search space for each loop dimension converges to a feasible interval defined by the maximum and minimum parallelism, effectively eliminating invalid configurations that exceed hardware capabilities or lead to resource idleness. Through this dimension-by-dimensional, progressive interval calculation, not only are the physical resource constraints of the deep learning accelerator satisfied, but the overall configuration search space is also compressed, improving the efficiency of subsequent space exploration and the resource utilization of the generated kernel.

[0026] Further, in the space exploration module, for each cyclic dimension, obtaining the feasible interval of parallelism and the feasible interval of block size for that cyclic dimension based on the parallelism rule and the cyclic block rule; obtaining the feasible interval of block size for that cyclic dimension based on the cyclic block rule includes: obtaining the cache capacity of the currently traversed memory level based on the hardware constraint parameters; obtaining the determined and undetermined cyclic block sizes based on the preset target kernel sequence; obtaining the matrix cyclic block product based on the determined and undetermined cyclic block sizes; determining the upper limit of the block size for that cyclic dimension based on the matrix cyclic block product and the cache capacity of the currently traversed memory level; and obtaining the feasible interval of the block size for that cyclic dimension based on the upper limit of the block size for that cyclic dimension.

[0027] In the above scheme, within the space exploration module, the total number of functional units is obtained based on hardware constraint parameters, and combined with the number of occupied functional units, the maximum parallelism of the current loop dimension is determined. Simultaneously, based on the unpartitioned loop dimensions in the current memory level, the minimum parallelism for that loop dimension that guarantees hardware utilization is derived. Thus, the parallelism search space for each loop dimension converges to a feasible interval defined by the maximum and minimum parallelism, effectively eliminating invalid configurations that exceed hardware capabilities or lead to resource idleness. Through this dimension-by-dimensional, progressive interval calculation, not only are the physical resource constraints of the deep learning accelerator satisfied, but the overall configuration search space is also compressed, improving the efficiency of subsequent space exploration and the resource utilization of the generated kernel.

[0028] This invention also provides a text generation method based on an optimized model kernel. The method obtains the optimized model kernel based on a large language model kernel generation method as described in this invention. The text generation method includes the following steps: obtaining input text, inputting the input text into a pre-trained large language model, so that the large language model performs text inference based on the input text and the optimized model kernel to obtain a text mapping result; and generating target text based on the text mapping result.

[0029] In the above solution, an optimized model kernel obtained by integrating the aforementioned large language model kernel generation method is used to perform core calculations for model inference. This optimized kernel, with its high hardware adaptability and optimized computational efficiency, reduces the inference latency and energy consumption of a single text generation. This enables the large language model to respond to user input faster in practical applications such as machine translation, intelligent question answering, and content creation, effectively improving the real-time performance and fluency of interaction. Simultaneously, the optimized model kernel enhances service capabilities by improving hardware resource utilization and supporting higher request throughput under the same computing power. Therefore, the above solution not only improves efficiency at the underlying computational level but also substantially improves the user experience and performance of various text generation services based on the large language model at the application level. Attached Figure Description

[0030] Figure 1 is a schematic flowchart of a large language model kernel generation method according to an embodiment of the present invention; Figure 2 is a schematic diagram of a large language model kernel generation system architecture according to an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please refer to Figure 1. This embodiment provides a method for generating a large language model kernel, including the following steps: Step S1: Obtain the hardware constraint parameters of the target large language model; Step S2: Construct resource constraint awareness rules based on the hardware constraint parameters, the resource constraint awareness rules including parallelism rules and cyclic block rules; Step S3: Perform spatial exploration on the target kernel based on the parallelism rules, cyclic block rules, and a preset sequence decision algorithm to generate the target large language model kernel; wherein, the sequence decision algorithm specifically includes: Step S31: Traverse several memory levels in the target kernel based on a preset hierarchical traversal order, and for the currently traversed memory level, execute... The following steps are as follows: Step S311: Traverse several loop dimensions in the memory hierarchy according to the preset loop dimension traversal order; Step S312: For each loop dimension, obtain the feasible interval of parallelism and the feasible interval of block size for that loop dimension based on the parallelism rule and the loop block rule; Step S313: Obtain the current memory hierarchy configuration parameters based on the feasible intervals of parallelism and the feasible intervals of block size corresponding to several loop dimensions; Step S314: Repeat the above steps until all memory hierarchy levels in the target kernel are traversed to obtain the target kernel configuration parameters; Step S4: Generate the target large language model kernel based on the target kernel configuration parameters.

[0033] In this embodiment, by acquiring hardware constraint parameters and constructing resource constraint-aware rules including parallelism rules and cyclic block rules, the kernel configuration space is effectively pruned based on hardware resource constraints, reducing the search range of invalid configurations and improving search efficiency. Secondly, a preset sequence decision algorithm is employed, traversing memory levels according to a preset hierarchical traversal order, determining the feasible intervals for parallelism and block size layer by layer and dimension by dimension. This avoids the problem of traditional methods easily getting trapped in local optima due to deciding all variables at once. This embodiment can efficiently explore non-convex configuration spaces while meeting the resource constraints of deep learning accelerators, reducing kernel generation time while improving the computational efficiency and resource utilization of large language models on dedicated hardware.

[0034] Further, for each loop dimension, the feasible intervals for parallelism and block size of the loop dimension are obtained based on the parallelism rules and loop block rules. Obtaining the feasible interval for parallelism of the loop dimension based on the parallelism rules includes: obtaining the total number of functional units based on the hardware constraint parameters; for each loop dimension, obtaining the number of occupied functional units based on the preset target kernel sequence; obtaining the maximum parallelism of the loop dimension based on the total number of functional units and the number of occupied functional units; obtaining unblocked loop dimensions based on the currently traversed memory level; obtaining the minimum parallelism of the loop dimension based on the unblocked loop dimensions and the maximum parallelism of the loop dimension; and obtaining the feasible interval for parallelism of the loop dimension at the currently traversed memory level based on the maximum and minimum parallelism.

[0035] In this embodiment, the total number of functional units is obtained based on hardware constraint parameters, and the maximum parallelism of the current loop dimension is determined by combining this with the number of occupied functional units. Simultaneously, based on the un-blocked loop dimensions in the current memory level, the minimum parallelism for that loop dimension that guarantees hardware utilization is derived. Thus, the parallelism search space for each loop dimension is converged to a feasible interval defined by the maximum and minimum parallelism, effectively eliminating invalid configurations that exceed hardware capabilities or lead to resource idleness. Through dimension-by-dimensional, progressive interval calculations, not only are the physical resource constraints of the deep learning accelerator satisfied, but the overall configuration search space is also compressed, improving the efficiency of subsequent space exploration and the resource utilization of the generated kernel.

[0036] In one embodiment, a method for generating a large language model kernel is provided, specifically illustrating the implementation of the parallelism rule in the resource constraint-aware rules. The parallelism rule is used to determine the effective range of parallelism values ​​for each loop dimension in the target kernel at a specific memory level of the deep learning accelerator. The core of the parallelism rule is to ensure that the total configured parallelism does not exceed the total number of hardware functional units, while minimizing idle functional units, thereby achieving efficient utilization of hardware resources under strict resource constraints.

[0037] It should be noted that the cyclic dimensions are related to the specific computational task. Taking Generalized Matrix Multiplication (GEMM) as an example, if the dimensions of matrix A and matrix B are (M,K) and (K,N) respectively, then the cyclic dimensions involved in the computation mainly include M, N, and K. Furthermore, L can represent the batch size in Batch Matrix Multiplication (BMM).

[0038] First, the key symbols involved in the implementation are defined. Memory levels are denoted as `level`, such as global buffers and local buffers. The kernel sequence `U` represents the kernel configuration sequence generated step-by-step during the sequence decision process. `NTFU` represents the total number of functional units in the current memory level; its value depends on the hardware configuration of the deep learning accelerator and is a fixed constant during the algorithm. `MaP` represents the maximum parallelism of the currently discussed loop dimension at the current memory level. `MiP` represents the corresponding minimum parallelism. `para_arr` records the decided parallelism for each memory level and loop dimension; for example, `para_arr[level]` extracts the parallelism of all loop dimensions at the specified level to form a vector. `tile_arr` records the decided block size for each memory level and loop dimension, and its processing is similar to `para_arr`. `NUFU` represents the product of the parallelism values ​​of the decided loop dimensions at the current memory level, i.e., the number of functional units already occupied. `ntile` represents the product of the original dimension sizes of all loop dimensions that have not yet been processed into blocks at the current memory level.

[0039] Based on the above definition, at the current decision time step, the specific implementation of obtaining the feasible interval of parallelism for the currently traversed memory level and the current loop dimension to be decided is as follows: First, obtain the total number of functional units (NTFU) of the current memory level based on the hardware constraint parameters.

[0040] Next, calculate the number of occupied functional units (NUFU), which is equal to the product of the parallelism of all cyclic dimensions that have been decided at the current level, i.e., NUFU = prod(para_arr[level]). For example, if the decision order is set to M, N, K, when initially discussing the parallelism of dimension M, no dimension has been decided yet, so NUFU = 1; when discussing the parallelism of dimension N, if the parallelism of dimension M has been determined to be 16, then NUFU = 16; when discussing the parallelism of dimension K, if the parallelism of dimensions M and N has been determined to be 16 and 4 respectively, then NUFU = 16 × 4 = 64.

[0041] Then, the maximum parallelism of the current dimension is obtained based on the total number of functional units and the number of functional units already occupied, i.e., MaP = floor(NTFU / NUFU). This ensures that the sum of the parallelism configured for the current dimension and the number of functional units already occupied does not exceed the total number of functional units NTFU.

[0042] Simultaneously, it is necessary to obtain the information of unblocked cyclic dimensions based on the current memory level and calculate ntile. ntile is equal to the product of the original sizes of all unblocked cyclic dimensions. For example, for a GEMM task with M=512, N=768, and K=3072, when discussing the parallelism of dimension M, if dimensions N and K are not blocked, then ntile=768×3072; when discussing the parallelism of dimension N, if only dimension K is not blocked, then ntile=3072.

[0043] Subsequently, the minimum parallelism is obtained based on the unblocked cyclic dimension and the maximum parallelism of the cyclic dimension, i.e., MiP=ceil(MaP / ntile).

[0044] Finally, based on the obtained maximum parallelism MaP and minimum parallelism MiP, the feasible range of parallelism for this loop dimension at the current memory level can be determined as [MiP, MaP].

[0045] In this embodiment, by combining fixed hardware constraint parameters with the dynamic decision-making process of kernel configuration, the feasible interval of parallelism is calculated progressively dimension by dimension. This embodiment narrows the parallelism search space of each dimension from an original broad range to a compact interval defined by the maximum and minimum parallelism, thereby effectively eliminating configurations that exceed the capabilities of hardware physical resources and configurations that lead to severe idleness of hardware functional units. This significantly reduces invalid searches in subsequent explorations, improves the overall efficiency of kernel generation, and enhances the resource utilization of the final generated kernel on dedicated hardware.

[0046] Further, for each loop dimension, the feasible intervals for parallelism and block size are obtained based on the parallelism rules and loop block rules. Obtaining the feasible interval for block size based on the loop block rules includes: obtaining the cache capacity of the currently traversed memory level based on the hardware constraint parameters; obtaining the determined and undetermined loop block sizes based on a preset target kernel sequence; obtaining the matrix loop block product based on the determined and undetermined loop block sizes; determining the upper limit of the block size for the loop dimension based on the matrix loop block product and the cache capacity of the currently traversed memory level; and obtaining the feasible interval for block size for the loop dimension based on the upper limit of the block size.

[0047] In this embodiment, based on the cache capacity of the current memory level and according to the determined and undetermined circular block sizes, the matrix circular block product is calculated to accurately determine the upper limit of the block size for that circular dimension, thereby obtaining its feasible block size range. This embodiment transforms the hardware storage constraints into explicit block restrictions for each circular dimension, ensuring that the generated kernel configuration strictly meets the cache capacity limit when storing data. By applying constraints layer by layer and dimension by dimension, all invalid block schemes that violate capacity constraints are eliminated, thereby compressing the configuration search space. This embodiment, while ensuring that hardware resource constraints are met, lays a reliable foundation for efficiently exploring block configurations and generating high-performance and resource-matched large language model kernels.

[0048] In one embodiment, a method for generating a large language model kernel is provided, specifically illustrating the implementation of the cyclic block rule in the resource constraint-aware rules. The cyclic block rule is used to determine the effective upper limit (MaT) of the block size for each cyclic dimension in the target kernel at a specific memory level of a deep learning accelerator (DLA). Its core objective is to ensure that the total size of matrix data blocks stored in each level of memory does not exceed the physical cache capacity (BUF) of that level, thereby satisfying hardware storage constraints, while simultaneously improving data locality and computational efficiency through optimized data block partitioning.

[0049] It should be noted that the block size refers to the size of the original cyclic dimension divided into several sub-blocks. The cyclic block division rule derives the feasible range of block sizes by constructing and solving capacity constraint inequalities, level by level and dimension by dimension.

[0050] First, the key symbols involved in the implementation process are defined: BUF: represents the cache capacity of the currently traversed memory level. Its value depends on the hardware configuration of DLA and is a fixed constant during the algorithm execution.

[0051] Loop block product (e.g., Loop_M_tile): Represents the maximum number of elements in a given dimension (e.g., dimension M) contained in the final sub-blocks obtained by partitioning along a certain loop dimension in a specified memory level and all subsequent levels closer to the computation unit. It is calculated as follows: starting from the memory level closest to the computation unit, the block size (tile_arr[level,dim]) of each level in a certain dimension is multiplied by the block size of the previous level in that dimension, and so on, up to the level currently being discussed. In the innermost computation level, each sub-block typically contains only a fixed number of elements (e.g., 1), and this value can be used as the starting point for recursive computation.

[0052] Matrix block product (e.g., Mat_A_tile): Represents the theoretical maximum cache space required to store a matrix (e.g., matrix A) at a specific memory level, measured by the number of elements it contains. Its value is equal to the product of the "circular block products" of all the loop dimensions involved in the matrix. For example, in General Matrix Multiplication (GEMM), for matrix A (dimension M×K), Mat_A_tile = Loop_M_tile × Loop_K_tile; in Batch Matrix Multiplication (BMM), for matrix A (dimension L×M×K), Mat_A_tile = Loop_L_tile × Loop_M_tile × Loop_K_tile.

[0053] Based on the above definitions, at the current decision time step, the specific implementation of obtaining the feasible range of block size for the currently traversed memory level and the currently undecided loop dimension is as follows: First, obtain the cache capacity BUF of the currently traversed memory level based on the hardware constraint parameters. Simultaneously, based on a preset kernel decision sequence, obtain the state information of the determined loop block size and the undetermined loop block size.

[0054] Next, based on the determined and undetermined cyclic block sizes, the cyclic block product of each matrix involved is calculated. Specifically, for each cyclic dimension, its cyclic block product is equal to the product of the upper limit of the block size MaT[dim] (i.e. the variable to be solved) of that dimension, which is being decided at the current level, and the determined block sizes of that dimension in all inner memory levels closer to the computation unit.

[0055] Subsequently, based on the calculated cyclic block products of each dimension, the matrix block products (Mat_A_tile, Mat_B_tile, Mat_C_tile, etc.) of each participating matrix are further calculated at the current memory level. This step summarizes the multi-dimensional block information of each matrix to determine its total data block size.

[0056] Then, the upper limit of the block size for this loop dimension is determined based on the block product of the matrix and the cache capacity of the currently traversed memory level. Specifically, this involves constructing and solving the capacity constraint inequality: The physical meaning of the capacity constraint inequality is that the total storage space occupied by all matrix data blocks that need to reside simultaneously in the current memory level cannot exceed the hardware cache capacity BUF of that level. By solving this inequality (usually by separating the variables in the inequality, i.e., the variable to be solved, MaT[dim]), the maximum allowed value of the block size upper limit MaT[dim] for each cyclic dimension can be analytically obtained.

[0057] Finally, based on the upper limit of the block size of this cyclic dimension, combined with the actual situation (such as the original size of this dimension), the feasible range of its block size can be obtained, which is usually an integer range from 1 to MaT[dim].

[0058] This embodiment transforms the physical storage constraints of the hardware (i.e., cache capacity) into explicit mathematical limitations on the block size for each circular dimension. By constructing a capacity constraint inequality with block size as the variable, and recursively calculating the data block size based on the determined and undetermined block states, the upper limit of each dimension's block size can be accurately derived. This ensures that during the hierarchical and dimension-by-dimensional block decision-making process, any invalid block partitioning schemes that might violate the cache capacity limit are preemptively excluded from the search space. This not only strictly guarantees the feasibility of the generated kernel configuration in terms of data storage but also greatly compresses the configuration space that needs to be explored, laying a reliable foundation for efficiently searching for high-performance kernel configurations under resource constraints.

[0059] Further, the current memory level configuration parameters are obtained based on the feasible intervals of parallelism and block size corresponding to several cyclic dimensions; including: for each cyclic dimension, inputting the feasible intervals of parallelism and block size corresponding to the cyclic dimension into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size respectively; and obtaining the current memory level configuration parameters based on the target parallelism and target block size corresponding to several cyclic dimensions.

[0060] In this embodiment, a preset policy network is used as input to take the feasible intervals for parallelism and block size of each loop dimension. The policy network then determines the target parallelism and target block size for each loop dimension based on the interval constraints. This embodiment directly transforms the resource constraint-aware search space into the decision basis of the policy network, ensuring that the final parameter selection conforms to hardware limitations. By integrating the target values ​​of all loop dimensions, the current memory level configuration parameters are automatically generated. This embodiment utilizes the policy network for targeted optimization within the feasible interval, overcoming the inefficiency of traditional methods in searching large spaces, thereby improving the generation quality and decision efficiency of kernel configuration.

[0061] Further, in the step of inputting the feasible intervals of parallelism and block size corresponding to the cyclic dimension into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size, the preset policy network performs the following steps: performing feature transformation based on the feasible intervals of parallelism and block size to obtain the corresponding parallelism probability distribution and block size probability distribution; and performing probability sampling based on the parallelism probability distribution and block size probability distribution to determine the target parallelism and target block size of the cyclic dimension.

[0062] In this embodiment, the preset policy network performs feature transformation on the feasible intervals of parallelism and block size to obtain corresponding probability distributions of parallelism and block size. Based on these distributions, probability sampling is performed to determine the target parallelism and target block size for each loop dimension. This embodiment transforms discrete hardware constraint intervals into continuous probability distributions, enabling the policy network to perform guided searches more smoothly and reasonably within the feasible space. By sampling from the probability distributions to determine specific parameters, it ensures that the final generated memory-level configuration parameters strictly adhere to resource constraints while effectively guiding towards potentially higher-performance regions. This improves the adaptability and efficiency of parameter decision-making, providing effective support for the generation of high-quality kernel configurations.

[0063] Further, the feasible intervals of parallelism and block size corresponding to the cyclic dimension are input into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size, respectively. The policy network is then trained and updated using a preset reinforcement learning algorithm. Specifically: the target large language model kernel is input into a preset performance analysis model, so that the target large language model kernel is evaluated to obtain a kernel performance evaluation result; a reward signal is obtained based on the kernel performance evaluation result; and the parameters of the preset policy network are trained and updated based on the reward signal and the preset reinforcement learning reward algorithm.

[0064] In this embodiment, the generated target large language model kernel is input into a preset performance analysis model for performance evaluation, yielding kernel performance evaluation results. A reward signal is then obtained based on these results, and the parameters of a preset policy network are trained and updated using a preset reinforcement learning algorithm. This embodiment utilizes the reward signal to drive the policy network to continuously learn and optimize, enabling it to adaptively adjust its decisions based on performance feedback. By guiding the update of network parameters through the reward signal, the policy network gradually associates the feasible intervals of parallelism and block size with better kernel performance, thus favoring the generation of high-performance configurations in subsequent decisions. This allows the kernel generation process to learn from iterative evaluations, effectively improving the quality and efficiency of the configurations generated by the policy network, and enhancing its convergence and adaptability when exploring resource-constrained spaces.

[0065] In one embodiment, a method for generating large language model kernels is provided, specifically illustrating the application of a reinforcement learning-based sequence decision algorithm in resource-constraint-aware kernel generation. The sequence decision algorithm applies a reinforcement learning framework to the automatic generation of kernel configurations, where a deep learning accelerator (DLA), its hardware constraints, and performance targets together constitute the environment. A policy network acts as an agent, interacting with the environment to gradually construct a complete kernel configuration through sequential decision-making.

[0066] It should be noted that this embodiment constructs resource constraint-aware rules and pre-calculates the feasible interval for each decision step, thereby directly transforming the physical limitations of the hardware into the decision boundary of the policy network, ensuring that the exploration process always remains within the effective configuration space.

[0067] In this embodiment, the state is defined as a sequence of kernels that have been generated step by step. Each vector in the kernel sequence corresponds to the result of a specific decision time step and contains seven pieces of information: memory level, reinforcement learning time step, cycle index, maximum block size (MaT), actual block size, maximum parallelism (MaP), and actual parallelism. The memory level, time step, and cycle index together locate the dimension and context to be decided. The processing order of the memory level follows the principle of moving from the nearest computational unit to the furthest, for example, in the order of local buffer, shared buffer, and dedicated buffer; the global buffer, due to its sufficient capacity, can have its configuration automatically calculated from previous decisions. The decision order for the cyclic dimension is predefined as the conventional order of Generalized Matrix Multiplication (GEMM), for example, M, N, K from the outside in.

[0068] At each decision time step, based on the current state (i.e., the existing kernel sequence) and hardware constraints, the feasible intervals for block size (upper limit MaT) and parallelism (upper limit MaP, lower limit MiP) of the current decision loop dimension can be calculated using the aforementioned resource constraint awareness rules. These two intervals define the hard constraint range of the current decision.

[0069] The process of generating specific configuration parameters for the current loop dimension based on the feasible interval is implemented by a parameterized policy network. Specifically, the feasible intervals for parallelism and block size corresponding to the loop dimension are input into a preset policy network, so that the policy network determines the target parallelism and target block size of the loop dimension based on the feasible intervals for parallelism and block size, respectively.

[0070] The policy network described in this embodiment is constructed using attention blocks based on the Transformer architecture. Its decision-making process is as follows: First, various information from the state sequence (cycle index, block size, parallelism, etc.) is embedded through a learnable embedding layer. Encode and match with the corresponding numerical encoding. Combined, an initial embedding representation is generated. Then, position encoding is added. By incorporating time step information, we obtain .then, Passing in sequence Deep feature transformation is performed on a series of concatenated attention blocks: Finally, the network is based on the output of the last attention block. Calculate the probability distributions for choosing block size and parallelism respectively: It should be noted that the above feature transformation process essentially completes the feature transformation of the input feasible interval, mapping the discrete hardware constraint interval into a smooth, guided parallelism probability distribution and block size probability distribution. The policy network then performs probability sampling based on the parallelism probability distribution and block size probability distribution to determine the specific values ​​of the target parallelism and target block size for that cyclic dimension. This embodiment ensures that the decision is made within the feasible interval, while giving the network the ability to explore better configurations.

[0071] The target block size and target parallelism determined in the current time step will be organized with the current state information into a new vector and appended to the end of the kernel sequence as input for the next decision time step. This process is repeated iteratively until decisions are made for all cyclic dimensions of all predetermined memory levels, thereby generating a complete set of target kernel configuration parameters.

[0072] Furthermore, the policy network is trained and updated using a preset reinforcement learning algorithm. During training, the target large language model kernel, generated based on complete configuration parameters, is input into a preset performance analysis model for performance evaluation, yielding kernel performance evaluation results (such as calculating the reciprocal of latency). A reward signal is obtained based on the kernel performance evaluation results. This reward signal is used to train and update the parameters of the preset policy network using a preset reinforcement learning reward algorithm (such as the REINFORCE algorithm). The training objective is typically to minimize latency; in this embodiment, the Adam optimizer is used for optimization.

[0073] In this embodiment, a reinforcement learning framework is used to connect hardware constraints, sequential decision-making, and performance feedback in a closed loop. The policy network conducts targeted exploration within the feasible range provided by resource constraint-aware rules, continuously optimizing itself using reward signals generated by performance evaluation. This enables the policy network to gradually learn to associate specific parallelism and block size selections with higher kernel performance, thereby adaptively generating high-performance kernel configurations while meeting strict hardware limitations, thus improving the efficiency and quality of automatic kernel generation.

[0074] This embodiment also provides a large language model kernel generation system, including: a hardware parameter acquisition module for acquiring hardware constraint parameters of a target large language model; a resource constraint awareness rule construction module for constructing resource constraint awareness rules based on the hardware constraint parameters, the resource constraint awareness rules including parallelism rules and cyclic block rules; and a space exploration module for performing space exploration on the target kernel based on the parallelism rules, cyclic block rules, and a preset sequence decision algorithm to generate the target large language model kernel; wherein, the sequence decision algorithm specifically involves traversing several memory levels in the target kernel according to a preset hierarchical traversal order. For the currently traversed memory level, the following steps are performed: traverse several cyclic dimensions in the memory level according to a preset cyclic dimension traversal order; for each cyclic dimension, obtain the feasible interval of parallelism and the feasible interval of block size for that cyclic dimension based on the parallelism rule and the cyclic block rule; obtain the current memory level configuration parameters based on the feasible intervals of parallelism and the feasible intervals of block size corresponding to several cyclic dimensions; repeat the above steps until all memory levels in the target kernel are traversed to obtain the target kernel configuration parameters; the kernel generation module is used to generate the target large language model kernel based on the target kernel configuration parameters.

[0075] In this embodiment, the hardware parameter acquisition module is used to acquire hardware constraint parameters; the resource constraint awareness rule construction module constructs resource constraint awareness rules including parallelism rules and cyclic block rules based on the hardware constraint parameters; the space exploration module, according to the resource constraint awareness rules and a preset sequence decision algorithm, traverses the memory levels and their cyclic dimensions in a preset order, obtains the feasible intervals of parallelism and block size dimension by dimension, and generates memory level configuration parameters for each memory level in sequence, finally obtaining the target kernel configuration parameters; the kernel generation module generates the target large language model kernel based on this. This embodiment, through modular design and sequential decision-making, systematically transforms hardware constraints into feasible parameter intervals layer by layer and dimension by dimension, thereby compressing the configuration search space, improving the automation and reliability of the generation process, and thus enabling efficient generation of a large language model kernel that conforms to hardware constraints and has optimized performance.

[0076] It should be noted that the space exploration module integrates a reinforcement learning environment and an intelligent agent. The environment encapsulates hardware constraints, performance analysis models, and reward calculation logic. The intelligent agent includes the policy network, used for sequential decision-making based on state sequences. The kernel generation module converts the final target kernel configuration parameters into specific program code or configuration instructions executable on the target deep learning accelerator.

[0077] Further, in the space exploration module, for each loop dimension, the feasible intervals of parallelism and block size for that loop dimension are obtained based on the parallelism rules and loop block rules. Obtaining the feasible intervals of parallelism for that loop dimension based on the parallelism rules includes: obtaining the total number of functional units based on the hardware constraint parameters; for each loop dimension, obtaining the number of occupied functional units based on a preset target kernel sequence; obtaining the maximum parallelism of that loop dimension based on the total number of functional units and the number of occupied functional units; obtaining unblocked loop dimensions based on the currently traversed memory level; obtaining the minimum parallelism of that loop dimension based on the unblocked loop dimensions and the maximum parallelism of that loop dimension; and obtaining the feasible intervals of parallelism for that loop dimension at the currently traversed memory level based on the maximum and minimum parallelism.

[0078] In this embodiment, in the space exploration module, the total number of functional units is obtained based on hardware constraint parameters, and the maximum parallelism of the current loop dimension is determined by combining this with the number of occupied functional units. Simultaneously, based on the unpartitioned loop dimensions in the current memory level, the minimum parallelism for that loop dimension that guarantees hardware utilization is derived. Thus, the parallelism search space for each loop dimension is converged into a feasible interval defined by the maximum and minimum parallelism, effectively eliminating invalid configurations that exceed hardware capabilities or lead to resource idleness. This dimension-by-dimensional, progressive interval calculation not only satisfies the physical resource constraints of the deep learning accelerator but also compresses the overall configuration search space, improving the efficiency of subsequent space exploration and the resource utilization of the generated kernel.

[0079] Further, in the space exploration module, for each cyclic dimension, obtaining the feasible interval of parallelism and the feasible interval of block size for that cyclic dimension based on the parallelism rule and the cyclic block rule; obtaining the feasible interval of block size for that cyclic dimension based on the cyclic block rule includes: obtaining the cache capacity of the currently traversed memory level based on the hardware constraint parameters; obtaining the determined and undetermined cyclic block sizes based on the preset target kernel sequence; obtaining the matrix cyclic block product based on the determined and undetermined cyclic block sizes; determining the upper limit of the block size for that cyclic dimension based on the matrix cyclic block product and the cache capacity of the currently traversed memory level; and obtaining the feasible interval of the block size for that cyclic dimension based on the upper limit of the block size for that cyclic dimension.

[0080] In this embodiment, in the space exploration module, the total number of functional units is obtained based on hardware constraint parameters, and the maximum parallelism of the current loop dimension is determined by combining this with the number of occupied functional units. Simultaneously, based on the unpartitioned loop dimensions in the current memory level, the minimum parallelism for that loop dimension that guarantees hardware utilization is derived. Thus, the parallelism search space for each loop dimension is converged into a feasible interval defined by the maximum and minimum parallelism, effectively eliminating invalid configurations that exceed hardware capabilities or lead to resource idleness. This dimension-by-dimensional, progressive interval calculation not only satisfies the physical resource constraints of the deep learning accelerator but also compresses the overall configuration search space, improving the efficiency of subsequent space exploration and the resource utilization of the generated kernel.

[0081] It should be noted that in the space exploration module, when calculating the cyclic block product of matrices, a corresponding storage occupancy estimation model needs to be constructed based on the matrix calculation type (GEMM / BMM) and the mapping relationship of data in each memory level (e.g., storing only blocks of a specific matrix in a certain level). The upper limit of the block size is obtained by solving a linear or nonlinear inequality with this upper limit as a variable to ensure that data storage does not overflow.

[0082] In one embodiment, a large language model kernel generation system is provided to generate a kernel configuration adapted to the TensorCore deep learning accelerator for a general matrix multiplication (GEMM) operator derived from the GPT-1 neural network module of the language model. The GEMM calculation involves three matrices: , where the dimensions M=512, K=768, N=3072.

[0083] First, the constraint parameters of the target hardware, namely TensorCore, are obtained through the hardware parameter acquisition module. Specifically, the TensorCore adopts a multi-level storage hierarchy structure including a dedicated buffer (L0), a local buffer (L1), a shared buffer (L2), and a global buffer (L3). This embodiment takes a single sub-core as the object and generates its configuration under the three memory levels of L0, L1, and L2. The hardware constraint parameters specifically include: the total number of available functional units (NTFU) in the local buffer (L1) of each sub-core is 256, the capacity of the shared buffer (L2) is 256KB, and the capacity of the local buffer (L1) is 2KB.

[0084] Subsequently, the resource constraint awareness rule construction module constructs parallelism rules and cyclic block rules based on the aforementioned hardware parameters. The space exploration module then performs space exploration and decision-making based on these two rules and a preset sequence decision algorithm, according to the kernel configuration.

[0085] This embodiment models the kernel generation process as a sequential decision-making process. At each decision time step, for a specific loop dimension under a specific memory level, the actual parallelism and block size are decided. To facilitate step-by-step decision-making, unconfigured parameters are temporarily set to the default value of 1. The state information on which the decision is based is organized into a kernel configuration vector, containing seven items: memory level, time step, loop index, maximum block size (MaT), actual block size, maximum parallelism (MaP), and actual parallelism.

[0086] In this embodiment, the decision-making process of the space exploration module is as follows: Initial time step (t=0), for the loop dimension "M" of the local buffer (L1). At this time, according to the parallelism rule: the number of used functional units NUFU=1, the maximum parallelism MaP=NTFU / NUFU=256 / 1=256. The unblocked dimensions are N and K, and their original size product ntile=768×3072, so the minimum parallelism MiP=ceil(MaP / ntile)=ceil(256 / (768×3072))=1. Therefore, the feasible parallelism interval is [1, 256]. Simultaneously, according to the loop blocking rule, since no blocking has been performed, the maximum block size MaT of dimension M is equal to its loop count 512. Based on this state, the preset sequence decision algorithm (represented as a reinforcement learning agent in this embodiment) makes a decision, setting the actual parallelism of loop "M" to 16 and the actual block size to 16.

[0087] In the next time step (t=1), for the cyclic dimension "N" of the local buffer (L1), the parallelism rule is as follows: the number of used functional units NUFU = 16 (M dimension) × 1 (K dimension, default) = 16, so MaP = 256 / 16 = 16. Only K remains as the unpartitioned dimension, ntile = 3072, so MiP = ceil(16 / 3072) = 1, and the feasible parallelism interval is [1, 16]. For the block size, the cyclic partitioning rule needs to be applied, and the mapping relationship of matrix C stored in the local buffer L1 needs to be considered. The current level M dimension has been partitioned into 16 blocks, which must satisfy the inequality: (16 × 1) × (MaT × 1) ≤ 2048 (2KB). Solving this, the maximum block size MaT for dimension N is found to be ≤ 128. Based on this, the sequence decision algorithm sets the actual parallelism of the cyclic "N" to 4, and the actual block size to 128.

[0088] In subsequent time steps, the space exploration module continues to generate configuration vectors for the remaining cyclic dimensions (such as the K dimension of the L1 layer) and the cyclic dimensions of other memory levels (L2, L0) in a predetermined order, according to the aforementioned rules and algorithms. Each decision is constrained by the results of all previous decisions, ensuring that the search is conducted within a high-quality feasible space.

[0089] Once all the circular dimensions of all constrained memory levels have been configured, the kernel generation module receives the final target kernel configuration parameters and generates a complete kernel solution that can be directly executed on the target TensorCore.

[0090] In this embodiment, the block partitioning strategy, parallelism configuration, and data mapping relationships for GEMM computation at each memory level are clearly defined. Since the generation process strictly follows the resource constraint rules derived from hardware parameters, the generated kernel configuration is hardware-executable and can effectively utilize hardware resources, thereby improving computational efficiency.

[0091] This embodiment also provides a text generation method based on an optimized model kernel. The optimized model kernel is obtained based on a large language model kernel generation method as described in this invention. The text generation method includes the following steps: obtaining input text, inputting the input text into a pre-trained large language model, so that the large language model performs text reasoning based on the input text and the optimized model kernel to obtain a text mapping result; and generating target text based on the text mapping result.

[0092] In this embodiment, by integrating the optimized model kernel obtained from the aforementioned large language model kernel generation method, the core computation of model inference is performed using the optimized model kernel, which boasts high hardware adaptability and optimized computational efficiency. This reduces the inference latency and energy consumption of a single text generation. This enables the large language model to respond to user input more quickly in practical applications such as machine translation, intelligent question answering, and content creation, effectively improving the real-time performance and fluency of interaction. Simultaneously, the optimized model kernel enhances service capabilities by improving hardware resource utilization and supporting higher request throughput under the same computing power. Therefore, this embodiment not only improves efficiency at the underlying computational level but also substantially improves the user experience and performance of various text generation services based on the large language model at the application level.

[0093] It should be noted that the optimized model kernel refers to the implementation of core computational operators (such as GEMM) optimized for the target hardware platform, generated by the large language model kernel generation method described in this invention. Integrating such an optimized kernel into the inference engine of a large language model can accelerate matrix calculations during the model's forward propagation process. The text generation method covers application scenarios including but not limited to machine translation, intelligent question answering, dialogue generation, text summarization, code generation, and all natural language processing tasks that generate text based on autoregressive or non-autoregressive methods.

[0094] In one embodiment, the text generation method based on an optimized model kernel provided by the present invention can be applied to actual inference tasks of large language models to accelerate their computation process. This method first utilizes a aforementioned large language model kernel generation method to generate a high-performance optimized model kernel for the target hardware (e.g., TensorCore) and the target computation operator (e.g., GEMM in a fully connected layer).

[0095] The specific process of text generation using this optimized model kernel is as follows: In machine translation scenarios, the system obtains the source language text to be translated as input text and inputs it into a pre-trained large language model (e.g., a model based on the Transformer architecture). During the inference process, the key GEMM calculations involved in its fully connected layers and other modules will no longer use a general or unoptimized kernel, but will instead call the model kernel generated by the method of this invention, which is optimized for specific hardware. Because the optimized kernel significantly reduces computational latency and energy consumption, the large language model can complete the layer-by-layer mapping and inference calculations from the source language to the target language more quickly, thereby rapidly obtaining the text mapping result (i.e., the logical representation of the target language) and ultimately generating fluent and accurate target text.

[0096] Similarly, in application scenarios such as intelligent question answering, text continuation, and content summarization, when a user inputs a question or the preceding text, the large language model with an optimized model kernel can perform inference calculations with lower latency, thereby generating answers or subsequent text faster and effectively improving the real-time performance of the interaction and the system response speed.

[0097] It should be noted that the large language model kernel generation method described in this invention improves computational efficiency by generating higher-quality kernel configurations; its application is not limited to text generation tasks. Any deep learning model application involving a large number of matrix multiplications or similar computational structures, such as multimodal tasks like image synthesis, speech recognition, and video understanding, can have its related processes accelerated by applying the high-performance kernel generated by this invention, thereby reducing overall task processing latency and improving responsiveness and energy efficiency.

[0098] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating a large language model kernel, characterized in that, The process includes the following steps: obtaining hardware constraint parameters of the target large language model; constructing resource constraint awareness rules based on the hardware constraint parameters, the resource constraint awareness rules including parallelism rules and cyclic block rules; performing spatial exploration of the target kernel based on the parallelism rules, cyclic block rules, and a preset sequence decision algorithm to generate the target large language model kernel; wherein, the sequence decision algorithm specifically involves: traversing several memory levels in the target kernel based on a preset hierarchical traversal order; for the currently traversed memory level, performing the following steps: traversing several cyclic dimensions in the memory level based on a preset cyclic dimension traversal order; for each cyclic dimension, obtaining the feasible interval of parallelism and the feasible interval of block size for that cyclic dimension based on the parallelism rules and the cyclic block rules; obtaining the current memory level configuration parameters based on the feasible intervals of parallelism and the feasible intervals of block size corresponding to several cyclic dimensions; repeating the above steps until all memory levels in the target kernel have been traversed to obtain the target kernel configuration parameters; and generating the target large language model kernel based on the target kernel configuration parameters.

2. The method for generating a large language model kernel according to claim 1, characterized in that, For each loop dimension, the feasible intervals for parallelism and block size are obtained based on the parallelism rule and the loop block rule. Obtaining the feasible parallelism interval for the loop dimension based on the parallelism rules includes: obtaining the total number of functional units based on the hardware constraint parameters; for each loop dimension, obtaining the number of occupied functional units based on the preset target kernel sequence; obtaining the maximum parallelism of the loop dimension based on the total number of functional units and the number of occupied functional units; obtaining the unblocked loop dimension based on the currently traversed memory level; obtaining the minimum parallelism of the loop dimension based on the unblocked loop dimension and the maximum parallelism of the loop dimension; and obtaining the feasible parallelism interval of the loop dimension at the currently traversed memory level based on the maximum parallelism and the minimum parallelism.

3. The method for generating a large language model kernel according to claim 1, characterized in that, For each loop dimension, the feasible intervals for parallelism and block size are obtained based on the parallelism rule and the loop block rule. Obtaining the feasible range of block size for the cyclic dimension based on the cyclic block rule includes: obtaining the cache capacity of the currently traversed memory level based on the hardware constraint parameters; obtaining the determined and undetermined cyclic block sizes based on the preset target kernel sequence; obtaining the matrix cyclic block product based on the determined and undetermined cyclic block sizes; determining the upper limit of the block size for the cyclic dimension based on the matrix cyclic block product and the cache capacity of the currently traversed memory level; and obtaining the feasible range of block size for the cyclic dimension based on the upper limit of the block size for the cyclic dimension.

4. The method for generating a large language model kernel according to claim 1, characterized in that, The current memory level configuration parameters are obtained based on the feasible intervals of parallelism and block size corresponding to several cyclic dimensions; including: for each cyclic dimension, inputting the feasible intervals of parallelism and block size corresponding to the cyclic dimension into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size respectively; and obtaining the current memory level configuration parameters based on the target parallelism and target block size corresponding to several cyclic dimensions.

5. The method for generating a large language model kernel according to claim 4, characterized in that, The step of inputting the feasible intervals of parallelism and block size corresponding to the cyclic dimension into a preset policy network, so that the policy network determines the target parallelism and target block size of the cyclic dimension based on the feasible intervals of parallelism and block size, involves the following steps in the preset policy network: performing feature transformation based on the feasible intervals of parallelism and block size to obtain the corresponding probability distributions of parallelism and block size; and performing probability sampling based on the probability distributions of parallelism and block size to determine the target parallelism and target block size of the cyclic dimension.

6. A method for generating a large language model kernel according to claim 4 or 5, characterized in that, The feasible intervals for parallelism and block size corresponding to the cyclic dimension are input into a preset policy network, so that the policy network determines the target parallelism and target block size for the cyclic dimension based on the feasible intervals for parallelism and block size, respectively. The policy network is trained and updated using a preset reinforcement learning algorithm. Specifically: the target large language model kernel is input into a preset performance analysis model, so that the target large language model kernel performs performance evaluation based on the target large language model kernel to obtain kernel performance evaluation results; a reward signal is obtained based on the kernel performance evaluation results; and the parameters of the preset policy network are trained and updated based on the reward signal and the preset reinforcement learning reward algorithm.

7. A large language model kernel generation system, characterized in that, include: The system includes a hardware parameter acquisition module for acquiring hardware constraint parameters of the target large language model; a resource constraint awareness rule construction module for constructing resource constraint awareness rules based on the hardware constraint parameters, including parallelism rules and cyclic block rules; and a space exploration module for performing space exploration on the target kernel based on the parallelism rules, cyclic block rules, and a preset sequence decision algorithm to generate the target large language model kernel. Specifically, the sequence decision algorithm involves: traversing several memory levels in the target kernel according to a preset hierarchical traversal order; for the currently traversed memory level, performing the following steps: traversing several cyclic dimensions in the memory level according to a preset cyclic dimension traversal order; for each cyclic dimension, obtaining the feasible intervals of parallelism and block size based on the parallelism rules and cyclic block rules; obtaining the current memory level configuration parameters based on the feasible intervals of parallelism and block size corresponding to several cyclic dimensions; repeating the above steps until all memory levels in the target kernel are traversed to obtain the target kernel configuration parameters. The kernel generation module is used to generate the target large language model kernel based on the target kernel configuration parameters.

8. The large language model kernel generation system according to claim 7, characterized in that, In the space exploration module, for each cyclic dimension, the feasible intervals of parallelism and block size for that cyclic dimension are obtained based on the parallelism rule and the cyclic block rule. Obtaining the feasible parallelism interval for the loop dimension based on the parallelism rules includes: obtaining the total number of functional units based on the hardware constraint parameters; for each loop dimension, obtaining the number of occupied functional units based on the preset target kernel sequence; obtaining the maximum parallelism of the loop dimension based on the total number of functional units and the number of occupied functional units; obtaining the unblocked loop dimension based on the currently traversed memory level; obtaining the minimum parallelism of the loop dimension based on the unblocked loop dimension and the maximum parallelism of the loop dimension; and obtaining the feasible parallelism interval of the loop dimension at the currently traversed memory level based on the maximum parallelism and the minimum parallelism.

9. A large language model kernel generation system according to claim 7, characterized in that, In the space exploration module, for each cyclic dimension, the feasible intervals of parallelism and block size for that cyclic dimension are obtained based on the parallelism rule and the cyclic block rule. Obtaining the feasible range of block size for the cyclic dimension based on the cyclic block rule includes: obtaining the cache capacity of the currently traversed memory level based on the hardware constraint parameters; obtaining the determined and undetermined cyclic block sizes based on the preset target kernel sequence; obtaining the matrix cyclic block product based on the determined and undetermined cyclic block sizes; determining the upper limit of the block size for the cyclic dimension based on the matrix cyclic block product and the cache capacity of the currently traversed memory level; and obtaining the feasible range of block size for the cyclic dimension based on the upper limit of the block size for the cyclic dimension.

10. A text generation method based on an optimized model kernel, characterized in that, This method obtains the optimized model kernel based on a large language model kernel generation method as described in claims 1-6. The text generation method includes the following steps: obtaining input text, inputting the input text into a pre-trained large language model, so that the large language model performs text inference based on the input text and the optimized model kernel to obtain a text mapping result; and generating target text based on the text mapping result.

Images