Kernel selection method and device during general matrix multiplication operation, equipment and storage medium

Abstract

The embodiment of the invention provides a kernel selection method and device during general matrix multiplication operation, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring general matrix multiplication problem size data; preprocessing the general matrix multiplication problem size data to obtain general matrix multiplication problem size features; based on a pre-trained problem size encoder, mapping the general matrix multiplication problem size feature into a problem size embedded vector of a preset dimension; taking the problem size embedded vector as a query vector, and performing nearest neighbor vector search in a pre-configured vector database to obtain a kernel configuration feature with the highest similarity; and outputting the kernel configuration feature as a selection result. The method is used for improving the kernel selection efficiency and precision during the operation of the general matrix multiplication.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a general matrix multiplication runtime kernel selection method, apparatus, device, and storage medium. Background Technology

[0002] In today's computing-driven era, General Matrix Multiply (GEMM), representing the multiplication of two matrices, is a core computational operation in high-performance computing and deep learning, and its efficiency is crucial to system performance. Whether it's large-scale scientific computing (such as weather forecasting and quantum mechanics simulations) or the training and inference of deep learning models (such as convolution and fully connected layer operations), especially large-parameter models like Large Language Models (LLMs) and multimodal models, over 80% of the computation relies on GEMM operations. Therefore, the performance of GEMM directly determines the efficiency of model iteration and application response, and has become a key technological challenge driving the development of artificial intelligence and high-performance computing.

[0003] The computational efficiency of GEMM largely depends on its matching kernel configuration parameters. These parameters define and constrain the parallel execution mode and resource allocation of a specific computational kernel executed on a GPU or accelerator card. An optimized kernel configuration can fully unleash the hardware's computing power potential, significantly reduce computation latency, and improve data throughput; while an unreasonable configuration can easily lead to low hardware resource utilization, significant waste of computing power, and even become the performance bottleneck of the entire system. Especially in the field of domestically developed accelerator cards, their independently developed hardware architecture differs from that of mainstream international accelerator cards, placing higher demands on the adaptability of kernel configurations—general configuration parameters often fail to fully activate the computing power advantages of domestically developed accelerator cards, and may even amplify performance losses due to architectural incompatibility.

[0004] Currently, there are two main approaches to kernel selection at runtime in GEMM: one is a search method based on pre-stored mapping tables and numerical distance, and the other is an online search method based on an automatic tuning framework. The first method, which relies solely on Euclidean distance to determine size similarity, is highly susceptible to mismatches and fails to select truly high-performance kernels, leading to significant performance losses. The second method utilizes optimization algorithms (such as genetic algorithms, Bayesian optimization, and reinforcement learning) to search for the optimal configuration at runtime or before deployment. However, this approach requires substantial time overhead and cannot meet the real-time response requirements of runtime.

[0005] Therefore, although existing kernel selection methods have achieved some success in most application scenarios, they still rely on manually designed numerical metrics or time-consuming search processes, making it difficult to accurately capture the complex relationship between the GEMM problem size and the optimal configuration. This results in insufficient selection efficiency and accuracy when facing diverse matrix operation scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a general matrix multiplication runtime kernel selection method, apparatus, device, and storage medium to improve the efficiency and accuracy of general matrix multiplication runtime kernel selection.

[0007] To achieve the above objectives, embodiments of the present invention provide a general matrix multiplication runtime kernel selection method, the method comprising: acquiring general matrix multiplication problem size data; preprocessing the general matrix multiplication problem size data to obtain general matrix multiplication problem size features; mapping the general matrix multiplication problem size features to a problem size embedding vector of a preset dimension based on a pre-trained problem size encoder; using the problem size embedding vector as a query vector to perform a nearest neighbor vector search in a pre-configured vector database to obtain the kernel configuration feature with the highest similarity; and outputting the kernel configuration feature as the selection result.

[0008] The method proposed in this invention maps GEMM problem sizes and kernel configurations of different modalities to vector representations of a unified dimension, thereby transforming the original numerical relationships into higher-level semantic associations, bringing semantically similar samples closer together in space. By jointly employing embedding learning and vector retrieval techniques, a precise mapping relationship between GEMM problem sizes and optimal kernel configurations is constructed, enabling high-precision, low-latency automatic selection of kernel configurations while reducing the complexity of offline mapping table construction and online query overhead. The kernel configuration feature with the highest similarity obtained through nearest neighbor vector search represents the kernel configuration with the highest computational efficiency under this general matrix multiplication problem size data.

[0009] Optionally, preprocessing the general matrix multiplication problem size data to obtain general matrix multiplication problem size features includes: extracting features from the general matrix multiplication problem size data to obtain original features, which include matrix dimension and / or batch size; and performing mathematical transformations based on the original features to obtain derived features, which include logarithmic features, matrix aspect ratio, shape factor, and / or computational intensity, wherein the general matrix multiplication problem size features include the original features and the derived features.

[0010] These derived features can capture deeper mathematical relationships between the original features, enhancing the model's understanding of the problem's essence. By integrating original and derived features, the ability to represent the problem's structure and computational characteristics can be strengthened, providing more discriminative input information for the embedded model.

[0011] Optionally, the training method for the problem size encoder includes: acquiring raw performance profiling data, which includes general matrix multiplication problem size data and kernel configuration parameter data, and there is a preset correspondence between the general matrix multiplication problem size data and the kernel configuration parameter data; preprocessing the raw performance profiling data to obtain the general matrix multiplication problem size features and kernel configuration features; and training a dual encoder joint embedding model based on the general matrix multiplication problem size features and kernel configuration features, wherein the dual encoder includes a problem size encoder and a kernel configuration encoder, the kernel configuration encoder is used to map the kernel configuration features to the kernel configuration embedding vector of the preset dimension, and the dual encoder joint embedding model training is used to maximize the cosine similarity between the general matrix multiplication problem size features and kernel configuration features that have a correspondence, and minimize the cosine similarity between the general matrix multiplication problem size features and kernel configuration features that do not have a correspondence.

[0012] During training, the problem size encoder and kernel configuration encoder are optimized based on maximizing and minimizing cosine similarity. The deep semantic relationship between the two is mined through a neural network model, which overcomes the matching deviation problem caused by different configurations due to similar numerical values. This realizes a paradigm shift from "numerical similarity" to "semantic similarity", fundamentally avoiding matching errors caused by simply relying on numerical distance.

[0013] Optionally, preprocessing the kernel configuration parameter data to obtain the kernel configuration features includes: extracting features from the kernel configuration parameter data to obtain the kernel configuration features, wherein the kernel configuration features include macroblock size, thread block size, workgroup size, vectorization width, global read vectorization width, and / or local read vectorization width in each dimension.

[0014] The kernel configuration features selected in this invention cover multi-level resource allocation from the thread level to the workgroup level, collectively influencing memory access patterns and data reuse strategies. These are core factors limiting GEMM performance, and converting them into feature vectors provides the ability to distinguish between different configuration functions and performance. Through feature engineering, not only is the integrity of the original information preserved, but the representational power of the features is also enhanced, thus laying a solid foundation for subsequent embedding learning.

[0015] Optionally, the method further includes: during the training of the dual encoder joint embedding model, in each training batch, for each question size embedding vector, taking all corresponding kernel configuration embedding vectors as positive samples and other kernel configuration embedding vectors as negative samples.

[0016] This setup makes full use of the correspondence in the data and dynamically constructs positive and negative sample pairs during training. It can effectively cope with the training complexity brought about by the "many-to-many" mapping relationship, improve the discriminative ability and generalization performance of the embedding representation, and significantly improve training efficiency and data utilization.

[0017] Optionally, the method further includes: during the training of the dual-encoder joint embedding model, optimizing the problem size encoder and kernel configuration encoder using an improved multi-label InfoNCE loss function, wherein the improved multi-label InfoNCE loss function is:

[0018] in, This represents the total number of samples in the current batch. Indicates the first The set of all positive samples of a sample The cosine similarity function is used. For temperature parameters, Indicates the first One sample.

[0019] The loss function described above introduces an adjustable temperature parameter τ to control the sharpness of the similarity distribution. This design allows the model to simultaneously optimize the similarity of multiple positive sample pairs, encouraging positive pairs to cluster in the embedding space while negative pairs move away from each other. The introduction of the temperature parameter τ enables the model to adaptively adjust its focus on difficult negative samples (i.e., negative samples with high similarity to positive samples), thereby learning more discriminative embedding representations.

[0020] Optionally, the configuration method of the vector database includes: generating kernel configuration embedding vectors corresponding to all kernel configuration features based on the kernel configuration encoder; constructing a vector index based on the generated kernel configuration embedding vectors and FAISS to obtain the vector database.

[0021] This index combines quantization and spatial partitioning techniques, which can significantly improve the efficiency of nearest neighbor search.

[0022] Optionally, the method further includes: normalizing the general matrix multiplication problem size features and kernel configuration features, wherein the normalized data is used as input to the problem size encoder and the kernel configuration encoder.

[0023] Normalization preprocessing is performed on all features used for training, mapping each feature to the interval [0,1], thereby eliminating the bias introduced by the differences in numerical magnitude and distribution pattern between feature dimensions, and improving the numerical stability and convergence efficiency of the embedding learning process.

[0024] Secondly, embodiments of the present invention provide a kernel selection device for a general matrix multiplication runtime. The device includes: an acquisition module for acquiring general matrix multiplication problem size data; a preprocessing module for preprocessing the general matrix multiplication problem size data to obtain general matrix multiplication problem size features; an encoding module for mapping the general matrix multiplication problem size features to a problem size embedding vector of a preset dimension based on a pre-trained problem size encoder; a search module for using the problem size embedding vector as a query vector to perform a nearest neighbor vector search in a pre-configured vector database to obtain the kernel configuration feature with the highest similarity; and an output module for outputting the kernel configuration feature as the selection result.

[0025] Thirdly, embodiments of the present invention provide a device for selecting a runtime kernel for general matrix multiplication, the device comprising a memory and a processor, the processor being configured to run a program, wherein the program is executed to perform any of the methods described herein.

[0026] Fourthly, embodiments of the present invention provide a machine-readable storage medium storing instructions that cause a machine to perform any of the methods described herein.

[0027] The above technical solution maps the GEMM problem size and kernel configuration to a unified-dimensional vector representation, transforming the original numerical relationships into high-level semantic associations. Based on this precise mapping relationship, and combined with vector retrieval technology, nearest neighbor vector searches are performed in a pre-configured vector database. This enables high-precision, low-latency automatic selection of kernel configuration features, while reducing the complexity of building offline mapping tables and the overhead of online queries.

[0028] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0029] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the general matrix multiplication runtime kernel selection method provided in this embodiment of the invention; Figure 2This is a flowchart illustrating another general matrix multiplication runtime kernel selection method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the joint embedding space visualization results based on t-SNE technology provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the general matrix multiplication runtime kernel selection device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the device provided in an embodiment of the present invention.

[0030] Explanation of reference numerals in the attached figures 101 Processor 102 Memory 103 bus 10 devices Detailed Implementation The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0031] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0032] Figure 1 This is a flowchart illustrating the general matrix multiplication runtime kernel selection method provided in this embodiment of the invention, as shown below. Figure 1 As shown, the method includes steps S1 to S5.

[0033] Step S1: Obtain the size data for the general matrix multiplication problem.

[0034] Step S2: Preprocess the general matrix multiplication problem size data to obtain the general matrix multiplication problem size characteristics, which are used to describe the scale characteristics of the matrix multiplication problem to be solved.

[0035] Step S3: Based on the pre-trained problem size encoder, map the general matrix multiplication problem size features into a problem size embedding vector of a preset dimension.

[0036] Step S4: Use the problem size embedding vector as the query vector, and perform a nearest neighbor vector search in the pre-configured vector database to obtain the kernel configuration feature with the highest similarity.

[0037] Step S5: Output the kernel configuration features as the selection result.

[0038] For ease of description, the “General Matrix Multiplication Problem Size Data” will be abbreviated as “GEMM Problem Size Data” and the “General Matrix Multiplication Problem Size Features” will be abbreviated as “GEMM Problem Size Features”.

[0039] Runtime kernel selection refers to the process of dynamically selecting and calling the most suitable kernel from a pre-prepared set of optimized kernels while the program is executing, based on the current computational task (such as matrix size). For the hardware characteristics of domestically produced accelerator cards, and considering different GEMM problem sizes, achieving efficient and accurate optimal kernel configuration selection is not only a core issue in improving GEMM computation efficiency, but also a crucial step in unleashing the computing potential of domestically produced accelerator cards and promoting the construction of an independent computing ecosystem.

[0040] To address the issue of large kernel selection bias caused by traditional methods relying on numerical distance, this invention proposes a joint embedding space construction method. This method maps GEMM problem sizes and kernel configurations of different modalities to vector representations of a unified dimension, thereby transforming the original numerical relationships into higher-level semantic associations, bringing semantically similar samples closer together in the space. Through joint embedding learning and vector retrieval techniques, a precise mapping relationship between GEMM problem sizes and optimal kernel configurations is constructed, enabling high-precision, low-latency automatic selection of kernel configurations while reducing the complexity of offline mapping table construction and online query overhead. The kernel configuration feature with the highest similarity obtained through nearest neighbor vector search represents the kernel configuration with the highest computational efficiency under this general matrix multiplication problem size data.

[0041] The GEMM problem size data obtained in step S1 is an unprocessed raw string containing many parameters that cannot be used directly. Therefore, in step S2, this string is preprocessed to obtain the required GEMM problem size features. Figure 2 This is a flowchart illustrating another general matrix multiplication runtime kernel selection method provided in this embodiment of the invention, as shown below. Figure 2 As shown, the GEMM problem size data is preprocessed to obtain GEMM problem size features, including feature engineering. This process includes two stages: the first stage is feature extraction, and the second stage is mathematical transformation.

[0042] In the first stage, feature extraction is performed on the GEMM problem size data to obtain original features, which include matrix dimensions (M, N, and K) and / or batch size B. In the second stage, mathematical transformations are performed on the original features to obtain derived features, which include logarithmic features, matrix aspect ratio, shape factor, and / or computational intensity. The GEMM problem size features include both the original features and the derived features.

[0043] Specifically, M represents the number of rows in matrix A and matrix C, N represents the number of columns in matrix B and matrix C, and K represents the number of columns in matrix A and the number of rows in matrix B.

[0044] Derived features are enhanced features obtained by performing mathematical transformations on the original features. Specifically: Logarithmic features include log(M+1), log(N+1), and log(K+1), which are used to smooth large numerical spans and non-uniform distributions, thereby improving the stability of model training. The matrix aspect ratio is M / N, which reflects the matrix structure and affects thread organization and memory access patterns. The shape factor is ratio / K, which comprehensively represents the overall shape characteristics of the matrix and provides richer geometric structure information for the model. Compute intensity measures the ratio of computation operations to memory access operations and is a key metric for evaluating kernel performance bottlenecks. It is calculated as follows: .

[0045] These derived features can capture deeper mathematical relationships between the original features, enhancing the model's understanding of the problem's essence. By integrating original and derived features, the ability to represent the problem's structure and computational characteristics can be strengthened, providing more discriminative input information for the embedded model.

[0046] This invention employs a contrastive learning-based training framework to effectively learn semantic relationships and adapt to many-to-many mappings in the training data. Specifically, the training method for the problem size encoder includes: acquiring raw performance profiling data, which includes GEMM problem size data and kernel configuration parameter data, and a preset correspondence exists between the GEMM problem size data and the kernel configuration parameter data; preprocessing the raw performance profiling data to obtain the GEMM problem size features and kernel configuration features; and training a dual-encoder joint embedding model based on the GEMM problem size features and kernel configuration features.

[0047] The dual encoders include a problem size encoder and a kernel configuration encoder. The kernel configuration encoder is used to map the kernel configuration features to kernel configuration embedding vectors of the preset dimension. The dual encoder joint embedding model training is used to maximize the cosine similarity between the GEMM problem size features and kernel configuration features that have a corresponding relationship, and minimize the cosine similarity between the GEMM problem size features and kernel configuration features that do not have a corresponding relationship.

[0048] In practical implementation, the problem-size encoder can employ a multilayer perceptron (MLP). This structure captures the complex interactions between parameters through multilayer nonlinear transformations, converting numerical features into semantically rich vector representations. Correspondingly, the kernel configuration encoder can adopt an MLP structure symmetrical to the problem-size encoder, mapping configuration parameters to embedding vectors of the same dimension as the problem-size embedding vectors, and clustering configurations with similar functions or performance in the embedding space. During training, the problem-size encoder and kernel configuration encoder are optimized based on maximizing and minimizing cosine similarity. A neural network model is used to mine the deep semantic connections between them, overcoming the matching bias caused by dissimilar configurations despite similar numerical values. This achieves a paradigm shift from "numerical similarity" to "semantic similarity," fundamentally avoiding matching errors caused by relying solely on numerical distance.

[0049] Furthermore, preprocessing the kernel configuration parameter data to obtain the kernel configuration features includes: performing feature engineering on the kernel configuration parameter data, including feature extraction, to obtain the kernel configuration features.

[0050] Kernel configuration features describe the specific parameter settings of the GPU kernel, which directly determine how computing resources are organized and utilized. This invention selects the following key configuration parameters as kernel configuration features: macroblock size, thread block size, workgroup size, vectorization width, global read vectorization width, and / or local read vectorization width in various dimensions. Specifically: Macroblock size MT (MacroTile) in various dimensions, which determines the size of the basic computational data block; The thread block size TT (ThreadTile) is a feature that controls the number of elements processed by each thread. WorkGroup size (WG) is a feature that affects the organization of threads. Vector Width (VW) is a feature that relates to memory access efficiency and computational throughput. Global Read Vector Width (GRVW) optimizes global memory access. Local Read Vector Width (LRVW) optimizes local memory access.

[0051] The GEMM problem size features and kernel configuration features selected in this invention cover multi-level resource allocation from the thread level to the workgroup level. These features collectively influence memory access patterns and data reuse strategies, and are core factors limiting GEMM performance. Converting them into feature vectors provides the ability to distinguish between different configuration functions and performance. Through feature engineering, not only is the integrity of the original information preserved, but the representational power of the features is also enhanced, thus laying a solid foundation for subsequent embedding learning.

[0052] Furthermore, the method also includes: normalizing the GEMM problem size features and kernel configuration features, wherein the normalized data is used as input to the problem size encoder and the kernel configuration encoder. During the training of the dual-encoder joint embedding model, all features used for training are pre-processed with normalization, mapping each feature to the interval [0,1]. This eliminates the bias introduced by differences in numerical magnitude and distribution between feature dimensions, improving the numerical stability and convergence efficiency of the embedding learning process.

[0053] In the training data, there is a pre-defined correspondence between GEMM problem size data and kernel configuration parameter data. In reality, this correspondence can be one-to-one, many-to-one, or many-to-many. Therefore, if a fixed negative sample set is pre-constructed during training using traditional methods, a high computational cost is required. To address this issue, this invention designs an intra-batch multi-label negative sampling mechanism. During the training of the dual-encoder joint embedding model, in each training batch, for each problem size embedding vector, all corresponding kernel configuration embedding vectors are treated as positive samples, and the remaining kernel configuration embedding vectors are treated as negative samples. This mechanism fully utilizes the correspondence in the data, dynamically constructing positive and negative sample pairs during training. This effectively addresses the training complexity caused by "many-to-many" mapping relationships, improves the discriminative power and generalization performance of the embedding representation, and significantly enhances training efficiency and data utilization.

[0054] Furthermore, the method also includes: during the training of the dual-encoder joint embedding model, optimizing the problem size encoder and kernel configuration encoder using an improved multi-label InfoNCE loss function, wherein the improved multi-label InfoNCE loss function is:

[0055] in, This represents the total number of samples in the current batch. Indicates the first The set of all positive samples of a sample The cosine similarity function is used. For temperature parameters, Indicates the first One sample.

[0056] The loss function described above introduces an adjustable temperature parameter τ to control the sharpness of the similarity distribution. This design allows the model to simultaneously optimize the similarity of multiple positive sample pairs, encouraging positive pairs to cluster in the embedding space while negative pairs move away from each other. The introduction of the temperature parameter τ enables the model to adaptively adjust its focus on difficult negative samples (i.e., negative samples with high similarity to positive samples), thereby learning more discriminative embedding representations.

[0057] During training, based on the calculated loss function, the parameters of both the problem-size encoder and the kernel configuration encoder are simultaneously optimized using the backpropagation algorithm. This ensures that the embedding vectors of the two modalities are aligned in a common space, achieving end-to-end joint learning. Furthermore, this invention employs the Adam optimizer and a dynamic learning rate strategy during training, thereby improving training stability while maintaining convergence speed.

[0058] To verify the embedding quality, this invention utilizes t-SNE technology to perform dimensionality reduction visualization of high-dimensional embeddings. The results show that semantically similar sizes and configurations are clustered in the embedding space, intuitively verifying the effectiveness of this method. Figure 3 This is a schematic diagram of the joint embedding space visualization results based on t-SNE technology provided in an embodiment of the present invention. In this diagram, 1000 sample data were randomly sampled. Blue dots represent the problem-size embedding vectors output by the problem-size encoder, orange dots represent the kernel configuration embedding vectors output by the kernel configuration encoder, and gray connecting lines show the semantic relationships between positive sample pairs (only some relationships are shown in the diagram). Figure 3 It can be seen that semantically similar sizes and configurations exhibit clustered distribution in the embedding space, which intuitively verifies that the joint embedding learning model can effectively capture the complex nonlinear relationship between problem size and optimal kernel configuration, realizing a paradigm shift from numerical matching to semantic association matching.

[0059] To meet the low-latency retrieval requirements of GEMM runtime, this invention constructs a configuration vector database during the offline training phase and designs an efficient query mechanism. Specifically, in the offline phase, based on the kernel configuration encoder, kernel configuration embedding vectors corresponding to all kernel configuration features are generated; a vector index is constructed based on the generated kernel configuration embedding vectors and FAISS (Facebook AI Similarity Search) to obtain the vector database. This index, combining quantization and spatial segmentation techniques, can significantly improve the efficiency of nearest neighbor search.

[0060] Furthermore, when constructing the vector database, all embedded vectors are L2 normalized. The inner product operation is then equivalent to cosine similarity calculation, thus improving retrieval speed while maintaining accuracy. Correspondingly, during the online inference stage, the query vector is also L2 normalized, and a cosine similarity nearest neighbor search is performed in the vector database, returning the configuration parameter with the highest similarity as output. Through this offline vector database construction and online similarity search inference mechanism, a configuration embedding index can be built using efficient vector indexing technology, achieving millisecond-level optimal configuration retrieval and meeting real-time kernel selection requirements.

[0061] To verify the effectiveness of the method proposed in this invention, it is compared with traditional table lookup methods based on numerical distance. The comparison results are shown in Table 1, where the method of this invention is labeled as emb+cos, and the traditional Euclidean distance table lookup method is labeled as euclidean. To evaluate prediction accuracy, the efficiency ratio is used as the core evaluation metric, defined as the ratio of the performance of the retrieved configuration (Perf_look-up) to the performance of the true optimal configuration (Perf_true), with a standard value of 1.0. In addition, the average search time (Avg_searchTime) is used as another evaluation metric.

[0062] Table 1. Comparison results between the present invention and the traditional Euclidean distance lookup method

[0063] The above comparison results show that the method proposed in this invention is significantly better than the traditional table lookup method based on numerical distance in terms of both prediction accuracy and inference efficiency.

[0064] In terms of prediction accuracy, the method of this invention achieves an average efficiency ratio of 0.9914 on the test set, significantly outperforming the traditional Euclidean distance lookup table method's 0.9586, with an accuracy improvement of approximately 3.42%, bringing it closer to the true optimal performance. In tests with different matrix sizes (distinguished by the value of K), the method of this invention demonstrates better adaptability and stability. In small-to-medium-sized matrix scenarios (K=64, 128, 256), the efficiency ratio of the method of this invention is consistently higher than 1.0, indicating that its selected configuration is even slightly better than the labeled optimal configuration. In large-scale matrix scenarios (K=512, 768, 1024), the method of this invention still maintains a significant advantage. As the matrix size increases, the efficiency ratio of both methods decreases, but the decrease in the method of this invention is smaller, demonstrating better robustness and generalization ability.

[0065] In terms of query time, this invention demonstrates an order-of-magnitude efficiency advantage. Test results show that the average search time of the method proposed in this invention is 0.0064 milliseconds per sample, while the average search time of the traditional Euclidean distance lookup table method is 3.6808 milliseconds per sample, representing a hundreds-fold improvement in search efficiency. This comparative result proves that the method proposed in this invention can better meet the needs of real-time inference scenarios and provide technical support for instantaneous kernel selection in deep learning frameworks.

[0066] Therefore, the method proposed in this invention achieves a paradigm shift from numerical matching to semantic matching through embedding learning technology. While ensuring extremely high inference efficiency, it significantly improves the accuracy of kernel selection, providing a practical technical solution for the efficient and automated optimization of GEMM operations.

[0067] Example 1: Please refer to Figure 2 The following is a complete explanation of the offline training phase and the online inference phase.

[0068] The offline training phase includes the following steps: Step S201: Obtain raw performance profiling data, including GEMM problem size data and kernel configuration parameter data, and there is a corresponding relationship between the two types of data.

[0069] Step S202: Perform feature engineering on the above two types of data to obtain the size features and kernel configuration features of the general matrix multiplication problem.

[0070] Step S203: Train the dual encoder joint embedding model based on the problem size features and kernel configuration features of the general matrix multiplication problem to obtain the trained problem size encoder and kernel configuration encoder.

[0071] Step S204: Based on the trained kernel configuration encoder, generate kernel configuration embedding vectors corresponding to all kernel configuration features, and construct a vector index based on the generated kernel configuration embedding vectors and FAISS to obtain a vector database.

[0072] The online reasoning phase includes the following steps: Step S301: Obtain GEMM problem size data.

[0073] Step S302: Perform feature engineering on the GEMM problem size data. This process is the same as in step S202, and the GEMM problem size features are obtained.

[0074] Step S303: Based on the problem size encoder trained in step S203, map the GEMM problem size features into a problem size embedding vector of a preset dimension.

[0075] Step S304: Use the problem size embedding vector as the query vector, and perform a nearest neighbor vector search in the vector database configured in step S204 to obtain the kernel configuration feature with the highest similarity.

[0076] Step S305: Output the kernel configuration features as the selection result.

[0077] Figure 4 This is a schematic diagram of the structure of the general matrix multiplication runtime kernel selection device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes: an acquisition module for acquiring general matrix multiplication problem size data; a preprocessing module for preprocessing the general matrix multiplication problem size data to obtain general matrix multiplication problem size features; an encoding module for mapping the general matrix multiplication problem size features to problem size embedding vectors of a preset dimension based on a pre-trained problem size encoder; a search module for using the problem size embedding vectors as query vectors to perform nearest neighbor vector search in a pre-configured vector database to obtain the kernel configuration feature with the highest similarity; and an output module for outputting the kernel configuration feature as the selection result.

[0078] The specific working principle and benefits of the general matrix multiplication runtime kernel selection device provided in this embodiment of the invention are similar to those of the general matrix multiplication runtime kernel selection method provided in this embodiment of the invention, and will not be repeated here.

[0079] This invention also provides a device for runtime kernel selection in general matrix multiplication, for example, such as... Figure 5 As shown, the device includes a memory and a processor, the processor being used to run a program, wherein the program is executed to perform the aforementioned general matrix multiplication runtime kernel selection method.

[0080] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and the kernel selection method for general matrix multiplication runtime can be implemented by adjusting kernel parameters.

[0081] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0082] This invention also provides a machine-readable storage medium storing instructions that cause a machine to execute the aforementioned general matrix multiplication runtime kernel selection method.

[0083] This invention also provides a computer program product that, when executed on a data processing device, is suitable for performing the above-described general matrix multiplication runtime kernel selection method for initialization.

[0084] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0089] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0090] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0091] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0092] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A general matrix multiplication runtime kernel selection method, characterized in that, The method includes: Obtain the size data for the general matrix multiplication problem; The size data of the general matrix multiplication problem is preprocessed to obtain the size features of the general matrix multiplication problem; Based on a pre-trained problem size encoder, the general matrix multiplication problem size features are mapped to a problem size embedding vector of a preset dimension; Using the question size embedding vector as the query vector, a nearest neighbor vector search is performed in a pre-configured vector database to obtain the kernel configuration feature with the highest similarity; and The kernel configuration features are output as the selection result.

2. The method according to claim 1, characterized in that, Preprocessing the general matrix multiplication problem size data yields the following general matrix multiplication problem size features: Feature extraction is performed on the size data of the general matrix multiplication problem to obtain the original features, which include the matrix dimension and / or batch size; and Based on the original features, mathematical transformations are performed to obtain derived features, which include logarithmic features, matrix aspect ratio, shape factor, and / or computational strength. The size features of the general matrix multiplication problem include the original features and the derived features.

3. The method according to claim 1, characterized in that, The training method for the problem size encoder includes: Obtain raw performance profile data, which includes general matrix multiplication problem size data and kernel configuration parameter data, and there is a preset correspondence between the general matrix multiplication problem size data and the kernel configuration parameter data; The raw performance profiling data is preprocessed to obtain the size characteristics and kernel configuration characteristics of the general matrix multiplication problem; and Based on the size features and kernel configuration features of the general matrix multiplication problem, a dual-encoder joint embedding model is trained. The dual encoder includes a problem size encoder and a kernel configuration encoder. The kernel configuration encoder is used to map the kernel configuration features to kernel configuration embedding vectors of a preset dimension. The dual encoder joint embedding model training is used to maximize the cosine similarity between the general matrix multiplication problem size features and kernel configuration features that have a correspondence, and minimize the cosine similarity between the general matrix multiplication problem size features and kernel configuration features that do not have a correspondence.

4. The method according to claim 3, characterized in that, The kernel configuration parameter data is preprocessed to obtain the kernel configuration features, including: Feature extraction is performed on the kernel configuration parameter data to obtain the kernel configuration features. The kernel configuration features include macroblock size, thread block size, workgroup size, vectorization width, global read vectorization width, and / or local read vectorization width in various dimensions.

5. The method according to claim 3, characterized in that, The method further includes: During the training of the dual encoder joint embedding model, in each training batch, for each question-size embedding vector, all corresponding kernel configuration embedding vectors are taken as positive samples, and the other kernel configuration embedding vectors are taken as negative samples.

6. The method according to claim 3, characterized in that, The method further includes: When training the dual-encoder joint embedding model, an improved multi-label InfoNCE loss function is used to optimize the problem size encoder and the kernel configuration encoder. The improved multi-label InfoNCE loss function is as follows: in, This represents the total number of samples in the current batch. Indicates the first The set of all positive samples of a sample The cosine similarity function is used. For temperature parameters, Indicates the first One sample.

7. The method according to claim 3, characterized in that, The configuration method for the vector database includes: Based on the kernel configuration encoder, generate kernel configuration embedding vectors corresponding to all kernel configuration features; Based on the generated kernel configuration embedding vector and FAISS, a vector index is constructed to obtain the vector database.

8. The method according to claim 3, characterized in that, The method further includes: The problem size features and kernel configuration features of the general matrix multiplication problem are normalized, and the normalized data is used as input to the problem size encoder and the kernel configuration encoder.

9. A general matrix multiplication runtime kernel selection device, characterized in that, The device includes: The acquisition module is used to obtain the size data for general matrix multiplication problems; The preprocessing module is used to preprocess the general matrix multiplication problem size data to obtain the general matrix multiplication problem size features; The encoding module is used to map the general matrix multiplication problem size features into a problem size embedding vector of a preset dimension based on a pre-trained problem size encoder; The search module is used to take the question size embedding vector as the query vector and perform a nearest neighbor vector search in a pre-configured vector database to obtain the kernel configuration feature with the highest similarity; and The output module is used to output the kernel configuration features as the selection result.

10. A device for runtime kernel selection in general matrix multiplication, the device comprising memory and a processor, characterized in that, The processor is used to run a program, wherein the program is run to perform: the method as described in any one of claims 1-8.

11. A machine-readable storage medium having instructions stored thereon for causing a machine to perform the method as described in any one of claims 1-8.

Images