Matrix multiplication optimization method and system for matrix acceleration unit

Abstract

The invention discloses a matrix multiplication optimization method and system for a matrix acceleration unit, and the method comprises the steps: jointly constructing a microkernel generation frame through a matrix multiplication instruction set provided by the matrix acceleration unit by adopting a thread rearrangement and address mapping strategy and a multi-stage pipeline mechanism, and generating a candidate microkernel set; feature extraction is carried out on an input matrix, a search space formed by a dynamic block and a scheduling strategy is constructed according to the features of the input matrix, an optimal microkernel is dynamically selected, and the scheduling strategy is dynamically adjusted; and constructing a performance prediction model, performing modeling and pruning on the candidate microkernel set, and preferentially selecting an optimal scheduling strategy combination in a search space to realize optimization of matrix multiplication oriented to the matrix acceleration unit. According to the method, the automation degree and adaptability of matrix multiplication optimization are remarkably improved, and the method is a universal high-performance optimization scheme which can be applied to various matrix multiplication architectures and supports a dynamic input scene.

Description

Technical Field

[0001] The present application relates to matrix multiplication acceleration technology, and in particular to a matrix multiplication optimization method and system for a matrix acceleration unit. Background Art

[0002] Matrix multiplication is a fundamental operation in computer science and engineering, widely used in scientific computing, machine learning, image processing, and other fields. With the advancement of hardware technology, dedicated matrix acceleration units (such as GPUs and Tensor Processing Units) have emerged to improve the computational performance and efficiency of matrix operations.

[0003] Traditional matrix multiplication optimization methods are primarily based on static analysis, optimizing for specific hardware platforms and matrix shapes at compile time. These methods often rely on static knowledge of the hardware architecture and matrix size, lacking adaptability to dynamic computational demands. However, in real-world applications, matrix shapes and hardware environments often change dynamically, and traditional static optimization methods often fail to provide optimal computational strategies, resulting in underutilized performance.

[0004] Existing matrix multiplication optimization methods primarily target regularly shaped matrices. However, convolution operations in deep learning and large models constantly change matrix size, leading to irregular matrices. The input matrices involved in convolution operations are typically large and have a non-fixed shape, while the size of the matrices in autoregressive models also changes with each new output. This makes traditional matrix multiplication optimization methods perform poorly when dealing with irregular matrices. Consequently, research on optimization for irregular matrices remains relatively limited.

[0005] In addition, with the development of big data and artificial intelligence, the scale and complexity of computing tasks continue to increase.

[0006] How to dynamically adjust computing strategies based on different hardware environments and matrix shapes on heterogeneous computing platforms to improve computing efficiency has become a technical challenge that needs to be solved urgently.

[0007] Currently, most optimization methods for matrix multiplication use traditional static optimization techniques. These methods are typically optimized for specific hardware and matrix shapes during the compilation phase, lacking adaptability to dynamic computing requirements. Existing technical solutions have significant limitations, especially when processing irregularly shaped matrices. The main problems and shortcomings are as follows:

[0008] 1) Limitations of static optimization methods. Most traditional matrix multiplication optimization methods are based on static optimization, which relies on the fixed shape of the matrix and hardware characteristics. Although static optimization can provide high computational efficiency for specific matrices and hardware configurations, once the matrix shape changes or the hardware environment changes, this static optimization method cannot adapt to the new computational requirements, resulting in underutilized performance. Especially in scenarios where the shape of the input matrix changes dynamically, existing optimization methods cannot adjust the computational strategy in real time, causing performance bottlenecks.

[0009] 2) Diversity issues of matrix multiplication instructions. Currently, different hardware architectures support multiple matrix multiplication instruction sets, which have significant differences in the size and data layout of matrix multiplication. For example, modern GPUs and TPUs support matrix multiplication units of different sizes (such as 16×16, 32×32, etc.), and in the supported instruction sets, different architectures may have different data layout methods (such as row-major order, column-major order, or other specific data formats). These differences make it difficult for existing technologies to perform dynamic optimization according to hardware characteristics when dealing with different hardware platforms. For example, some platforms may use mixed precision (such as FP16 and FP32) to accelerate calculations, while other platforms may not support this precision. Existing optimization methods often cannot adapt to the diversity of these matrix multiplication instructions and cannot dynamically adjust the computational strategy according to the instruction set and data layout of the hardware, resulting in the inability to fully utilize the efficiency of matrix multiplication calculations on various hardware platforms.

[0010] 3) Lack of dynamic adaptation to the characteristics of irregular matrices. Existing matrix multiplication optimization methods usually adopt fixed loop unrolling and parallel computing strategies, which can be relatively effective for matrices with regular shapes, but for irregular matrices, the processing ability of existing technologies is greatly reduced. When dealing with irregular matrices, the dimensions, shape, and data distribution of the matrix have an important impact on the computational process. Existing methods usually ignore these characteristics of the matrix and fail to dynamically adjust the loop unrolling method, parallel computing strategy, and memory access pattern according to the characteristics of the irregular matrix, resulting in underutilized computing resources and suboptimal performance.

[0011] Based on the above analysis, there is an urgent need to research and develop an adaptive tuning framework that achieves high performance in matrix-matrix multiplication of irregular matrices by implementing a Cache-based optimal blocking algorithm and pre-blocking at runtime. Summary of the Invention

[0012] To solve the problems existing in the above-mentioned prior art, the present invention proposes a dynamic shape adaptive matrix multiplication optimization method and system for a matrix acceleration unit, achieving high performance in matrix-matrix multiplication of irregular matrices.

[0013] In a first aspect, an embodiment of the present application provides a matrix multiplication optimization method for a matrix acceleration unit. The method includes:

[0014] Static pre-optimization step: Through the matrix multiplication instruction set provided by the matrix acceleration unit, a micro-kernel generation framework is jointly constructed by adopting thread rearrangement, address mapping strategy and multi-level pipeline mechanism to generate a set of candidate micro-kernels;

[0015] Dynamic block reorganization step: Feature extraction is performed on the input matrix, and a search space composed of a dynamic block and scheduling strategy is constructed according to the input matrix features. In the search space, the optimal micro-kernel is dynamically selected and the scheduling strategy is dynamically adjusted according to the input matrix features and the pre-generated set of candidate micro-kernels;

[0016] Performance modeling step: Based on the features of the input matrix and the hardware features, a performance prediction model is constructed to model and prune the set of candidate micro-kernels, and the optimal scheduling strategy combination is preferentially selected in the search space to achieve the optimization of matrix multiplication for the matrix acceleration unit.

[0017] In an embodiment of the present invention, the above matrix multiplication optimization method for a matrix acceleration unit further includes:

[0018] Automatic update step: Automatically switch the scheduling strategy during runtime, and optimize and update the scheduling strategy in real time according to the dynamic changes of the input matrix features or the floating of platform resources.

[0019] In an embodiment of the present invention, the above static pre-optimization step includes:

[0020] Collect the features of the target hardware platform and construct a hardware feature library. The hardware feature library includes: matrix multiplication instruction set, number of parallel processing units, shared memory size, single instruction multiple data width;

[0021] Fuse the matrix multiplication instruction set provided by the hardware and perform instruction scheduling. During this process, call the matrix instruction code generator written according to the hardware platform to generate corresponding computation kernels based on different computation block sizes set by the static pre-optimization.

[0022] Adopt thread rearrangement and address mapping strategy to check whether there are bank conflicts in the reads and writes related to the shared memory. If there are conflicts, perform address remapping to make the memory access addressing of the threads staggered to avoid conflicts. Among them, the address mapping strategy is: divide the shared memory into square blocks, perform an exclusive OR operation on the column coordinates and row coordinates within the square block to obtain new column or row coordinates, while the other coordinate remains unchanged.

[0023] Adopt a multi - level pipeline mechanism, use a pre - written multi - level pipeline code template to generate a set of candidate micro - kernels. In the multi - level pipeline code template, through initiating multi - level asynchronous memory access, the calculation is masked by the asynchronous memory access, and the data required for synchronous calculation is waited for by instructions.

[0024] In the embodiments of the present invention, the above - mentioned dynamic block recombination step includes:

[0025] Generate combinations of various block sizes (such as 16x16, 32x32, or 64x64) in the search space, and allocate different processing threads for parallel calculation for each block method. In order to avoid an excessive number of enumerated block methods, the block strategy pre - formulates a small number of coarse - grained block templates, obtains coarse - grained blocks through mapping of the coarse - grained block templates, and further divides each block in the coarse - grained blocks into equal - sized sub - blocks on this basis.

[0026] Loop scheduling optimization: To avoid the problem of low utilization of computing resources in the case where some dimensions in the input are too small for the block scheduling oriented to the output shape, consider block - dividing and scheduling the batch processing or K dimension, and add corresponding scheduling strategies in the search space.

[0027] In the embodiments of the present invention, the above - mentioned performance modeling step includes:

[0028] Pre - train a performance prediction model based on input features, hardware features, and historical execution statistical information;

[0029] Based on the performance prediction model, preferentially select scheduling strategies with higher expected performance in the search space.

[0030] In a second aspect, the embodiments of the present invention provide a matrix multiplication optimization system for a matrix acceleration unit, which adopts the matrix multiplication optimization method for a matrix acceleration unit as described above. The system includes:

[0031] Static pre - optimization module: Through the matrix multiplication instruction set provided by the matrix acceleration unit, jointly construct a micro - kernel generation framework by using thread rearrangement, address mapping strategy, and multi - level pipeline mechanism to generate a set of candidate micro - kernels;

[0032] Dynamic block recombination module: Extract features for the input matrix, construct a search space composed of dynamic block and scheduling strategies according to the input matrix features. In the search space, dynamically select the optimal micro - kernel and dynamically adjust the scheduling strategy according to the input matrix features and the pre - generated set of candidate micro - kernels;

[0033] Performance modeling module: Based on the features of the input matrix and hardware features, construct a performance prediction model, model and prune the set of candidate micro - kernels, and preferentially select the optimal scheduling strategy combination in the search space to achieve the optimization of matrix multiplication for the matrix acceleration unit.

[0034] Automatic update module: Automatically switch the scheduling policy during runtime, and optimize and update the scheduling policy in real-time according to the dynamic changes of the input matrix features or the floating of platform resources.

[0035] In the embodiments of the present invention, the above-mentioned static pre-optimization module includes:

[0036] Collect the features of the target hardware platform and construct a hardware feature library, where the hardware feature library includes: matrix multiplication instruction set, number of parallel processing units, shared memory size, single instruction multiple data width;

[0037] Fuse the matrix multiplication instruction set provided by the hardware for instruction scheduling. During this process, call the matrix instruction code generator written according to the hardware platform to generate corresponding computing kernels based on different computing block sizes set by static pre-optimization.

[0038] Adopt a thread rearrangement and address mapping strategy to check whether there are bank conflicts in the reads and writes related to the shared memory. If there are conflicts, perform address remapping to make the memory access addressing of the threads staggered to avoid conflicts. Among them, the address mapping strategy is: divide the shared memory into square blocks, perform an exclusive OR operation on the column coordinates and row coordinates within the square block to obtain new column or row coordinates, while the other coordinate remains unchanged.

[0039] Adopt a multi-level pipeline mechanism and use a pre-written multi-level pipeline code template to generate a set of candidate micro-kernels. Among them, in the multi-level pipeline code template, by initiating multi-level asynchronous memory access, the calculation is masked by asynchronous memory access, and the data required for synchronous calculation is waited for by instructions.

[0040] In the embodiments of the present invention, the above-mentioned dynamic block reorganization module includes:

[0041] Generate combinations of various block sizes (such as 16x16, 32x32, or 64x64) in the search space, and allocate different processing threads for parallel calculation for each block method. In order to avoid too many enumerated block methods, the block strategy pre-determines a small number of coarse-grained block templates, obtains coarse-grained blocks through mapping of the coarse-grained block templates, and then divides each block in the coarse-grained blocks into equal-sized sub-blocks on this basis.

[0042] Loop scheduling optimization. To avoid the problem of low computing resource utilization in the case of too small certain dimensions in the input for the block scheduling facing the output shape, consider performing block division and scheduling on the batch processing or K dimension, and add corresponding scheduling strategies in the search space.

[0043] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the matrix multiplication optimization method for a matrix acceleration unit are implemented.

[0044] In a fourth aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the matrix multiplication optimization method for a matrix acceleration unit are implemented.

[0045] Compared with the related prior art, it has the following outstanding beneficial effects:

[0046] 1) The method of the present invention proposes a high-performance micro-kernel generation framework: In the static optimization stage, by constructing a high-performance micro-kernel generation framework and combining the hardware structure and algorithm characteristics, a highly adaptable kernel configuration is generated. The core optimizations include three aspects: First, integrating the matrix multiplication instruction set provided by the hardware to achieve efficient instruction scheduling; second, avoiding bank conflicts in shared memory and improving memory access efficiency through thread rearrangement and address mapping strategies; third, introducing a multi-level pipeline mechanism to promote the parallel progress of computing and memory access, maximizing the masking of memory access latency and improving throughput performance.

[0047] 2) The method of the present invention proposes a micro-kernel reorganization technology: As a key mechanism in the dynamic block reorganization stage, the micro-kernel reorganization technology can dynamically select the optimal micro-kernel and adjust its execution strategy according to the real-time shape of the input matrix and the hardware operation state. The system constructs a performance prediction model based on information such as dimensional characteristics, data layout, and memory occupancy, models and prunes the candidate kernels, and selects the optimal combination. At the same time, it supports automatic switching and optimization updates during runtime, enabling the system to always maintain the best performance in the face of dynamic changes in input or floating platform resources, enhancing the overall computational stability and portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0049] Figure 1 It is a schematic diagram of the matrix multiplication optimization method for a matrix acceleration unit of the present invention;

[0050] Figure 2 It is a schematic diagram of the high-performance micro-kernel generation framework in a specific embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of the matrix multiplication optimization system for a matrix acceleration unit of the present invention;

[0052] Figure 4 This is a schematic diagram of the computer hardware of the present invention. Detailed implementation manners

[0053] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following items (pieces)" or similar expressions refer to any combination of these items, including any combination of single item (piece) or plural items (pieces). For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c can be single or multiple.

[0054] It should also be understood that the term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be specifically understood by referring to the context before and after.

[0055] It should also be understood that in various embodiments of the present invention, the magnitudes of the serial numbers of the above processes do not mean the sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0056] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0057] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0058] In addition, in each embodiment of the present invention, the functional units can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0059] When the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0060] To make the above features and effects of the present invention more clearly and understandably described, specific embodiments are hereinafter given and detailed descriptions are made in conjunction with the accompanying drawings of the specification. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are only for illustrative purposes. The protection scope of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the appended claims.

[0061] The following is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in cooperation with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. To avoid repetition, they are not elaborated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0062] The method of the present invention for matrix-matrix multiplication with an input of an irregular-shaped matrix has been widely applied in many applications such as deep learning. However, the traditional implementation and optimization methods for general matrix-matrix multiplication are not suitable for calculating matrix-matrix multiplication of non-regular shapes, resulting in poor performance for matrix-matrix multiplication of non-regular matrices. Therefore, the present invention proposes an adaptive tuning framework, which realizes high performance for matrix-matrix multiplication of irregular matrices by implementing an optimal block algorithm based on Cache and pre-blocking at runtime.

[0063] The present invention aims to provide a dynamic shape adaptive matrix multiplication optimization method and system for a matrix acceleration unit. The present invention proposes a two-stage matrix multiplication optimization mechanism combining static pre-optimization and dynamic block recombination to achieve cross-platform and high-performance matrix calculations.

[0064] In view of the problems of unstable performance and poor adaptability of traditional matrix multiplication when dealing with dynamic input shapes and heterogeneous hardware platforms, the present invention proposes a dynamic optimization method and system for input-adaptive matrix multiplication for matrix acceleration units. Through a two-stage execution mechanism combining static pre-optimization and dynamic block reorganization, the method automatically adapts to the characteristics of the hardware architecture and the shape of the input matrix, generates an optimal execution strategy, and effectively improves the computational performance of matrix multiplication under multi-platform and multi-form inputs.

[0065] Through a high-performance micro-kernel generation framework, the system can pre-generate efficient kernels according to the differences in matrix multiplication instructions before execution, avoiding the overhead of repeated optimization during runtime; during the execution stage, the system can dynamically select the micro-kernel reorganization method according to the matrix input characteristics and adjust the loop structure and scheduling strategy to achieve fast adaptation and performance migration of the matrix acceleration unit.

[0066] The following details the method of the embodiments of the present application with specific embodiments:

[0067] Embodiment 1

[0068] As Figure 1 shown in Figure 1 is a schematic diagram of the matrix multiplication optimization method for matrix acceleration units of the present invention; in view of the performance adaptation problem of matrix multiplication units and dynamic inputs, the present invention proposes an input-adaptive matrix multiplication optimization method. The overall process includes three steps: micro-kernel pre-generation, input analysis, dynamic blocking, and execution plan generation, ensuring that matrix multiplication can be efficiently executed under various matrix shapes and hardware platforms.

[0069] Specifically, the embodiments of the present application provide a matrix multiplication optimization method for matrix acceleration units, and the method includes:

[0070] Static pre-optimization step 101: Through the matrix multiplication instruction set provided by the matrix acceleration unit, a micro-kernel generation framework is jointly constructed by using thread rearrangement, address mapping strategy, and multi-level pipeline mechanism to generate a set of candidate micro-kernels;

[0071] Dynamic block reorganization step 102: Feature extraction is performed on the input matrix, and a search space composed of a dynamic block and a scheduling strategy is constructed according to the input matrix characteristics. In the search space, the optimal micro-kernel is dynamically selected according to the input matrix characteristics and the pre-generated set of candidate micro-kernels, and the scheduling strategy is dynamically adjusted;

[0072] Performance modeling step 103: Based on the characteristics of the input matrix and the hardware characteristics, a performance prediction model is constructed to model and prune the set of candidate micro-kernels, and the optimal scheduling strategy combination is preferentially selected in the search space to achieve the optimization of matrix multiplication for matrix acceleration units.

[0073] Automatic update step 104: Automatically switch the scheduling policy during runtime, and optimize and update the scheduling policy in real time according to the dynamic changes of the input matrix features or the floating of platform resources.

[0074] In the embodiment of the present invention, the above-mentioned static pre-optimization step 101 includes:

[0075] Collect the features of the target hardware platform and construct a hardware feature library, where the hardware feature library includes: matrix multiplication instruction set, number of parallel processing units, shared memory size, single instruction multiple data width;

[0076] Fuse the matrix multiplication instruction set provided by the hardware and perform instruction scheduling. During this process, call the matrix instruction code generator written according to the hardware platform to generate corresponding computing kernels based on different computing block sizes set by static pre-optimization.

[0077] Adopt a thread rearrangement and address mapping strategy to check whether there are bank conflicts in the read and write related to the shared memory. If there is a conflict, perform address remapping to make the memory access addressing of the threads staggered to avoid conflicts. Among them, the address mapping strategy is: divide the shared memory into square blocks, perform an exclusive OR operation on the column coordinates and row coordinates within the square block to obtain new column or row coordinates, while the other coordinate remains unchanged.

[0078] Adopt a multi-level pipeline mechanism and use a pre-written multi-level pipeline code template to generate a set of candidate micro-kernels. Among them, in the multi-level pipeline code template, by initiating multi-level asynchronous memory access, the calculation is masked with asynchronous memory access, and the data required for synchronous calculation is waited for by instructions.

[0079] More specifically, micro-kernel pre-generation and static optimization include:

[0080] In the process of implementing the technology of the present invention, it is first necessary to conduct an in-depth feature analysis of the target hardware platform and collect its detailed hardware information such as matrix multiplication instruction set, number of parallel processing units, shared memory size, SIMD (single instruction multiple data) width, etc. Based on the collected data, construct a hardware feature library to provide a solid foundation for subsequent micro-kernel generation and optimization.

[0081] Next, as Figure 2As shown in the figure, the present invention utilizes a high-performance microkernel generation framework, takes the size and precision of the required microkernel as input, and runs a code generation program. The above microkernel optimization method is incorporated into this program, thereby generating corresponding codes for different microkernel sizes. These microkernel sizes cover common matrix multiplication sizes, such as 8×8, 16×16, etc. The codes corresponding to each size will use different matrix multiplication instruction sets to adapt to the characteristics of the hardware. The core optimization is mainly reflected in three aspects: First, by fusing the matrix multiplication instruction sets provided by the hardware, matrix multiplication calculation logic with matrix multiplication instructions as the core is added to the code generation framework to achieve efficient instruction scheduling; Second, a thread rearrangement and address mapping strategy is adopted, and the rows and columns in the shared memory are exclusive-ORed to make the memory access addresses staggered, avoiding bank conflicts in the shared memory, thereby improving the memory access efficiency; Finally, a multi-level pipeline mechanism is introduced. By using asynchronous copies corresponding to each level and synchronization instructions before each level of calculation, the calculation and memory access operations can be executed in parallel, masking the memory access latency to the greatest extent and further improving the throughput performance of the system.

[0082] In the embodiment of the present invention, the above dynamic block reorganization step 102 includes:

[0083] Generate multiple combinations of block sizes (such as 16x16, 32x32, or 64x64) in the search space, and allocate different processing threads for parallel calculation for each block method. In order to avoid an excessive number of enumerated block methods, the block strategy pre-establishes a small number of coarse-grained block templates, obtains coarse-grained blocks through mapping of the coarse-grained block templates, and further divides each block in the coarse-grained block into equal-sized sub-blocks on this basis.

[0084] Loop scheduling optimization. To avoid the problem of low utilization of computing resources when the block scheduling for the output shape has too small dimensions in the input, consider block and schedule the batch processing or K dimension, and add corresponding scheduling strategies to the search space.

[0085] More specifically, the input analysis and dynamic block strategy generation include: during the runtime phase, when performing feature extraction on the input matrix, the following three key dimensions need to be considered first: the dimensions of the matrix (m, n, k), the computing precision (such as FP32, FP16, BF16), and the batch size (Batch Size). To fully utilize the efficiency of the matrix processing unit, a search space for dynamic block and scheduling strategies can be constructed based on the input features. In terms of block recombination, the system generates a set of possible choices for microkernel recombination according to the input features and the pre-generated set of microkernels. On this basis, corresponding scheduling choices are generated in the search space, including: the block parallel thread division strategy, which first determines the coarse-grained block template through the output size, then equally divides the coarse-grained block result into the size corresponding to the microkernel, and assigns it to different processing threads to achieve parallel computing; loop scheduling, when a certain dimension of the input is too small, by introducing parallel other dimensions, such as the batch processing dimension and the K dimension, to avoid waste of computing resources.

[0086] In the embodiment of the present invention, the above performance modeling step 103 includes:

[0087] Pre-training a performance prediction model based on the input features, hardware features, and historical execution statistical information;

[0088] Based on the performance prediction model, preferentially select scheduling strategies with higher expected performance in the search space.

[0089] To avoid the time overhead caused by traversing the entire search space at runtime, the present invention performs pruning through performance modeling. The performance model is based on the static microkernel execution time, and approximates the time after recombination by summing the microkernel times in each recombination method, so as to prune the solutions with too long combined times in advance. Based on the input features, hardware features, and historical execution statistical information, quickly predict the execution efficiency of the candidate strategies. In this way, strategies with higher expected performance can be preferentially selected in the search space, reducing unnecessary calculations and improving the overall system performance. This method effectively optimizes the execution efficiency of matrix calculations and fully utilizes hardware resources by dynamically adjusting the block and scheduling strategies and combining performance modeling.

[0090] In addition, the execution plan generation and performance evaluation include: based on the preprocessing results of steps 101 and 102, the system completes the deployment of the final scheduling plan through the execution plan generation module. First, the system will conduct comprehensive tests using the constructed search space. By evaluating these combinations, the system can select the optimal configuration for the current platform and input, thereby maximizing the computing efficiency.

[0091] As described above, the method of the present invention can be preferably implemented.

[0092] This invention has been implemented and verified on mainstream matrix multiplication acceleration units (such as NVIDIA Tensor Core, AMD Matrix Core). Experimental results show that without the need for manual adjustment by developers, high performance comparable to handwritten kernels can be achieved, and better performance than general operator libraries in irregular matrix multiplication. It has good cross-platform portability and versatility.

[0093] In summary, the present invention significantly improves the automation and adaptability of matrix multiplication optimization. It is a general high-performance optimization solution that can be applied to various matrix multiplication architectures and supports dynamic input scenarios.

[0094] Example 2

[0095] like Figure 3 As shown, an embodiment of the present application provides a matrix multiplication optimization system for a matrix acceleration unit, which adopts the matrix multiplication optimization method for the matrix acceleration unit as described above, and the system includes:

[0096] Static pre-optimization module 201: uses the matrix multiplication instruction set provided by the matrix acceleration unit, adopts thread reordering, address mapping strategy and multi-stage pipeline mechanism to jointly build a microkernel generation framework and generate a set of candidate microkernels;

[0097] Dynamic Block Recombination Module 202: Extracts features from the input matrix and constructs a search space consisting of dynamic blocks and scheduling strategies based on the input matrix features. In the search space, the optimal microkernel is dynamically selected and the scheduling strategy is dynamically adjusted based on the input matrix features and the pre-generated candidate microkernel set.

[0098] Performance Modeling Module 203: Based on the characteristics of the input matrix and hardware characteristics, a performance prediction model is constructed, and the candidate microkernel set is modeled and pruned. The optimal scheduling strategy combination is preferentially selected in the search space to optimize the matrix multiplication for the matrix acceleration unit.

[0099] Automatic update module 204: automatically switches the scheduling strategy during runtime, and optimizes and updates the scheduling strategy in real time according to the dynamic changes of input matrix characteristics or platform resource fluctuations.

[0100] In the embodiment of the present invention, the static pre-optimization module 201 includes:

[0101] Collect the characteristics of the target hardware platform and build a hardware feature library. The hardware feature library includes: matrix multiplication instruction set, number of parallel processing units, shared memory size, and single instruction multiple data width;

[0102] Integrate the matrix multiplication instruction set provided by the hardware and perform instruction scheduling. During this process, call the matrix instruction code generator written according to the hardware platform to generate corresponding computing kernels based on different computing block sizes set by static pre-optimization.

[0103] Adopt the thread rearrangement and address mapping strategy to check whether there are bank conflicts in the reads and writes related to the shared memory. If there are conflicts, perform address remapping to make the memory access addressing of the threads staggered and avoid conflicts. Among them, the address mapping strategy is: divide the shared memory into square blocks, perform an exclusive OR operation on the column coordinates and row coordinates within the square block to obtain new column or row coordinates, while the other coordinate remains unchanged.

[0104] Adopt a multi-level pipeline mechanism and use a pre-written multi-level pipeline code template to generate a set of candidate micro-kernels. Among them, in the multi-level pipeline code template, by initiating multi-level asynchronous memory access, the calculation is masked with the asynchronous memory access, and the data required for the calculation is synchronized by waiting for instructions.

[0105] In the embodiment of the present invention, the above-mentioned dynamic block reorganization module 202 includes:

[0106] Generate corresponding block strategies in the search space, determine that the matrix is divided into multiple blocks according to a predetermined division strategy and assigned to different processing threads. Among them, the block strategy is: map to a small number of pre-established coarse-grained block templates, obtain coarse-grained blocks through the mapping of the coarse-grained block templates, and on this basis, divide each block in the coarse-grained blocks into equal-sized sub-blocks. Scheduling

[0107] Loop scheduling optimization. To avoid the problem of low computing resource utilization when the block scheduling for the output shape has some dimensions too small in the input, consider dividing and scheduling the batch processing or K dimension, and add corresponding scheduling strategies in the search space.

[0108] Coarse strategy for loop scheduling optimization. To avoid the problem of low computing resource utilization when the block scheduling for the output shape has some dimensions too small in the input, consider dividing and scheduling the batch processing or K dimension, and add corresponding scheduling strategies in the search space.

[0109] Embodiment III

[0110] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the matrix multiplication optimization method for the matrix acceleration unit are implemented.

[0111] Embodiment IV

[0112] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the matrix multiplication optimization method for a matrix acceleration unit as described are implemented.

[0113] In addition, the matrix multiplication optimization method for a matrix acceleration unit according to the embodiment of the present application described in Figure 1 can be implemented by an electronic device, such as a computer device. Figure 4 FIG. is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0114] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Among them, as Figure 4 shown, the processor 81, the memory 82, and the communication interface 83 are connected through the bus 80 and complete communication with each other.

[0115] Specifically, the above-mentioned processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits implementing the embodiments of the present application.

[0116] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81.

[0117] The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement any one of the matrix multiplication optimization methods for a matrix acceleration unit in the above embodiments.

[0118] The technical features of the above-mentioned embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0119] The above-mentioned embodiments only represent several implementation manners of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several deformations and improvements can be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A matrix multiplication optimization method for a matrix acceleration unit, characterized in that, The method includes: Static pre-optimization step: By using the matrix multiplication instruction set provided by the matrix acceleration unit, adopting thread rearrangement, address mapping strategy and multi-level pipeline mechanism, jointly constructing a micro-kernel generation framework to generate a set of candidate micro-kernels; Dynamic block reorganization step: Extract features for the input matrix, construct a search space composed of a dynamic block and scheduling strategy according to the input matrix features. In the search space, according to the input matrix features and the pre-generated set of candidate micro-kernels, dynamically select the optimal micro-kernel and dynamically adjust the scheduling strategy; Performance modeling step: Based on the features of the input matrix and hardware features, construct a performance prediction model, model and prune the set of candidate micro-kernels, and preferentially select the optimal scheduling strategy combination in the search space to achieve the optimization of matrix multiplication for the matrix acceleration unit.

2. The matrix multiplication optimization method for a matrix acceleration unit according to claim 1, wherein The method further includes: Automatic update step: Automatically switch the scheduling strategy during runtime, and optimize and update the scheduling strategy in real time according to the dynamic changes of the input matrix features or platform resource fluctuations.

3. The matrix multiplication optimization method for a matrix acceleration unit according to claim 1 or 2, characterized in that The static pre-optimization step includes: Collect the features of the target hardware platform and construct a hardware feature library, where the hardware feature library includes: matrix multiplication instruction set, number of parallel processing units, shared memory size, single instruction multiple data width; Fuse the matrix multiplication instruction set provided by the hardware, perform instruction scheduling, and call the matrix instruction code generator written according to the hardware platform to generate corresponding computing kernels based on different computing block sizes set by static pre-optimization; Adopt thread rearrangement and address mapping strategy to check whether there are bank conflicts in the read and write related to the shared memory. If there is a conflict, perform address remapping to make the memory access addressing of the threads staggered. Divide the shared memory into square blocks, perform an exclusive OR operation on the column coordinates and row coordinates within the square block to obtain new column or row coordinates, while the other coordinate remains unchanged; Adopt a multi-level pipeline mechanism, use the pre-written multi-level pipeline code template to generate a set of candidate micro-kernels. In the multi-level pipeline code template, by initiating multi-level asynchronous memory access, cover the calculation with asynchronous memory access. Wait for the instruction to synchronize the required data for calculation. The n-level pipeline initiates n asynchronous copies of the global memory at the beginning of the program, synchronizes the corresponding asynchronous copies during each calculation, and initiates another asynchronous copy after the calculation ends.

4. The matrix multiplication optimization method for a matrix acceleration unit according to claim 1 or 2, characterized in that The dynamic block reorganization step includes: Generate corresponding scheduling strategies in the search space. According to the block parallel thread division strategy, determine that the matrix is divided into multiple blocks according to a predetermined division strategy and assigned to different processing threads. In order to avoid too many enumerated block division methods, the block division strategy pre-defines a small number of coarse-grained block templates, obtains the coarse-grained blocks through the mapping of the coarse-grained block templates, and further divides each block in the coarse-grained blocks into equal-sized sub-blocks; Loop scheduling optimization, perform block division and scheduling on batch processing or K dimension, and add corresponding scheduling strategies in the search space.

5. The matrix multiplication optimization method for a matrix acceleration unit according to claim 1 or 2, characterized in that The performance modeling step includes: Pre-train a performance prediction model based on input features, hardware features, and historical execution statistical information; Based on the performance prediction model, preferentially select the scheduling policies with higher expected performance in the search space.

6. A matrix multiplication optimization system for a matrix acceleration unit, which adopts the matrix multiplication optimization method for a matrix acceleration unit as described in any one of claims 1-5, characterized in that, The system includes: Static pre-optimization module: Through the matrix multiplication instruction set provided by the matrix acceleration unit, adopt thread rearrangement, address mapping strategy and multi-level pipeline mechanism to jointly construct a micro-kernel generation framework and generate a set of candidate micro-kernels; Dynamic block reorganization module: Extract features for the input matrix, construct a search space composed of dynamic blocks and scheduling policies according to the input matrix features. In the search space, dynamically select the optimal micro-kernel and dynamically adjust the scheduling policy according to the input matrix features and the pre-generated set of candidate micro-kernels; Performance modeling module: Based on the features of the input matrix and hardware features, construct a performance prediction model, perform modeling and pruning on the set of candidate micro-kernels, and preferentially select the optimal scheduling policy combination in the search space to achieve the optimization of matrix multiplication for the matrix acceleration unit; Automatic update module: Automatically switch the scheduling policy during runtime, and optimize and update the scheduling policy in real time according to the dynamic changes of the input matrix features or the floating of platform resources.

7. The matrix multiplication optimization system for a matrix acceleration unit according to claim 6, wherein The static pre-optimization module includes the following execution steps: Collect the features of the target hardware platform and construct a hardware feature library, where the hardware feature library includes: matrix multiplication instruction set, number of parallel processing units, shared memory size, single instruction multiple data width; Integrate the matrix multiplication instruction set provided by the hardware, perform instruction scheduling, and call the matrix instruction code generator written according to the hardware platform to generate corresponding computing kernels based on different computing block sizes set by static pre-optimization; Adopt thread rearrangement and address mapping strategy to check whether there are bank conflicts in the read and write related to the shared memory. If there is a conflict, perform address remapping to make the memory access addressing of the threads staggered, divide the shared memory into square blocks, perform exclusive OR operations on the column coordinates and row coordinates within the square block to obtain new column or row coordinates, while the other coordinate remains unchanged; Adopt a multi-level pipeline mechanism, use the pre-written multi-level pipeline code template to generate a set of candidate micro-kernels. In the multi-level pipeline code template, the calculation is masked by asynchronous memory access by initiating multi-level asynchronous memory access. Wait for the instruction to synchronize the required data for calculation. The n-level pipeline initiates n asynchronous copies of the global memory at the beginning of the program, synchronizes the corresponding asynchronous copies during each calculation, and initiates another asynchronous copy after the calculation ends.

8. The matrix multiplication optimization system for a matrix acceleration unit according to claim 6, characterized in that, The dynamic block reorganization module includes: Generate corresponding scheduling policies in the search space. According to the block parallel thread division strategy, determine that the matrix is divided into multiple blocks according to a predetermined division strategy and assigned to different processing threads. In order to avoid too many enumerated block methods, the block strategy pre-defines a small number of coarse-grained block templates, obtains coarse-grained blocks through mapping of the coarse-grained block templates, and further divides each block in the coarse-grained block into equal-sized sub-blocks; Loop scheduling optimization, perform block division and scheduling on batch processing or K dimension, and add corresponding scheduling policies in the search space.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the matrix multiplication optimization method for a matrix acceleration unit described in any one of claims 1-5.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the matrix multiplication optimization method for a matrix acceleration unit described in any one of claims 1 to 5.

Images