Convolution operator compilation optimization method and device for multi-core parallel architecture

Abstract

The application provides a convolution operator compilation optimization method and device for a multi-core parallel architecture. The method comprises the following steps: an input converter module extracts and removes convolution parameters in a model or user input, and generates a convolution descriptor group; a controller module determines whether to start optimization for the input module according to historical records, and transmits the convolution descriptors that need to be optimized to an optimizer module; the optimizer module is responsible for automatic optimization of the operator according to a configured parameter space, and outputs an optimal parameter configuration; a code generation module generates a HIP code according to the received optimization parameter combination and the convolution descriptor, and generates corresponding numbers for the kernel functions generated by each parameter. A kernel function execution module compiles and executes the generated code, tests the performance of each kernel function through a test framework, returns corresponding fitness information according to the numbers, and returns the fitness information as a list to the optimizer module.

Description

Technical Field

[0001] This invention relates to the fields of parallel computing and deep learning operator compilation optimization, and particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for compiling and optimizing convolution operators for multi-core parallel architectures. Background Technology

[0002] Convolutional Neural Networks (CNNs) are increasingly widely used due to their superior adaptability in fields such as computer vision, digital image processing, and text-to-image processing. Convolution, especially 2D convolution, is one of the most commonly used operators in CNN models, and its performance directly impacts the training and inference efficiency of the neural network. Meanwhile, GPUs, with their high parallelism, high-bandwidth memory access, and outstanding floating-point computing capabilities, are widely used to accelerate the computation of deep learning.

[0003] To further improve the computational speed of deep learning neural network models such as CNNs on GPUs, many commercial high-performance operator libraries and compilers for deep learning prioritize the optimization of convolution operators. These are implemented by integrating various manually optimized low-level operators, or through automatic code generation and parameter tuning. However, for computing acceleration devices like AMD GPUs, optimization of convolution operators for such hardware remains insufficient. Furthermore, the optimization effects of different combinations of optimizations vary depending on the input parameters and hardware platform, making it impossible to find a single, universally efficient convolution implementation.

[0004] Currently, to accelerate convolution operator computation on GPUs, the mainstream approach is to design and optimize operator implementations for the dedicated matrix multiply-add (MMA) acceleration units integrated on GPUs. The main implementation methods for convolution operators fall into four categories: direct convolution algorithms, FFT convolution algorithms, convolution algorithms based on Generalized Matrix Multiplication (GEMM), and Winograd convolution algorithms. Among these, the GEMM convolution algorithm has been proven to be an efficient implementation of convolution operators suitable for matrix multiply-add units on GPUs.

[0005] Currently, MIOpen is a high-performance machine learning operator library for the ROCm software stack. MIOpen incorporates the four convolution implementation techniques mentioned above and has built a large number of manually optimized operator implementations for each implementation and supported hardware architecture. It can also search its own operator library for the most efficient operator based on user-input convolution parameters. However, research shows that MIOpen's convolution implementation lags behind hardware peak performance by more than 30%, indicating room for performance improvement. Furthermore, many methods in MIOpen are implemented in assembly language, which poses challenges for users to perform custom operator fusion with other operators.

[0006] Furthermore, mainstream deep learning compilers and optimizers, such as TVM and Triton, have integrated convolution operator generation and automatic tuning functions, and have achieved cross-platform support. However, TVM is currently not adapted for the MMA-specific acceleration unit of AMD GPUs' CDNA architecture, and can only use scalar units for computation. At the same time, the code tuning logic built into these compilers mainly uses general code optimization strategies. Although compilers like Triton have specific optimizations for matrix multiplication, the memory access and computation patterns of convolution still differ somewhat from matrix multiplication, and their optimization strategies do not fully match the characteristics of GEMM convolution computation. Despite thorough tuning, the performance of the convolution operators generated by these optimizers is still far lower than the peak performance of hardware with the same precision.

[0007] Extensive research has been conducted on GEMM convolutions, employing various methods to alleviate bottlenecks in parallelism and memory access. However, most of these works utilize their custom, efficient data formats as the standard input and convolution kernel tensor organization. In contrast, mainstream deep learning frameworks like PyTorch and TensorFlow primarily support specific data formats, such as NCHW or NHWC standard tensor inputs stored in memory. NCHW is a common 4D tensor data format in deep learning and a fundamental data format for current mainstream deep learning frameworks, where N represents Batch Size, C represents the number of Channels, and H and W represent the height and width of each input image. The pure memory access latency of converting from NCHW to a custom data format impacts the latency of high-performance operators, limiting their overall performance. Furthermore, many works use zero-padding to address bank conflicts in shared memory access within computational units. However, this method wastes shared memory space, leading to a decrease in the number of schedulable thread blocks per unit and reducing hardware parallelism. Summary of the Invention

[0008] To address the issue that convolution operators in multi-core parallel architectures cannot fully utilize hardware computing power, and the shortcomings of existing high-performance optimization and implementation schemes for convolution operators, this invention proposes a compilation optimization method and system for convolution operators in multi-core parallel architectures, comprising:

[0009] 1. Blocking strategy and tuning parameters for cache hierarchy design in multi-core parallel architecture: Implicit matrix multiplication and convolution calculation and memory access tasks need to be divided into GPU thread blocks and warps layer by layer, while controlling the occupation of resources such as shared memory and registers, and providing tuning parameters.

[0010] 2. Automatically selected soft pipelining technique: The soft pipelining strategy can improve the parallelism of memory access and computation when appropriate. At the same time, the tuner will automatically select whether to use the strategy to prevent inappropriate use from causing performance degradation.

[0011] 3. High-efficiency data format conversion technology: This technology addresses the low conversion efficiency of the NCHW input format to the NC*HW_X high-efficiency format. By changing the tensor arrangement of different channels, high-efficiency format conversion is achieved for graphs of various input sizes. NC*HW_X is a high-efficiency data format based on NCHW, which groups X consecutive channel dimensions of data for each point (with the same H and W) in each input graph within the same batch together. Because the converted data format groups X channel dimensions of data together, the size of X can be adjusted to meet the high bandwidth memory access requirements of the GPU, allowing multiple data to be read simultaneously to improve memory access bandwidth.

[0012] 4. Adjustable and efficient data format application: For different load conditions, the size of the X dimension of the NC*HW_X format is used as an adjustable parameter to automatically search for configurations with high locality or high data reuse rate, thereby improving memory access efficiency.

[0013] 5. A non-filling building conflict elimination strategy for GPU shared memory and its automated implementation: This method avoids building conflicts and wastes shared memory space, thus improving CUOccupancy. An automatic coordinate calculation method is also proposed and applied to the code generator.

[0014] 6. A fast and automatic optimization method for parameters: Using a genetic algorithm, the population is rapidly iterated by increasing the mutation rate and crossover rate. An optimal inheritance mechanism is added to the "roulette wheel" selection method to ensure the effective inheritance of "genes" in high-fitness parameter groups and achieve rapid convergence.

[0015] Specifically, addressing the shortcomings of existing technologies, this invention proposes a compilation optimization method for convolution operators in multi-core parallel architectures, including:

[0016] Step 1: Convert the input parameters of the convolution to be tuned into convolution descriptors;

[0017] Step 2: Perform deduplication on the collected convolution descriptors to generate a list of convolution descriptors;

[0018] Step 3: Read the Log file into dictionary D. The Log file stores the history of convolution descriptors and their corresponding optimal parameter combinations; initialize dictionary S to store convolution descriptors and their corresponding optimal parameter combinations.

[0019] Step 4: Read the convolution descriptors sequentially from the descriptor list and determine whether they are in the dictionary D. If they are, proceed to step 7; otherwise, determine whether they are in the dictionary S. If they are, proceed to step 4 again to retrieve the next convolution descriptor; otherwise, proceed to step 5.

[0020] Step 5: The controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the optimal combination of parameters;

[0021] Step 6: Write the convolution descriptor and its corresponding optimal parameter combination into dictionary D, and then execute step 8;

[0022] Step 7: Read the optimal combination of parameters corresponding to the convolution descriptor from the dictionary D;

[0023] Step 8: Write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S. Determine if all convolution descriptors in the convolution descriptor list have been optimized. If so, proceed to step 9; otherwise, proceed to step 4.

[0024] Step 9: The controller numbers the convolution descriptors in the dictionary S and saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; it generates HIP code based on the dictionary S and the code template; by compiling the HIP code, an executable file is obtained, and the neural network accelerator runs the executable file based on the convolution descriptors and the Model.log file to obtain the convolution operation result.

[0025] The aforementioned compilation optimization method for convolution operators in multi-core parallel architectures, wherein step 5 includes:

[0026] Step 51: Initialize the dictionary D used to record the fitness of optimized combinations, and initialize the population;

[0027] Step 52: Traverse the individuals in the population and check the configuration validity of the individual based on the hardware resource information of the backend hardware. For invalid configurations, set the fitness to 0 directly.

[0028] Step 53: Check if the individuals in the population that passed the legality check are in the dictionary D. If they are, directly read their fitness; otherwise, proceed to step 54.

[0029] Step 54: Group the individuals in the population that have not yet acquired fitness by number to generate the combination of parameters to be tested;

[0030] Step 55: The code generation module generates kernel function code based on the combination of parameters to be tested; and the generation process includes determining whether to generate code using soft pipeline technology or not based on the tuning parameters.

[0031] This soft pipeline technology includes parallel execution of global memory access, shared memory access, and MatrixCore calculations; data stored in shared memory must wait for the global memory access to finish before it can be retrieved, and this continues until all thread bundles have filled the data obtained from global memory into shared memory, enabling data exchange and sharing between thread bundles; and according to the requirements of MFMA instructions, each piece of data is placed into the register of the specified thread.

[0032] This invention provides a method for eliminating shared memory conflicts in implicit matrix multiplication and convolution with no padding in a multi-core parallel architecture. The shared memory operation includes two processes: storing the data taken from global memory into a specified location in shared memory; and each thread taking the data it needs from the specified location in shared memory. The arrangement needs to ensure that no shared memory conflicts occur in either of these processes.

[0033] The implicit matrix multiplication convolution kernel generated in step 55 adopts the NC*HW_X format; this data format takes the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as an optimization parameter to explore the optimal configuration.

[0034] Specifically, step 55 converts the NCHW format to the Nc*HW_X format using the following method:

[0035] First, the input tensor is reshaped to generate a batch of identically sized matrices M to be transposed. a Then, transpose each of the batch matrices to obtain the batch of... The matrix is ​​then reshaped to obtain the output tensor in Nc*HW_X format.

[0036] The method for transposing each of the Batch matrices is NCHW=N·X·(C / X)·HW=N·X·c*·HW, which is the transpose of the matrix of batch=N with X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix.

[0037] Step 56: Submit the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and return the fitness information of the parameter combination to be tested;

[0038] Step 57: Update the fitness information to dictionary D;

[0039] Step 58: Select the optimal parameter combination with the highest fitness in the current population as the parent generation, and select at least one optimal parameter combination from the other multiple optimal parameter combinations in the current population (excluding the optimal parameter combination with the highest fitness) as the parent generation using the roulette wheel algorithm. All parents perform mutation and crossover of the parameter combination genes according to the preset crossover rate and mutation rate to update the population. Repeat step 52 until the optimization reaches the specified number of rounds. Then select the individual parameter with the best fitness in the dictionary D as the best optimal parameter combination.

[0040] The aforementioned convolution operator compilation optimization method for multi-core parallel architectures, wherein step 56 includes:

[0041] Step 561: Initialize the set F used to store fitness information;

[0042] Step 562: Call the compilation script to compile and link the kernel function code into the test framework;

[0043] Step 563: Start the test program, input the kernel function code to be tested and the convolution descriptor into the test framework, test and return the fitness of each kernel function code;

[0044] Step 564: Write the measured fitness into the position corresponding to the kernel function code ID in the fitness information set F.

[0045] The aforementioned convolution operator compilation optimization method for multi-core parallel architectures, wherein step 9, generating HIP code based on the dictionary S and the code template, specifically includes:

[0046] For a convolution descriptor and its optimized parameter combination, a preset function template is selected; this function template contains multiple identifiers; the code generator of the neural network accelerator generates a code block in the function template based on the convolution parameters in the convolution descriptor and the optimal optimized parameter combination; the code block is filled into the corresponding identifier position in the template to obtain the kernel function; the code generator generates an outer wrapper function based on the number, so as to call the corresponding kernel function by number; the HIP file is generated based on the outer wrapper function and the kernel function.

[0047] The aforementioned convolution operator compilation optimization method for multi-core parallel architectures, wherein in step 5, the tuner is used to calculate the arithmetic strength AI of the operator in the convolution to be tuned using the following formula, and to determine the matrix partitioning strategy of the implicit matrix multiplication result matrix based on the arithmetic strength:

[0048]

[0049] In the formula TILE M and TILE N These represent the block sizes in the M and N directions of the resulting matrix, respectively; FH and FW are the kernel sizes, respectively; IC is the number of input channels; and a larger TILE is selected. M and TILE N Divide into blocks, and use TILE K Set to a size suitable for task allocation.

[0050] This invention also proposes a compilation optimization device for convolution operators in multi-core parallel architectures, including:

[0051] Module 1: Convert the input parameters of the convolution to be tuned into convolution descriptors;

[0052] Module 2: Perform deduplication on the collected convolution descriptors to generate a list of convolution descriptors;

[0053] Module 3: Read the Log file into dictionary D, which stores the history of convolution descriptors and their corresponding optimal parameter combinations; initialize dictionary S for storing convolution descriptors and their corresponding optimal parameter combinations.

[0054] Module 4: Read convolution descriptors sequentially from the descriptor list and determine whether they are in the dictionary D. If they are, execute Module 7; otherwise, determine whether they are in the dictionary S. If they are, execute Module 4 again to retrieve the next convolution descriptor; otherwise, execute Module 5.

[0055] Module 5: The controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the optimal combination of parameters;

[0056] Module 6: Write the convolution descriptor and its corresponding optimal parameter combination into dictionary D, and execute module 8;

[0057] Module 7: Read the optimal combination of parameters corresponding to the convolution descriptor from the dictionary D;

[0058] Module 8: Write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S. Determine if all convolution descriptors in the convolution descriptor list have been optimized. If so, execute Module 9; otherwise, execute Module 4.

[0059] Module 9: The controller numbers the convolution descriptors in the dictionary S and saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; it generates HIP code based on the dictionary S and the code template; by compiling the HIP code, an executable file is obtained, and the neural network accelerator runs the executable file based on the convolution descriptors and the Model.log file to obtain the convolution operation result.

[0060] The aforementioned convolution operator compilation optimization device for multi-core parallel architecture, wherein module 5 includes:

[0061] Module 51: Initialize the dictionary D used to record the fitness of optimal combinations, and initialize the population;

[0062] Module 52: Traverse the individuals in the population, check the configuration validity of the individual based on the hardware resource information of the backend hardware, and set the fitness of the individual to 0 directly for invalid configurations.

[0063] Module 53: Check if the individuals in the population that have passed the legality check are in the dictionary D. If they are, directly read their fitness; otherwise, execute module 54.

[0064] Module 54: Group the individuals in the population that have not yet acquired fitness by numbering them, and generate combinations of parameters to be tested;

[0065] Module 55, the code generation module generates kernel function code based on the combination of parameters to be tested; and the generation process includes determining whether to generate code using soft pipeline technology or not based on the tuning parameters;

[0066] This soft pipeline technology includes parallel execution of global memory access, shared memory access, and MatrixCore calculations; data stored in shared memory must wait for the global memory access to finish before it can be retrieved, and this continues until all thread bundles have filled the data obtained from global memory into shared memory, enabling data exchange and sharing between thread bundles; and according to the requirements of MFMA instructions, each piece of data is placed into the register of the specified thread.

[0067] This invention provides a method for eliminating shared memory conflicts in implicit matrix multiplication and convolution with no padding in a multi-core parallel architecture. The shared memory operation includes two processes: storing the data taken from global memory into a specified location in shared memory; and each thread taking the data it needs from the specified location in shared memory. The arrangement needs to ensure that no shared memory conflicts occur in either of these processes.

[0068] The implicit matrix multiplication convolution kernel generated by module 55 adopts the NC*HW_X format; this data format takes the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as an optimization parameter to explore the optimal configuration.

[0069] Specifically, module 55 converts the NCHW format to the Nc*HW_X format using the following method:

[0070] First, the input tensor is reshaped to generate a batch of identically sized matrices M to be transposed. a Then, transpose each of the batch matrices to obtain the batch of... The matrix is ​​then reshaped to obtain the output tensor in Nc*HW_X format.

[0071] The method for transposing each of the Batch matrices is NCHW=N·X·(C / X)·HW=N·X·c*·HW, which is the transpose of the matrix of batch=N with X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix.

[0072] Module 56: Hand over the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and return the fitness information of the parameter combination to be tested;

[0073] Module 57: Update the fitness information to dictionary D;

[0074] Module 58: Select the optimal parameter combination with the highest fitness in the current population as the parent generation, and select at least one optimal parameter combination from the other multiple optimal parameter combinations in the current population (excluding the optimal parameter combination with the highest fitness) as the parent generation. All parents perform mutation and crossover of the parameter combination genes according to the preset crossover rate and mutation rate to update the population. Module 52 is executed again until the optimization reaches the specified number of rounds. Then, the individual parameter with the best fitness in the dictionary D is selected as the best optimal parameter combination.

[0075] This module 56 includes:

[0076] Module 561: Initialize the set F used to store fitness information;

[0077] Module 562: Call the compilation script to compile and link the kernel function code into the test framework;

[0078] Module 563: Start the test program, input the kernel function code to be tested and the convolution descriptor into the test framework, test and return the fitness of each kernel function code;

[0079] Module 564: Write the measured fitness into the fitness information set F at the position corresponding to the kernel function code ID;

[0080] Module 9 generates HIP code based on the dictionary S and the code template, specifically including:

[0081] For a convolution descriptor and its optimized parameter combination, a preset function template is selected; this function template contains multiple identifiers; the code generator of the neural network accelerator generates a code block in the function template based on the convolution parameters in the convolution descriptor and the optimal optimized parameter combination; this code block is filled into the corresponding identifier position in the template to obtain the kernel function; the code generator generates an outer wrapper function based on the number, so as to call the corresponding kernel function by number; the HIP file is generated based on the outer wrapper function and the kernel function;

[0082] The tuner in module 5 is used to calculate the arithmetic strength AI of the operators in the convolution to be tuned using the following formula, and to determine the block strategy of the implicit matrix multiplication result matrix based on the arithmetic strength:

[0083]

[0084] In the formula TILE M and TILE N These represent the block sizes in the M and N directions of the resulting matrix, respectively; FH and FW are the kernel sizes, respectively; IC is the number of input channels; and a larger TILE is selected. M and TILE N Divide into blocks, and use TILE K Set to a size suitable for task allocation.

[0085] The present invention also proposes an electronic device, including the aforementioned convolution operator compilation and optimization device, which may be connected to an information display device for displaying the evaluation results using user-set display parameters, attributes, or through an artificial intelligence model.

[0086] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the convolution operator compilation optimization method.

[0087] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the convolution operator compilation optimization method.

[0088] As can be seen from the above solutions, the advantages of the present invention are:

[0089] 1) Compilation optimization based on multi-core parallel architecture characteristics: This system is developed based on the CDNA architecture and incorporates a variety of memory access and parallel optimization methods to suit the architectural characteristics of the platform hardware;

[0090] 2) Highly scalable code generation and automatic tuning: It can automatically generate and select appropriate tuning parameter configurations based on different inputs, and provide the optimal convolution code implementation under the given convolution input. The generated kernel function is mostly HIP C++ code, which is convenient for users to extend and integrate other operators.

[0091] 3) A scheme to eliminate shared memory conflicts without padding was used: This method eliminates conflicts while maximizing the utilization of shared memory space, further improving the hardware's memory access efficiency and parallelism.

[0092] 4) Efficient NCHW format conversion method: This method designs a new format conversion method, which solves the difficulty of efficient format conversion when NCHW format input is applied to implicit matrix multiplication and convolution, and can still perform efficient format conversion under small input image size;

[0093] 5) Improved system resource utilization and reduced execution time and operator tuning time for deep learning inference tasks: For multiple common CNN network models, compared with the function operator libraries of manufacturers, the method provided by this invention is on par with or exceeds their performance, and can achieve an inference performance speedup of up to 1.23×; compared with typical AI compilers, this invention can achieve a single operator performance speedup of 1.56× to 3.29×, and a tuning speedup of 22.0× to 38.5×. Attached Figure Description

[0094] Figure 1 A diagram of the system module structure for compiling and optimizing convolution operators for multi-core parallel architectures;

[0095] Figure 2 A flowchart of the compilation and optimization system for convolution operators oriented to multi-core parallel architectures;

[0096] Figure 3 A graph showing the effect of block size on the arithmetic strength of the convolution operator;

[0097] Figure 4 This is a schematic diagram of the flow stage division;

[0098] Figure 5 A diagram illustrating memory access discontinuity issues for NHWC and NCHW input formats;

[0099] Figure 6 This is a schematic diagram illustrating the tensor conversion principle from NCHW to Nc*HW_X.

[0100] Figure 7Diagram showing conflict determination and shared memory layout for buildings;

[0101] Figure 8 Here is a flowchart of the tuner's workflow;

[0102] Figure 9 Workflow diagram for code generation module;

[0103] Figure 10 Here is a flowchart of the kernel function execution module.

[0104] Figure 11 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0105] Figure 12 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0106] Figure 13 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0107] Figure label:

[0108] A - First electronic device;

[0109] B-convolution operator compiler optimization device;

[0110] C-Data acquisition equipment;

[0111] D-Information display device;

[0112] 1000 - Second electronic device;

[0113] Ⅰ-Computational Unit;

[0114] II-ROM;

[0115] III-RAM;

[0116] N-bus;

[0117] V-Interface;

[0118] VI - Input Unit;

[0119] VII - Output Unit;

[0120] VIII - Storage medium;

[0121] IX - Communication Unit. Detailed Implementation

[0122] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0123] Without further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0125] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0126] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0127] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0128] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0129] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0130] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0131] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0132] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0133] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0136] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] Based on the aforementioned features, this invention provides a convolution operator optimization method and a convolution operator compilation optimization system for multi-core parallel architectures, such as... Figure 1 As shown, the core components include:

[0138] 1. The automatic tuning module is responsible for the overall control of the system and the execution of the automatic tuning algorithm. This module consists of three sub-modules: the input converter module is responsible for extracting and deduplicating convolution parameters from the model or user input, and generating convolution descriptor sets; the controller module determines whether to initiate tuning of the input module based on historical records, and passes the convolution descriptors that need tuning to the tuner module; the tuner module is responsible for automatically tuning operators according to the configured parameter space, and outputting the optimal parameter configuration.

[0139] 2. Code Generation Module. This module is responsible for generating HIP code based on multiple sets of tuning parameter combinations and convolution descriptors received, and assigning corresponding numbers to the kernel functions generated for each parameter.

[0140] 3. Kernel Function Execution Module. This module is responsible for compiling and executing the generated code, testing the performance of each kernel function using a testing framework, returning the corresponding fitness information by number, and summarizing it into a list to return to the tuner module.

[0141] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0142] This invention provides a method and system for compiling and optimizing convolution operators for multi-core parallel architectures. The aim is to develop a high-performance automatic compiler optimizer for convolution operators on multi-core parallel architectures, combining the relevant hardware characteristics of the platform and employing various optimization methods to improve the performance of single convolution operators and accelerate deep neural network inference.

[0143] The following description, in conjunction with the accompanying drawings, will describe and explain the working steps of the compiler optimizer system in this invention, the implementation principles of each module's functions, and the characteristics of the operator optimization implementation method.

[0144] like Figure 1 As shown, the system implementation in this invention comprises three modules: an automatic optimization module, a code generation module, and a kernel function execution module. Each module further contains several sub-modules. Figure 2 This describes the workflow of the compiler optimizer system in this invention when using the method provided by this invention.

[0145] Automatic tuning module. This module consists of three parts: an input converter, a controller, and a tuner. This module controls the operation of the entire system. The input converter processes user input or 2D convolution operator parameters extracted from the model and submits them to the controller. The controller decides whether to initiate tuning for the convolution descriptor and hands over the descriptors that need tuning to the tuner. The tuner runs tuning routines, iteratively tuning the input parameters to obtain the optimal parameters. Finally, the code generation module generates executable code.

[0146] For the automatic tuning module, this invention provides a method for parallel optimization of convolution operators and memory access optimization for multi-core parallel architectures. The features of this module are as follows: Figures 3 to 7 As shown, it has the following 5 characteristics:

[0147] 1) The tuner performs matrix partitioning strategy analysis based on arithmetic strength: According to the Roofline model, the theoretical peak performance achievable by an operator is affected by its arithmetic strength. Insufficient arithmetic strength leads to limited memory access, affecting the program's theoretical peak performance. The formula for estimating arithmetic strength can be given as follows:

[0148]

[0149] TILE M and TILE N These represent the block sizes in the M and N directions of the resulting matrix from implicit matrix multiplication (GEMM), respectively. The arithmetic strength is only related to TILE. M and TILE N Related, and TILE M and TILE N The larger the kernel size (FH and FW), the higher the arithmetic strength; simultaneously, for the same block partitioning strategy, the larger the kernel size (FH and FW), the higher the input channel count (IC), and the greater the corresponding arithmetic strength. Figure 3 As shown. However, excessively large chunks consume more on-chip resources (registers, shared memory, etc.), leading to reduced occupancy; and when the result matrix is ​​small, an excessively large chunking strategy can cause the chunk matrix itself to exceed the boundary of the result matrix, resulting in meaningless redundant tasks. Therefore, for a given convolution task, an efficient strategy tends to choose a larger TILE within a reasonable range. M and TILE N Perform block partitioning, and simultaneously, due to the arithmetic strength and the block size in the K direction of the result matrix, TILE K It is irrelevant, therefore it should be tuned to TILE. K Search for a suitable value, which will TILE K Set to a size suitable for task allocation.

[0150] 2) Optional soft pipelining optimization techniques: To improve the parallelism of memory access computation, this invention uses a set of pipelining strategies to cover memory access latency through computation, such as... Figure 4 As shown, access to global memory, access to shared memory, and MatrixCore computation (referring to MFMA operations) can be executed in parallel, but data-related issues need to be considered. First, the data stored in shared memory must wait for the global memory access to finish before it can be retrieved. Then, since all threads within the same thread cooperate in retrieving data, and during the subsequent MFMA computation, each thread bundle not only needs to read its own data from global memory but also data retrieved by other thread bundles from global memory, synchronization is required within the thread block until all thread bundles have filled the shared memory with the data obtained from global memory, enabling data exchange and sharing between thread bundles. Finally, according to the requirements of the MFMA instructions, each piece of data is precisely placed into the register of the designated thread for computation. This strategy can, to some extent, cover memory access latency and improve parallelism.

[0151] Specifically, this invention uses the use of this technique as a tunable parameter because soft pipelining requires twice the shared memory space compared to implementations without this strategy. Since each thread block requires the same amount of shared memory, and the shared memory space on each Compute Unit (CU) is also limited, a single thread block occupying more shared memory space affects the actual number of schedulable thread blocks on the CU, thus reducing the program's unit utilization rate. The performance relationship between the unit utilization priority strategy and the soft pipelining strategy is uncertain, requiring tuning to determine which strategy to choose. The tuner can automatically generate code using soft pipelining and code not using soft pipelining, and then test their fitness to determine the program's performance under this optimization strategy.

[0152] 3) The use of the efficient Nc*HW_X data format. For NHWC format input, the IC dimension is often 1 to 2 orders of magnitude larger than FH and FW. This leads to a large span between the addresses fetched by adjacent threads and low memory access merging, reducing L2 cache hit rate. Figure 5 As shown in (a); for NCHW format input, because matrix M I The coordinates (h) of the points in the input graph i ,w i The relationship is not one-to-one. Directly using NCHW as the input format cannot efficiently extract data from the Input (and Kernel) tensors in 16-byte units. Figure 5As shown in (b).

[0153] In its actual implementation, this system uses the Nc*HW_X data format. Nc*HW_X is a widely used data format in implicit matrix multiplication convolution optimization. It extracts several X input channel dimensions as the innermost dimension, achieving continuity in data arrangement. Furthermore, this design uses the size of the X dimension as a tuning parameter to explore the optimal configuration.

[0154] 4) Efficient NCHW Data Format Conversion Method: This invention provides an efficient data format conversion method from NCHW to Nc*HW_X. For NCHW input, the process of converting it to Nc*HW_X format can be viewed as a multi-batch matrix transpose process, such as... Figure 6 As shown, firstly, the input tensor is reshaped to generate a batch of identically sized matrices M to be transposed. a Then, perform a matrix transpose on each of these batches of matrices to obtain a batch of... The matrix is ​​then reshaped to obtain the output tensor in NcHW_X format. Since tensors are arranged in row-major order in memory, the dimension reshaping operation is not actually required; the main transformation is the multi-batch matrix transpose process. The traditional method follows the pattern NCHW = N·(C / X)·X·HW = N·c*·X·HW, which is the matrix transpose of X·HW with a batch size of N·c*, as shown below. Figure 6 As shown in Figure ①, this method results in a small number of matrices requiring transpose for matrices with small H and W, making efficient data retrieval difficult in practical implementation and incurring significant format conversion overhead. The method provided in this invention is NCHW = N·X·(C / X)·HW = N·X·c*·HW, that is, the transpose of a matrix of size N with X·c*HW, as shown in Figure ①. Figure 6 As shown in Figure ②, extracting the (IC / X) dimension and appending it to the width of the matrix effectively alleviates the problem of a single matrix being too small, thus facilitating efficient implementation. Simultaneously performing the same transformation on both the Input tensor and the Kernel tensor ensures correct pairing when multiplying corresponding elements, thereby achieving correctness. Here, IC is used to correspond to... Figure 6 The notation in the text essentially refers to the same meaning as the parameter C.

[0155] 5) A Non-Padding Shared Memory Conflict Elimination Method and its Automation: This invention provides a non-padding shared memory conflict elimination method and its automation for implicit matrix multiplication and convolution in multi-core parallel architectures within kernel function code. Padding is a common method for eliminating shared memory conflicts, but it leads to wasted shared memory space and reduces the number of threads that can be executed in parallel on a computing unit, i.e., it reduces the program's unit occupancy. While adopting a non-padding strategy increases the factors to consider in the layout, the advantage is that shared memory space is not wasted, resulting in higher unit occupancy or the ability to cache more data. Shared memory operations (data exchange and sharing) can be divided into two parts: 1. Storing the data retrieved from global memory (temporarily stored in registers) into a specified location in shared memory; 2. Each thread retrieves the required data from the specified location in shared memory. The layout needs to ensure that neither of these processes results in shared memory conflicts, such as... Figure 7 As shown.

[0156] This module is divided into three sub-modules: input converter, controller, and tuner. The input converter module operates according to the following steps:

[0157] Step 1. Extract all 2D convolution parameters from the user input or the deep neural network and convert them into convolution descriptors. The parameters of the 2D convolution include the shape of the input tensor (N, IC, IH, IW), the shape of the convolution kernel tensor (OC, IC, KH, KW), and (N, OC, OH, OW), as well as the size of the stride and padding, and whether bias is enabled. Record these parameters in a list, which is a convolution descriptor that describes the basic information about this convolution operator.

[0158] Step 2. Perform deduplication on the collected convolution descriptors to avoid redundant optimization, and then generate a list of convolution descriptors to be optimized. If the basic information of convolution recorded in two convolution operator descriptors is exactly the same, this invention considers the two convolution descriptors to be duplicates. Deduplication is to eliminate this duplication, avoid redundant optimization, and save time.

[0159] The controller module is implemented and operated according to the following steps:

[0160] Step 3. The controller first reads the Log file into dictionary D. The Log file stores the historical records of convolution descriptors and their corresponding optimal optimization parameters. Simultaneously, an empty dictionary S is initialized to store the optimized convolution descriptors and their corresponding optimal parameter combinations.

[0161] Step 4. The controller reads the convolution descriptors to be optimized sequentially from the descriptor list and determines whether they are in dictionary D. If they are, it jumps to step 7; otherwise, it continues to determine whether the descriptor is in dictionary S. If it is, it executes step 4 again to retrieve the next descriptor; otherwise, it goes to step 5.

[0162] Step 5. The controller submits the convolution descriptor to the tuner for tuning to obtain the optimal combination of parameters.

[0163] Step 6. This descriptor and its corresponding combination of optimization parameters will be written into dictionary D. Proceed to step 8.

[0164] Step 7. Read the optimization parameters corresponding to the descriptor from D.

[0165] Step 8. Write this descriptor and its corresponding optimization parameters into dictionary S. Determine whether all convolution descriptors have been optimized. If so, proceed to step 9; otherwise, go to step 4.

[0166] Step 9. The controller numbers the descriptors in dictionary S and saves the correspondence between the numbers and descriptors to the Model.log file, while also saving dictionary D to the Log file. Dictionary S is then submitted to the code generator (code generation module).

[0167] The tuner module's process proceeds as follows: Figure 8 As shown:

[0168] Step 51. Initialize an empty fitness set dictionary D to record the fitness of tested optimal combinations, and randomly generate a population as the initial population. The population refers to the set of multiple generated "optimization parameter combinations". The population is a basic concept in classical genetic algorithms, and this invention adopts this basic concept.

[0169] Step 52. At the start of each round of optimization, first traverse the individuals in the population, and preliminarily check the legality of the configuration based on the hardware resource information of the neural network accelerator. For invalid configurations, the fitness is directly set to 0. The neural network accelerator can be a GPU, NPU, etc.

[0170] Step 53. For individuals in the population that pass the legitimacy check, first check whether the individual is in D. If it is, directly read its fitness. This avoids repeated testing and reduces testing time.

[0171] Step 54. For individuals in the population that have not acquired fitness, number and group these individuals to generate combinations of parameters to be tested.

[0172] Step 55. Pass the combination of parameters to be tested to the code generation module and call it to generate code.

[0173] Step 56. The generated code is handed over to the kernel function execution module for compilation and testing, and fitness information is returned.

[0174] Step 57. Update the dictionary D with the parameter combinations and fitness information for the new test.

[0175] Step 58. Select the parents for the offspring population using the optimal inheritance + "roulette wheel" selection method: the parameter with the highest fitness is directly used as a parent, and the remaining parents are selected by the "roulette wheel" method based on fitness. For example, if the fitness set of the current population is {1,2,3,4,5,6,7,8,9}, then the "child" with fitness of 9 (that is, an optimized parameter combination) has the highest fitness, and this invention directly uses it as a parent. Then, the remaining parents are selected from the remaining "children". Assuming a total of 3 parents are needed, one parent has already been decided, and the remaining two need to be selected. The selection method is the classic "roulette wheel" method: that is, the higher the fitness of the child, the greater the chance of the child being selected. The probability calculation formula is: (the fitness of this child) / (the sum of the fitness of all n children).

[0176] Step 59. Parents perform mutations and crossovers of the "genes" according to the pre-specified crossover and mutation rates to generate a new generation of population.

[0177] Step 510. After tuning to the specified number of rounds, perform a fitness test on the final population again (steps 52 to 56), obtain the individual parameters with the best fitness, and return them to the front-end module. Each iteration generates a series of "optimization scheme" combinations for the convolution descriptor to be tuned, and each "optimization scheme" is an optimization parameter combination. The optimization parameter combination is essentially a list, and each entry in the list corresponds to a tuning option in the optimization space.

[0178] Code generation module. This module is called by the tuner and controller, and is responsible for generating HIP code based on the received combination of optimization parameters and the code template. Furthermore, for multiple sets of input code, its corresponding executable code is numbered and packaged for submission to the kernel function execution module for compilation and testing. Figure 9 As shown.

[0179] The code generation module receives one or more convolution descriptors or groups of convolution descriptors, and a set of optimization parameters. If it is a single convolution descriptor, the set of optimization parameters is assumed to be specific to that descriptor; otherwise, there is a one-to-one correspondence between convolution descriptors and optimization parameter sets. This module is implemented and runs according to the following steps:

[0180] Step 101. For each convolution descriptor and its optimization parameters, first select a pre-written function template based on whether pipelining is used. The code generation rule for whether to use a soft pipelining version is determined by the soft pipelining entry in the optimization parameter combination; one of the optimization parameters in the combination determines whether soft pipelining is used.

[0181] Step 102. The function template contains several identifiers, and optimized code blocks need to be generated and added at the corresponding locations. The code generation module generates these code blocks based on the convolution parameters and optimization parameters in the convolution descriptor.

[0182] Step 103. Fill the code block into the corresponding identifier position in the template.

[0183] Step 104. After all the code has been generated, the code generation module generates an external wrapper function based on the number. This function calls the corresponding kernel function (the kernel running on the GPU) through the ID.

[0184] Step 105. Depending on the needs of the front-end module (controller or tuner), return the kernel function code set or write the code and wrapper into a HIP file.

[0185] The HIP file is compiled using the HIPCC compiler. The compiled program is then bound to the front-end API via an interface provided by the front-end framework for invocation. During invocation, the corresponding convolution parameters and their corresponding numbers are read from Model.log to launch the appropriate kernel function. This accelerator is applicable to AMD Instinct MI100 and AMD Instinct MI200 series accelerators, and also has the potential to be applied to AMD Instinct MI300 series accelerators and Hygon K100DCU accelerators.

[0186] Kernel function execution module. This module is called by the tuner and is responsible for compiling and executing the generated code. It also tests the performance of each kernel function using a testing framework, returns the corresponding fitness information by number, and summarizes it into a list to return to the tuner module, such as... Figure 10 As shown.

[0187] The kernel function execution module receives a set of kernel function IDs to be tested and a convolution descriptor as input. This module is implemented and run according to the following steps:

[0188] Step 561. Initialize an empty set F to store fitness information.

[0189] Step 562. Call the compilation script to compile the generated kernel function HIP code using HIPCC and link it into the test framework.

[0190] Step 563. Start the test program, input the kernel function ID to be tested and the input parameters of the convolution. The test framework will automatically test and return the fitness of each kernel function. Fitness represents the performance of the generated code based on this combination of optimization parameters when running on this hardware, which is a basic concept in genetic algorithms.

[0191] Step 564. The kernel function execution module reads the fitness information returned by the test program, writes each fitness into the position corresponding to the ID in F, and returns F to the tuner.

[0192] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0193] This invention also proposes a compilation optimization device for convolution operators in multi-core parallel architectures, including:

[0194] Module 1: Convert the input parameters of the convolution to be tuned into convolution descriptors;

[0195] Module 2: Perform deduplication on the collected convolution descriptors to generate a list of convolution descriptors;

[0196] Module 3: Read the Log file into dictionary D, which stores the history of convolution descriptors and their corresponding optimal parameter combinations; initialize dictionary S for storing convolution descriptors and their corresponding optimal parameter combinations.

[0197] Module 4: Read convolution descriptors sequentially from the descriptor list and determine whether they are in the dictionary D. If they are, execute Module 7; otherwise, determine whether they are in the dictionary S. If they are, execute Module 4 again to retrieve the next convolution descriptor; otherwise, execute Module 5.

[0198] Module 5: The controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the optimal combination of parameters;

[0199] Module 6: Write the convolution descriptor and its corresponding optimal parameter combination into dictionary D, and execute module 8;

[0200] Module 7: Read the optimal combination of parameters corresponding to the convolution descriptor from the dictionary D;

[0201] Module 8: Write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S. Determine if all convolution descriptors in the convolution descriptor list have been optimized. If so, execute Module 9; otherwise, execute Module 4.

[0202] Module 9: The controller numbers the convolution descriptors in the dictionary S and saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; it generates HIP code based on the dictionary S and the code template; by compiling the HIP code, an executable file is obtained, and the neural network accelerator runs the executable file based on the convolution descriptors and the Model.log file to obtain the convolution operation result.

[0203] The aforementioned convolution operator compilation optimization device for multi-core parallel architecture, wherein module 5 includes:

[0204] Module 51: Initialize the dictionary D used to record the fitness of optimal combinations, and initialize the population;

[0205] Module 52: Traverse the individuals in the population, check the configuration validity of the individual based on the hardware resource information of the backend hardware, and set the fitness of the individual to 0 directly for invalid configurations.

[0206] Module 53: Check if the individuals in the population that have passed the legality check are in the dictionary D. If they are, directly read their fitness; otherwise, execute module 54.

[0207] Module 54: Group the individuals in the population that have not yet acquired fitness by numbering them, and generate combinations of parameters to be tested;

[0208] Module 55, the code generation module generates kernel function code based on the combination of parameters to be tested; and the generation process includes determining whether to generate code using soft pipeline technology or not based on the tuning parameters;

[0209] This soft pipeline technology includes parallel execution of global memory access, shared memory access, and MatrixCore calculations; data stored in shared memory must wait for the global memory access to finish before it can be retrieved, and this continues until all thread bundles have filled the data obtained from global memory into shared memory, enabling data exchange and sharing between thread bundles; and according to the requirements of MFMA instructions, each piece of data is placed into the register of the specified thread.

[0210] This invention provides a method for eliminating shared memory conflicts in implicit matrix multiplication and convolution with no padding in a multi-core parallel architecture. The shared memory operation includes two processes: storing the data taken from global memory into a specified location in shared memory; and each thread taking the data it needs from the specified location in shared memory. The arrangement needs to ensure that no shared memory conflicts occur in either of these processes.

[0211] The implicit matrix multiplication convolution kernel generated by module 55 adopts the NC*HW_X format; this data format takes the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as an optimization parameter to explore the optimal configuration.

[0212] Specifically, module 55 converts the NCHW format to the Nc*HW_X format using the following method:

[0213] First, the input tensor is reshaped to generate a batch of identically sized matrices M to be transposed. a Then, transpose each of the batch matrices to obtain the batch of... The matrix is ​​then reshaped to obtain the output tensor in Nc*HW_X format.

[0214] The method for transposing each of the Batch matrices is NCHW=N·X·(C / X)·HW=N·X·c*·HW, which is the transpose of the matrix of batch=N with X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix.

[0215] Module 56: Hand over the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and return the fitness information of the parameter combination to be tested;

[0216] Module 57: Update the fitness information to dictionary D;

[0217] Module 58: Select the optimal parameter combination with the highest fitness in the current population as the parent generation, and select at least one optimal parameter combination from the other multiple optimal parameter combinations in the current population (excluding the optimal parameter combination with the highest fitness) as the parent generation. All parents perform mutation and crossover of the parameter combination genes according to the preset crossover rate and mutation rate to update the population. Module 52 is executed again until the optimization reaches the specified number of rounds. Then, the individual parameter with the best fitness in the dictionary D is selected as the best optimal parameter combination.

[0218] This module 56 includes:

[0219] Module 561: Initialize the set F used to store fitness information;

[0220] Module 562: Call the compilation script to compile and link the kernel function code into the test framework;

[0221] Module 563: Start the test program, input the kernel function code to be tested and the convolution descriptor into the test framework, test and return the fitness of each kernel function code;

[0222] Module 564: Write the measured fitness into the fitness information set F at the position corresponding to the kernel function code ID;

[0223] Module 9 generates HIP code based on the dictionary S and the code template, specifically including:

[0224] For a convolution descriptor and its optimized parameter combination, a preset function template is selected; this function template contains multiple identifiers; the code generator of the neural network accelerator generates a code block in the function template based on the convolution parameters in the convolution descriptor and the optimal optimized parameter combination; this code block is filled into the corresponding identifier position in the template to obtain the kernel function; the code generator generates an outer wrapper function based on the number, so as to call the corresponding kernel function by number; the HIP file is generated based on the outer wrapper function and the kernel function;

[0225] The tuner in module 5 is used to calculate the arithmetic strength AI of the operators in the convolution to be tuned using the following formula, and to determine the block strategy of the implicit matrix multiplication result matrix based on the arithmetic strength:

[0226]

[0227] In the formula TILE M and TILE N These represent the block sizes in the M and N directions of the resulting matrix, respectively; FH and FW are the kernel sizes, respectively; IC is the number of input channels; and a larger TILE is selected. M and TILE N Divide into blocks, and use TILE K Set to a size suitable for task allocation.

[0228] like Figure 11 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, including the convolution operator compilation and optimization device.

[0229] like Figure 12 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to acquire the input parameters of the convolution to be optimized, and the information display device D is used to display the convolution operation results obtained by the present invention.

[0230] Information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with specified key information, such as convolution operation results, Model.log files, and HIP code, based on user-defined display parameters and / or attributes. Display parameters could be, for example, the data range to be displayed, and display attributes could be, for example, the font, color, and whether scrolling is enabled. This allows the user to access this information more quickly without having to access secondary pages or scroll through pages, saving user effort. Alternatively, this information display mechanism can be an artificial intelligence (AI) display model, which can learn the user's key information based on previous usage habits, such as viewing time, number of clicks, and number of edits, and then automatically present the user with rich and necessary key information.

[0231] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multi-core parallel architecture convolution operator compilation optimization method provided by the above methods.

[0232] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that executes a convolution operator compilation optimization method for the multi-core parallel architecture. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0233] Figure 13 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0234] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0235] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0236] The computing unit I can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0237] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for compiling and optimizing convolution operators for multi-core parallel architectures, used in a compilation and optimization system for convolution operators in multi-core parallel architectures, the compilation and optimization system for convolution operators comprising: The code generation module, kernel function execution module, and automatic tuning module consisting of an input converter, a controller, and a tuner are characterized in that the convolution operator compilation optimization method includes: Step 1: Convert the input parameters of the convolution to be tuned into convolution descriptors; Step 2: Perform deduplication on the collected convolution descriptors to generate a list of convolution descriptors; Step 3: Read the Log file into dictionary D. The Log file stores the history of convolution descriptors and their corresponding optimal parameter combinations; initialize dictionary S to store convolution descriptors and their corresponding optimal parameter combinations. Step 4: Read the convolution descriptors sequentially from the descriptor list and determine whether they are in the dictionary D. If they are, proceed to step 7; otherwise, determine whether they are in the dictionary S. If they are, proceed to step 4 again to retrieve the next convolution descriptor; otherwise, proceed to step 5. Step 5: The controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the optimal combination of parameters; Step 6: Write the convolution descriptor and its corresponding optimal parameter combination into dictionary D, and then execute step 8; Step 7: Read the optimal combination of parameters corresponding to the convolution descriptor from the dictionary D; Step 8: Write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S. Determine if all convolution descriptors in the convolution descriptor list have been optimized. If so, proceed to step 9; otherwise, proceed to step 4. Step 9: The controller assigns numbers to the convolution descriptors in the dictionary S and saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; it generates HIP code based on the dictionary S and the code template; by compiling the HIP code, an executable file is obtained, and the neural network accelerator runs the executable file based on the convolution descriptors and the Model.log file to obtain the convolution operation result.

2. The compilation optimization method for convolution operators for multi-core parallel architectures as described in claim 1, characterized in that, Step 5 includes: Step 51: Initialize the dictionary D used to record the fitness of optimized combinations, and initialize the population; Step 52: Traverse the individuals in the population and check the configuration validity of the individual based on the hardware resource information of the backend hardware. For invalid configurations, set the fitness to 0 directly. Step 53: Check if the individuals in the population that passed the legality check are in the dictionary D. If they are, directly read their fitness; otherwise, proceed to step 54. Step 54: Group the individuals in the population that have not yet acquired fitness by number to generate the combination of parameters to be tested; Step 55: The code generation module generates kernel function code based on the combination of parameters to be tested; and the generation process includes determining whether to generate code using soft pipelining technology or not based on the tuning parameters. This soft pipeline technology includes parallel execution of global memory access, shared memory access, and Matrix Core calculations; data stored in shared memory must wait for the global memory access to finish before it can be retrieved, and this continues until all thread bundles have filled the data obtained from global memory into shared memory, enabling data exchange and sharing between thread bundles; and according to the requirements of MFMA instructions, each piece of data is placed into the register of the designated thread. The shared memory operation consists of two processes: first, retrieving data from global memory and storing it in a specified location in shared memory; and second, each thread retrieving the data it needs from a specified location in shared memory. The layout must ensure that neither of these processes will cause conflicts in the shared memory. The implicit matrix multiplication convolution kernel generated in step 55 adopts the NC*HW_X format; the NC*HW_X format takes the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as an optimization parameter to explore the optimal configuration. Specifically, step 55 converts the NCHW format to the NC*HW_X format using the following method: First, the input tensor is reshaped to generate a batch of matrices of the same size to be transposed. Then, transpose each of the batch matrices to obtain the batch of... The matrix is ​​then reshaped to obtain the output tensor in NC*HW_X format. The method for transposing each of the Batch matrices is NCHW = N·X·(C / X)·HW = N·X·C*·HW, which is the transpose of the matrix of batch = N with X·C*HW, extracting the (C / X) dimension and appending it to the width of the matrix. Step 56: Submit the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and return the fitness information of the parameter combination to be tested; Step 57: Update the fitness information to dictionary D; Step 58: Select the optimal parameter combination with the highest fitness in the current population as the parent generation, and select at least one optimal parameter combination from the other multiple optimal parameter combinations in the current population (excluding the optimal parameter combination with the highest fitness) as the parent generation using the roulette wheel algorithm. All parents perform mutation and crossover of the parameter combination genes according to the preset crossover rate and mutation rate to update the population. Repeat step 52 until the optimization reaches the specified number of rounds. Then select the individual parameter with the best fitness in the dictionary D as the best optimal parameter combination.

3. The compilation optimization method for convolution operators for multi-core parallel architectures as described in claim 2, characterized in that, Step 56 includes: Step 561: Initialize the set F used to store fitness information; Step 562: Call the compilation script to compile and link the kernel function code into the test framework; Step 563: Start the test program, input the kernel function code to be tested and the convolution descriptor into the test framework, test and return the fitness of each kernel function code; Step 564: Write the measured fitness into the position corresponding to the kernel function code ID in the fitness information set F.

4. The compilation optimization method for convolution operators for multi-core parallel architectures as described in any one of claims 1 to 3, characterized in that, Step 9, which generates HIP code based on the dictionary S and the code template, specifically includes: For a convolution descriptor and its optimized parameter combination, a preset function template is selected; this function template contains multiple identifiers; the code generator of the neural network accelerator generates a code block in the function template based on the convolution parameters in the convolution descriptor and the optimal optimized parameter combination; the code block is filled into the corresponding identifier position in the template to obtain the kernel function; the code generator generates an outer wrapper function based on the number, so as to call the corresponding kernel function by number; a HIP file is generated based on the outer wrapper function and the kernel function.

5. The compilation optimization method for convolution operators for multi-core parallel architectures as described in claim 1, characterized in that, In step 5, the tuner is used to calculate the arithmetic strength AI of the operators in the convolution to be tuned using the following formula, and to determine the block strategy of the implicit matrix multiplication result matrix based on the arithmetic strength: In the formula and These represent the block sizes in the M and N directions of the resulting matrix, respectively; FH and FW are the kernel sizes, respectively; and IC is the number of input channels. A larger value is selected. and Divide into blocks, Set to a size suitable for task allocation.

6. A compilation optimization device for convolution operators in a multi-core parallel architecture, characterized in that, include: Module 1: Convert the input parameters of the convolution to be tuned into convolution descriptors; Module 2: Perform deduplication on the collected convolution descriptors to generate a list of convolution descriptors; Module 3: Read the Log file into dictionary D, which stores the history of convolution descriptors and their corresponding optimal parameter combinations; initialize dictionary S for storing convolution descriptors and their corresponding optimal parameter combinations. Module 4: Read convolution descriptors sequentially from the descriptor list and determine whether they are in the dictionary D. If they are, execute Module 7; otherwise, determine whether they are in the dictionary S. If they are, execute Module 4 again to retrieve the next convolution descriptor; otherwise, execute Module 5. Module 5: The controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the optimal combination of parameters; Module 6: Write the convolution descriptor and its corresponding optimal parameter combination into dictionary D, and execute module 8; Module 7: Read the optimal combination of parameters corresponding to the convolution descriptor from the dictionary D; Module 8: Write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S. Determine if all convolution descriptors in the convolution descriptor list have been optimized. If so, execute Module 9; otherwise, execute Module 4. Module 9: The controller numbers the convolution descriptors in the dictionary S and saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; it generates HIP code based on the dictionary S and the code template; by compiling the HIP code, an executable file is obtained, and the neural network accelerator runs the executable file based on the convolution descriptors and the Model.log file to obtain the convolution operation result.

7. The convolution operator compilation optimization device for multi-core parallel architecture as described in claim 6, Its features are, in, Module 5 includes: Module 51: Initialize the dictionary D used to record the fitness of optimal combinations, and initialize the population; Module 52: Traverse the individuals in the population, check the configuration validity of the individual based on the hardware resource information of the backend hardware, and set the fitness of the individual to 0 directly for invalid configurations. Module 53: Check if the individuals in the population that have passed the legality check are in the dictionary D. If they are, directly read their fitness; otherwise, execute module 54. Module 54: Group the individuals in the population that have not yet acquired fitness by numbering them, and generate combinations of parameters to be tested; Module 55, the code generation module generates kernel function code based on the combination of parameters to be tested; and the generation process includes determining whether to generate code using soft pipeline technology or not based on the tuning parameters; This soft pipeline technology includes parallel execution of global memory access, shared memory access, and Matrix Core calculations; data stored in shared memory must wait for the global memory access to finish before it can be retrieved, and this continues until all thread bundles have filled the data obtained from global memory into shared memory, enabling data exchange and sharing between thread bundles; and according to the requirements of MFMA instructions, each piece of data is placed into the register of the designated thread. The shared memory operation consists of two processes: first, retrieving data from global memory and storing it in a specified location in shared memory; and second, each thread retrieving the data it needs from a specified location in shared memory. The layout must ensure that neither of these processes will cause conflicts in the shared memory. The implicit matrix multiplication convolution kernel generated by module 55 adopts the NC*HW_X format; the NC*HW_X format takes the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as an optimization parameter to explore the optimal configuration. Specifically, module 55 converts the NCHW format to the NC*HW_X format using the following method: First, the input tensor is reshaped to generate a batch of matrices of the same size to be transposed. Then, transpose each of the batch matrices to obtain the batch of... The matrix is ​​then reshaped to obtain the output tensor in NC*HW_X format. The method for transposing each of the Batch matrices is NCHW = N·X·(C / X)·HW = N·X·C*·HW, which is the transpose of the matrix of batch = N with X·C*HW, extracting the (C / X) dimension and appending it to the width of the matrix. Module 56: Hand over the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and return the fitness information of the parameter combination to be tested; Module 57: Update the fitness information to dictionary D; Module 58: Select the optimal parameter combination with the highest fitness in the current population as the parent generation, and select at least one optimal parameter combination from the other multiple optimal parameter combinations in the current population (excluding the optimal parameter combination with the highest fitness) as the parent generation. All parents perform mutation and crossover of the parameter combination genes according to the preset crossover rate and mutation rate to update the population. Module 52 is executed again until the optimization reaches the specified number of rounds. Then, select the individual parameter with the best fitness in the dictionary D as the best optimal parameter combination. This module 56 includes: Module 561: Initialize the set F used to store fitness information; Module 562: Call the compilation script to compile and link the kernel function code into the test framework; Module 563: Start the test program, input the kernel function code to be tested and the convolution descriptor into the test framework, test and return the fitness of each kernel function code; Module 564: Write the measured fitness into the fitness information set F at the position corresponding to the kernel function code ID; Module 9 generates HIP code based on the dictionary S and the code template, specifically including: For a convolution descriptor and its optimized parameter combination, a preset function template is selected; this function template contains multiple identifiers; the code generator of the neural network accelerator generates a code block in the function template based on the convolution parameters in the convolution descriptor and the optimal optimized parameter combination; this code block is filled into the corresponding identifier position in the template to obtain the kernel function; the code generator generates an outer wrapper function based on the number, so as to call the corresponding kernel function by number; a HIP file is generated based on the outer wrapper function and the kernel function; The tuner in module 5 is used to calculate the arithmetic strength AI of the operators in the convolution to be tuned using the following formula, and to determine the block strategy of the implicit matrix multiplication result matrix based on the arithmetic strength: In the formula and These represent the block sizes in the M and N directions of the resulting matrix, respectively; FH and FW are the kernel sizes, respectively; and IC is the number of input channels. A larger value is selected. and Divide into blocks, Set to a size suitable for task allocation.

8. An electronic device, characterized in that, The device includes a convolution operator compilation and optimization apparatus as described in claim 6 or 7, wherein the electronic device is connected to an information display device, which is used to display the convolution operation result with user-set display parameters, attributes, or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the convolution operator compilation optimization method according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the compilation optimization method for the convolution operator as described in any one of claims 1-5.

Images