Convolution operator compilation optimization method and device for multi-core parallel system structure

Abstract

The invention provides a convolution operator compilation optimization method and device for a multi-core parallel system structure. Comprising the following steps: an input converter module extracts convolution parameters in a deduplication model or input by a user, and generates a convolution descriptor group; the controller module determines whether to start tuning on the input module or not according to the historical record, and transmits the convolution descriptor needing tuning to the tuner module; the tuner module is responsible for automatically tuning operators according to the configured parameter space and outputting optimal parameter configuration; and the code generation module generates an HIP code according to the received tuning parameter combination and the convolution descriptor, and performs corresponding numbering on a kernel function generated by each parameter. And the kernel function execution module compiles and executes the generated codes, tests the performance of each kernel function through a test framework, returns corresponding fitness information according to numbers, summarizes the fitness information into a list and returns the list to the tuner module.

Description

Technical Field

[0001] The present invention relates to the field of computer parallel computing and deep learning operator compilation optimization, and in particular to a convolution operator compilation optimization method, device, electronic device, computer-readable storage medium and computer program product for a multi-core parallel architecture. Background Art

[0002] Convolutional Neural Network (CNN) is increasingly used due to its excellent adaptability in computer vision, digital image processing, and "vintage map" applications. Convolution, especially 2D convolution, is one of the most commonly used operators in CNN models, and its performance will have a direct impact on the training and reasoning efficiency of neural networks. At the same time, GPUs are widely used in deep learning computational acceleration due to their high parallelism, high-bandwidth memory access, and outstanding floating-point computing capabilities.

[0003] In order to further improve the running speed of computing tasks of deep learning neural network models such as CNN on GPU devices, many commercial high-performance operator libraries and compilers for deep learning regard the optimization of convolution operators as an important task. They are implemented by integrating a variety of manual underlying deep optimization operators, or by automatic code generation and parameter tuning. However, computing acceleration devices represented by AMD GPUs still do not have enough convolution operator optimization for such hardware. Moreover, the optimization effects of various optimization combinations for convolution operators with different input parameters and different hardware platforms are also different, and it is impossible to find an absolutely efficient universal convolution implementation.

[0004] In order to accelerate the calculation of convolution operators on GPUs, the mainstream method in the industry is to design and optimize the operator implementation for the dedicated acceleration unit of matrix multiply add (MMA) integrated on the GPU. At present, the main implementation methods of convolution operators include four categories: direct convolution algorithm, FFT convolution algorithm, convolution algorithm based on general matrix multiplication (GEMM) and Winograd convolution algorithm. Among them, the GEMM convolution algorithm has been proven to be an efficient implementation of the convolution operator for the matrix multiply add unit on the GPU.

[0005] Currently, MIOpen is a high-performance machine learning operator library for the ROCm software stack. MIOpen has built-in the above four convolution implementation technology routes, and has built a large number of manually optimized operator implementations for each implementation and supported hardware architecture. At the same time, it can search for the most efficient operator in its own operator library for matching according to the convolution parameters entered by the user. However, studies have shown that the convolution implementation of MIOpen is more than 30% lower than the peak performance of hardware, and there is still room for performance improvement. At the same time, a large number of methods in MIOpen are implemented in assembly language, which makes it difficult for users to customize operator fusion with other operators.

[0006] In addition, mainstream deep learning compiler optimizers represented by TVM and Triton also integrate the functions of convolution operator generation and automatic tuning, and realize the features supported across multiple hardware platforms. However, TVM has not yet been adapted for the MMA dedicated acceleration unit of the CDNA architecture of AMD GPU, and can only use scalar units for calculations. At the same time, the built-in code tuning logic of these compilers is mainly based on general code optimization strategies. Although compilers such as Triton have made special optimizations for matrix multiplication, the memory access and calculation mode of convolution are still somewhat different from matrix multiplication, and its optimization strategy does not fully match the characteristics of GEMM convolution calculation. Despite being fully tuned, the performance of the convolution operators generated by these optimizers is still far lower than the peak performance of hardware with the same precision.

[0007] There are a lot of research works on GEMM convolution. These works can alleviate the bottleneck of parallelism or memory access through various methods. However, most of these works use their customized efficient data format as the standard input and convolution kernel tensor organization form. However, most of the existing mainstream deep learning frameworks such as Pytorch and Tensorflow only support data in specific formats. For example, the order of data storage in memory is NCHW or NHWC standard format tensor input. NCHW is the conventional data format of 4D tensors in deep learning and the basic data format of the current mainstream deep learning framework. N represents Batch Size, C represents the number of channels, and H and W represent the height and width of each input image. The pure memory access process delay of the conversion from NCHW format to the customized data format will affect the latency of the high-performance operator and limit the overall performance of the operator. At the same time, many works use the 0 padding method to solve the bank conflict problem of shared memory access of computing units. However, this method will waste shared memory space, which will lead to a decrease in the number of thread blocks that can be scheduled on a single computing unit and reduce hardware parallelism. Summary of the invention

[0008] In view of the problem that the convolution operator in the multi-core parallel architecture cannot fully utilize the hardware computing power, and the shortcomings of the above-mentioned existing convolution operator high-performance optimization and implementation solutions, the present invention proposes a convolution operator compilation optimization method and system for the multi-core parallel architecture, including:

[0009] 1. Block strategy and tuning parameters for cache hierarchy design of multi-core parallel architecture: The calculation and memory access tasks of implicit matrix multiplication and convolution need to be divided layer by layer into GPU thread blocks and warps, while controlling the occupancy of resources such as shared memory and registers, and providing tuning parameters.

[0010] 2. Automatically selected soft pipelining technology: The soft pipelining strategy can improve the parallelism of memory access and computing under appropriate circumstances. At the same time, the tuner will automatically choose whether to use the strategy to prevent performance degradation caused by its inappropriate use.

[0011] 3. Efficient data format conversion technology: solve the problem of low conversion efficiency from NCHW input format to NC*HW_X efficient format. By changing the arrangement mode of tensors of different channels, efficient format conversion under various input size images is achieved. NC*HW_X is an efficient data format based on NCHW that puts together the data of X consecutive Channel dimensions for each point (same H, W) in each input image in the same batch; since the converted data format puts the data of X Channel dimensions together, the size of X can be adjusted to meet the high-bandwidth memory access requirements of the GPU, and multiple data can be read in at the same time to improve the memory access bandwidth.

[0012] 4. Use of adjustable and efficient data formats: For different load conditions, the size of the X dimension of the NC*HW_X format is used as an adjustable parameter to automatically search for configurations with high locality or high data reuse rate to improve memory access efficiency.

[0013] 5. A strategy for eliminating building conflicts without padding and automatic implementation for GPU shared memory: This method does not generate building conflicts and does not waste shared memory space, which is beneficial to improving unit occupancy (CUOccupancy). At the same time, a method for automatically arranging coordinate calculations is proposed and applied to the code generator.

[0014] 6. Rapid automatic tuning method for optimization parameters: Using genetic algorithms, by increasing the mutation rate and crossover rate, rapid iteration of the population is achieved, and an optimal inheritance mechanism is added on the basis of the "roulette" screening method to ensure the effective inheritance of the "gene" of the high fitness parameter group and achieve rapid convergence.

[0015] Specifically, in view of the deficiencies of the prior art, the present invention proposes a convolution operator compilation optimization method for a multi-core parallel architecture, which includes:

[0016] Step 1: Convert the input parameters of the convolution to be tuned into a convolution descriptor;

[0017] Step 2: Perform a deduplication operation on the collected convolution descriptors to generate a convolution descriptor list;

[0018] Step 3, read the log file into the dictionary D, which stores the historical records of the convolution descriptor and the corresponding best optimization parameter combination; initialize the dictionary S for storing the convolution descriptor and its corresponding best optimization parameter combination;

[0019] Step 4, read the convolution descriptors from the descriptor list in sequence, and determine whether they are in the dictionary D. If so, execute step 7, otherwise determine whether they are in the dictionary S. If so, execute step 4 again to continue to take the next convolution descriptor, otherwise execute step 5;

[0020] Step 5: The controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the best optimization parameter combination;

[0021] Step 6: Write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary D, and execute step 8;

[0022] Step 7: Read the best optimization parameter combination corresponding to the convolution descriptor from the dictionary D;

[0023] Step 8: write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S, and determine whether all convolution descriptors in the convolution descriptor list have been optimized. If so, execute step 9; otherwise, execute step 4.

[0024] Step 9, the controller numbers the convolution descriptors in the dictionary S, saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; generates a HIP code according to the dictionary S and the code template; obtains an execution file by compiling the HIP code, and the neural network accelerator runs the execution file according to the convolution descriptor and the Model.log file to obtain a convolution operation result.

[0025] The convolution operator compilation optimization method for multi-core parallel architecture, wherein step 5 comprises:

[0026] Step 51, initializing a dictionary D for recording the fitness of the optimized combination, and initializing the population;

[0027] Step 52: traverse the individuals in the population, check the legality of the configuration of the individual according to the hardware resource information of the backend hardware, and directly set the fitness of the illegal configuration to 0;

[0028] Step 53, check whether the individuals in the population that have passed the legality check are in the dictionary D, if so, directly read their fitness, otherwise execute step 54;

[0029] Step 54, grouping the individuals in the population that have not obtained fitness, and generating a parameter combination to be tested;

[0030] Step 55, the code generation module generates a kernel function code according to the parameter combination to be tested; and the generation process includes determining whether to generate a code using the soft pipelining technology or a code not using the soft pipelining technology according to the tuning parameters;

[0031] The soft pipelining technology includes parallel execution of global memory access, shared memory access and MatrixCore calculation; the data stored in the shared memory must wait for the global memory access to be completed before it can be obtained, and until all thread warps fill the data obtained from the global memory into the shared memory, data exchange and sharing between thread warps are performed; and according to the requirements of the MFMA instruction, each data is placed in the register of the specified thread;

[0032] The present invention provides a shared memory non-filled building conflict elimination method for implicit matrix multiplication and convolution of multi-core parallel architecture in shared memory, and the operation of shared memory includes two processes: storing the data taken from the global memory to the specified location in the shared memory; and each thread bundle takes out the data it needs from the specified location of the shared memory respectively; the arrangement needs to ensure that the two processes will not cause shared memory building conflicts;

[0033] The implicit matrix multiplication convolution kernel generated in step 55 adopts the NC*HW_X format; this data format uses the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as a tuning parameter to explore the optimal configuration;

[0034] Specifically, step 55 converts the NCHW format to the Nc*HW_X format by the following method:

[0035] First, by reshaping the input tensor, a batch of matrices M of the same size to be transposed are generated. a ; Then perform matrix transposition on the Batch matrices to obtain the Batch matrices Matrix; finally, reshape the dimension to get the output tensor in Nc*HW_X format;

[0036] The method of performing matrix transposition on each batch matrix is ​​NCHW = N·X·(C / X)·HW = N·X·c*·HW, that is, the matrix transposition of batch = N X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix;

[0037] Step 56: hand over the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and return the fitness information of the parameter combination to be tested;

[0038] Step 57: Update the fitness information into the dictionary D;

[0039] Step 58, select the optimization parameter combination with the highest fitness in the current population as the parent, and select at least one optimization parameter combination from the remaining multiple optimization parameter combinations in the current population except the optimization parameter combination with the highest fitness through the roulette algorithm as the parent, all parents perform mutation and crossover of parameter combination genes according to the preset crossover rate and mutation rate to update the population and execute step 52 again until the tuning reaches the specified number of rounds, and then select the individual parameter with the best fitness in the dictionary D as the best optimization parameter combination.

[0040] The convolution operator compilation optimization method for multi-core parallel architecture, wherein step 56 includes:

[0041] Step 561, initializing a fitness information set F for saving;

[0042] Step 562: Call the compilation script to compile and link the kernel function code into the test framework;

[0043] Step 563: start the test program, input the kernel function code to be tested and the convolution descriptor into the test framework, test and return the fitness of each kernel function code;

[0044] Step 564: Write the measured fitness into the position corresponding to the kernel function code ID in the fitness information set F.

[0045] The convolution operator compilation optimization method for multi-core parallel architecture, wherein the step 9 generates the HIP code according to the dictionary S and the code template, specifically includes:

[0046] For a convolution descriptor and its optimization parameter combination, a preset function template is selected; there are multiple identifiers in the function template; the code generator of the neural network accelerator generates a code block in the function template according to the convolution parameters in the convolution descriptor and the best optimization parameter combination; the code block is filled into the corresponding identifier position in the template to obtain a kernel function; the code generator generates a peripheral encapsulation function according to the number to call the corresponding kernel function through the number; and the HIP file is generated according to the peripheral encapsulation function and the kernel function.

[0047] The convolution operator compilation optimization method for multi-core parallel architecture, wherein the tuner in step 5 is used to calculate the arithmetic intensity AI of the operator in the convolution to be tuned by the following formula, and determine the result matrix block strategy of implicit matrix multiplication based on the arithmetic intensity:

[0048]

[0049] Where TILE M and TILE N Respectively represent the block size of the result matrix in the M and N directions; FH and FW are the convolution kernel sizes, IC is the number of input channels, and the larger TILE is selected. M and TILE N Divide into blocks and TILE K Set to a size appropriate for the task assignment.

[0050] The present invention also proposes a convolution operator compilation optimization device for a multi-core parallel architecture, which includes:

[0051] Module 1: Convert the input parameters of the convolution to be tuned into convolution descriptors;

[0052] Module 2: Perform deduplication operation on the collected convolution descriptors to generate a convolution descriptor list;

[0053] Module 3, read the log file into the dictionary D, the log file saves the historical record of the convolution descriptor and the corresponding best optimization parameter combination; initialize the dictionary S for saving the convolution descriptor and its corresponding best optimization parameter combination;

[0054] Module 4, read the convolution descriptors from the descriptor list in sequence, and determine whether they are in the dictionary D. If so, execute module 7. Otherwise, determine whether they are in the dictionary S. If so, execute module 4 again to continue to take the next convolution descriptor. Otherwise, execute module 5.

[0055] Module 5, the controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the best optimization parameter combination;

[0056] Module 6: write the convolution descriptor and its corresponding optimal optimization parameter combination into dictionary D, and execute module 8;

[0057] Module 7: reading the best optimization parameter combination corresponding to the convolution descriptor from the dictionary D;

[0058] Module 8, write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S, and determine whether all convolution descriptors in the convolution descriptor list are optimized. If yes, execute module 9, otherwise execute module 4;

[0059] Module 9, the controller numbers the convolution descriptors in the dictionary S, saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; generates a HIP code according to the dictionary S and the code template; obtains an execution file by compiling the HIP code, and the neural network accelerator runs the execution file according to the convolution descriptor and the Model.log file to obtain a convolution operation result.

[0060] The convolution operator compilation optimization device for multi-core parallel architecture, wherein the module 5 includes:

[0061] Module 51, initializing a dictionary D for recording the fitness of the optimized combination, and initializing the population;

[0062] Module 52, traverse the individuals in the population, check the legality of the configuration of the individual according to the hardware resource information of the backend hardware, and directly set the fitness of the illegal configuration to 0;

[0063] Module 53, check whether the individuals in the population that have passed the legality check are in the dictionary D, if so, directly read their fitness, otherwise execute module 54;

[0064] Module 54, grouping the individuals in the population that have not obtained fitness, and generating a parameter combination to be tested;

[0065] Module 55, a code generation module generates a kernel function code according to the parameter combination to be tested; and the generation process includes determining whether to generate a code using the soft pipelining technology or a code not using the soft pipelining technology according to the tuning parameters;

[0066] The soft pipelining technology includes parallel execution of global memory access, shared memory access and MatrixCore calculation; the data stored in the shared memory must wait for the global memory access to be completed before it can be obtained, and until all thread warps fill the data obtained from the global memory into the shared memory, data exchange and sharing between thread warps are performed; and according to the requirements of the MFMA instruction, each data is placed in the register of the specified thread;

[0067] The present invention provides a shared memory non-filled building conflict elimination method for implicit matrix multiplication and convolution of multi-core parallel architecture in shared memory, and the operation of shared memory includes two processes: storing the data taken from the global memory to the specified location in the shared memory; and each thread bundle takes out the data it needs from the specified location of the shared memory respectively; the arrangement needs to ensure that the two processes will not cause shared memory building conflicts;

[0068] The implicit matrix multiplication convolution kernel generated by module 55 adopts the NC*HW_X format; this data format uses the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as a tuning parameter to explore the optimal configuration;

[0069] Specifically, the module 55 converts the NCHW format to the Nc*HW_X format by the following method:

[0070] First, by reshaping the input tensor, a batch of matrices M of the same size to be transposed are generated. a ; Then perform matrix transposition on the Batch matrices to obtain the Batch matrices Matrix; finally, reshape the dimension to get the output tensor in Nc*HW_X format;

[0071] The method of performing matrix transposition on each batch matrix is ​​NCHW = N·X·(C / X)·HW = N·X·c*·HW, that is, the matrix transposition of batch = N X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix;

[0072] Module 56: delivering the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and returning fitness information of the parameter combination to be tested;

[0073] Module 57, updating the fitness information into the dictionary D;

[0074] Module 58, select the optimization parameter combination with the highest fitness in the current population as the parent, and select at least one optimization parameter combination from the remaining multiple optimization parameter combinations in the current population except the optimization parameter combination with the highest fitness through the roulette algorithm as the parent, all parents perform mutation and crossover of parameter combination genes according to the preset crossover rate and mutation rate to update the population and execute module 52 again, until the tuning reaches the specified number of rounds, and select the individual parameter with the best fitness in the dictionary D as the best optimization parameter combination;

[0075] The module 56 comprises:

[0076] Module 561, initialization for saving fitness information set F;

[0077] Module 562, calling the compilation script, compiling and linking the kernel function code into the test framework;

[0078] Module 563, start the test program, input the kernel function code to be tested and the convolution descriptor to the test framework, test and return the fitness of each kernel function code;

[0079] Module 564, write the measured fitness into the position corresponding to the kernel function code ID in the fitness information set F;

[0080] The module 9 generates a HIP code according to the dictionary S and the code template, specifically including:

[0081] For a convolution descriptor and its optimization parameter combination, a preset function template is selected; there are multiple identifiers in the function template; the code generator of the neural network accelerator generates a code block in the function template according to the convolution parameters in the convolution descriptor and the best optimization parameter combination; the code block is filled into the corresponding identifier position in the template to obtain a kernel function; the code generator generates a peripheral encapsulation function according to the number to call the corresponding kernel function through the number; the HIP file is generated according to the peripheral encapsulation function and the kernel function;

[0082] The tuner in module 5 is used to calculate the arithmetic strength AI of the operator in the convolution to be tuned by the following formula, and determine the result matrix block strategy of implicit matrix multiplication based on the arithmetic strength:

[0083]

[0084] Where TILE M and TILE N Respectively represent the block size of the result matrix in the M and N directions; FH and FW are the convolution kernel sizes, IC is the number of input channels, and the larger TILE is selected. M and TILE N Divide into blocks and TILE K Set to a size appropriate for the task assignment.

[0085] The present invention also proposes an electronic device, which includes the convolution operator compilation optimization device, and the electronic device is connected to an information display device, which is used to display the evaluation results with display parameters and attributes set by the user or through an artificial intelligence model.

[0086] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the convolution operator compilation optimization method are implemented.

[0087] The present invention also proposes a computer program product, comprising a computer program, wherein the computer program implements the steps of the convolution operator compilation optimization method when executed by a processor.

[0088] It can be seen from the above scheme that the advantages of the present invention are:

[0089] 1) Compilation optimization based on multi-core parallel architecture characteristics: This system is developed based on the CDNA architecture and incorporates a variety of memory access and parallel optimization methods based on the architecture characteristics of the platform hardware;

[0090] 2) Highly scalable code generation and automatic tuning: It can automatically generate and select appropriate tuning parameter configurations based on different inputs, and provide the optimal convolution code implementation under the convolution input. The generated kernel function body is mostly HIP C++ code, which is convenient for users to expand and integrate other operators;

[0091] 3) A solution without padding to eliminate shared memory building conflicts is used: This method not only eliminates building conflicts, but also maximizes the use of shared memory space, further improving the memory access efficiency and parallelism of the hardware;

[0092] 4) Efficient NCHW format conversion method: This method designs a new format conversion method to solve the difficulty of efficient format conversion when NCHW format input is applied to implicit matrix multiplication and convolution, and can still perform efficient format conversion under small-size input graphs;

[0093] 5) Improved system resource utilization, reduced execution time and operator tuning time for deep learning reasoning tasks: For multiple common CNN network models, compared with the manufacturer's function operator library, the method provided by the present invention is on par with or exceeds its performance, and can achieve a maximum reasoning performance acceleration ratio of 1.23×; compared with typical AI compilers, this method can achieve a single operator performance acceleration ratio of 1.56× to 3.29×, and a tuning speed acceleration ratio of 22.0× to 38.5×. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 Compile and optimize the system module structure diagram for the convolution operator for multi-core parallel architecture;

[0095] Figure 2 Optimizing the system workflow for convolution operator compilation for multi-core parallel architectures;

[0096] Figure 3 This is a graph showing the effect of block size on the arithmetic strength of the convolution operator;

[0097] Figure 4 It is a schematic diagram of pipeline stage division;

[0098] Figure 5 Memory access discontinuity problem diagram for NHWC and NCHW input formats;

[0099] Figure 6 The schematic diagram of the NCHW to Nc*HW_X tensor conversion;

[0100] Figure 7It is a diagram for building conflict determination and shared memory layout;

[0101] Figure 8 This is the workflow diagram of the tuner;

[0102] Fig. 9 Generate module workflow diagram for code;

[0103] Fig.10 It is the workflow diagram of kernel function execution module;

[0104] Fig.11 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0105] Fig.12 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0106] Fig.13 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0107] Reference numerals:

[0108] A-First electronic device;

[0109] B-convolution operator compilation optimization device;

[0110] C-data acquisition equipment;

[0111] D-information display device;

[0112] 1000 - second electronic device;

[0113] Ⅰ-computational unit;

[0114] II-ROM;

[0115] III-RAM;

[0116] IV-bus;

[0117] V-interface;

[0118] VI - input unit;

[0119] VII-output unit;

[0120] VIII- Storage medium;

[0121] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0122] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0123] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0124] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0125] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0126] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0127] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0128] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0129] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0130] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0131] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0132] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0133] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0136] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0137] Based on the above-mentioned characteristic convolution operator optimization method, the present invention provides a convolution operator compilation optimization system for multi-core parallel architecture, such as Figure 1 As shown, the core parts include:

[0138] 1. Automatic tuning module, responsible for the overall control of the system and running the automatic tuning algorithm. This module consists of three sub-modules: the input converter module is responsible for extracting and deduplicating the convolution parameters in the model or input by the user, and generating a convolution descriptor group; the controller module decides whether to start tuning the input module based on the historical records, and passes the convolution descriptors that need to be tuned to the tuner module; the tuner module is responsible for automatically tuning the operator according to the configured parameter space and outputting the optimal parameter configuration;

[0139] 2. Code generation module: It is responsible for generating HIP codes according to the received multiple sets of tuning parameter combinations and convolution descriptors, and numbering the kernel functions generated by each parameter.

[0140] 3. Kernel function execution module. Responsible for compiling and executing the generated code, testing the performance of each kernel function through the test framework, returning the corresponding fitness information by number and summarizing it into a list and returning it to the tuner module.

[0141] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0142] The present invention provides a convolution operator compilation optimization method and system for multi-core parallel architecture. The purpose is to develop a high-performance convolution operator automatic compilation optimizer for multi-core parallel architecture, combine the relevant hardware characteristics of the relevant platform, use a variety of optimization methods, improve the performance of the convolution single operator, and accelerate the reasoning of deep neural networks.

[0143] The following will describe and explain the working steps of the compiler optimizer system in the present invention, the implementation principles of the functions of each module, and the characteristics of the operator optimization implementation method in conjunction with the accompanying drawings.

[0144] like Figure 1 As shown, the implementation of the system in the present invention includes three modules, an automatic tuning module, a code generation module and a kernel function execution module. Each module includes several submodules. Figure 2 This is the workflow of the compiler optimizer system of the present invention when the method provided by the present invention is adopted.

[0145] Automatic tuning module. This module consists of three parts: input converter, controller and tuner. This module controls the operation of the entire system. The input converter processes the 2D convolution operator parameters input by the user or extracted from the model and submits them to the controller. The controller decides whether to start tuning for the convolution descriptor and hands the descriptor to be tuned to the tuner. The tuner runs the tuning routine, iteratively tunes the input parameters, obtains the optimal parameters, and finally generates executable code by the code generation module.

[0146] For the automatic tuning module, the present invention provides a method for parallel optimization of convolution operators and memory access optimization for multi-core parallel architecture. The features of this module are as follows: Figures 3 to 7 As shown, there are five specific features:

[0147] 1) The tuner performs matrix partitioning strategy analysis based on arithmetic intensity: According to the theory of the Roofline model, the theoretical peak performance that an operator can achieve is affected by its arithmetic intensity. Too low arithmetic intensity will lead to limited memory access, affecting the theoretical peak performance that the program can achieve. The estimation formula for arithmetic intensity can be given as:

[0148]

[0149] TILE M and TILE N Respectively represent the block size in the M and N directions of the result matrix of the implicit GEMM. The arithmetic strength is only related to TILE M and TILE N Related, and TILE M and TILE N The larger the value, the higher the arithmetic intensity. At the same time, for the same block strategy, the larger the size of the convolution kernel (FH and FW), the higher the number of input channels IC, and the corresponding arithmetic intensity will also be greater, such as Figure 3 However, too large a block will occupy more on-chip resources (registers, shared memory, etc.), resulting in reduced occupancy; and when the result matrix is ​​small, an overly large block strategy will cause the score block matrix itself to exceed the boundary of the result matrix, resulting in meaningless redundant tasks. Therefore, for a given convolution task, an efficient strategy tends to choose a larger TILE within a reasonable range as much as possible. M and TILE N Divide into blocks, and due to the arithmetic intensity and the block size TILE in the K direction of the result matrix K It is irrelevant, so the tuner should be used to set the TILE K Search for a suitable value, which will TILE K Set to a size appropriate for the task assignment.

[0150] 2) Optional soft pipelining optimization technology: In order to improve the parallelism of memory access calculations and cover the memory access delay through calculations, the present invention uses a set of pipelining strategies, such as Figure 4 As shown. Access to global memory, access to shared memory and MatrixCore calculation (referring to MFMA operation) can be executed in parallel, but data-related issues need to be considered. First, the number stored in shared memory must wait for the global memory access to end before it can be obtained. Then, since all threads in the same thread cooperate to obtain data, when performing MFMA calculation in the next step, each thread bundle must not only read the data it obtains from the global memory, but also read the data obtained from the global memory by other thread bundles. Therefore, the thread bundles in the thread block need to be synchronized until all thread bundles fill the data obtained from the global memory into the shared memory, and exchange and share data between thread bundles. Finally, according to the requirements of the MFMA instruction, each data is accurately placed in the register of the specified thread for calculation. This strategy can cover the memory access delay to a certain extent and improve the degree of parallelism.

[0151] In particular, the present invention uses this technology as a tunable parameter, the reason being that compared to the implementation without using this strategy, the soft pipelining technology needs to occupy double the shared memory space. Since the size of the shared memory space required for each thread block is the same, and the size of the shared memory on each computing unit (Compute Unit, CU) is also limited, a single thread block occupies more shared memory space, which will affect the actual number of thread blocks that can be scheduled on the computing unit, that is, reduce the unit occupancy rate of the program. The performance intensity relationship between the unit occupancy priority strategy and the soft pipelining strategy is uncertain, and it is necessary to determine which strategy to select through tuning. Among them, the tuner can automatically generate code using soft pipelining and code not using soft pipelining, and then test its fitness to determine how the program performs under this optimization strategy.

[0152] 3) Application of Nc*HW_X efficient data format. For NHWC format input, the size of the IC dimension is often 1 to 2 orders of magnitude larger than FH and FW. At this time, when fetching data, it will cause the span of adjacent thread fetch addresses to be large and the memory access merging degree to be low, reducing the L2Cache hit rate. Figure 5 (a) As shown; for NCHW format input, because the matrix M I The coordinates of the points in the input graph (h i ,w i ) is not a one-to-one correspondence. Directly using NCHW as the input format cannot efficiently retrieve the Input (and Kernel) tensor in 16-Byte units, such as Figure 5(b) as shown.

[0153] This system uses the Nc*HW_X data format in actual implementation. Nc*HW_X is a data format widely used in implicit matrix multiplication and convolution optimization. It proposes several X input channel dimensions as the innermost dimension to achieve continuity in data arrangement. At the same time, this design uses the size of the X dimension as a tuning parameter to explore the optimal configuration.

[0154] 4) NCHW efficient data format conversion method: The present invention provides an efficient data format conversion method from NCHW to Nc*HW_X. For NCHW input, the process of converting it into Nc*HW_X format can be regarded as a multi-batch matrix transposition process, such as Figure 6 As shown. First, by reshaping the input tensor, a batch of matrices M of the same size to be transposed are generated. a ; Then perform matrix transposition (Batched MatrixTranspose) on the Batch matrices to obtain Batch Matrix; Finally, reshape the dimensions to get the output tensor in NcHW_X format. Since tensors are arranged in row-major order in memory, the dimension reshaping operation does not actually need to be performed. The main conversion is the multi-batch matrix transposition process. The traditional method is NCHW = N·(C / X)·X·HW = N·c*·X·HW, that is, batch = N·c* X·HW matrix transposition, such as Figure 6 As shown in ①, for matrices with smaller H and W, the single matrix to be transposed is too small, which makes it difficult to obtain data efficiently in actual implementation and brings about a large format conversion overhead. The method provided by the present invention is NCHW=N·X·(C / X)·HW=N·X·c*·HW, that is, the transposition of the matrix of X·c*HW of batch=N, as shown in Figure 6 As shown in ②, the (IC / X) dimension is extracted and added to the width of the matrix, which effectively alleviates the problem of a single matrix being too small and facilitates efficient implementation. At the same time, the same transformation is performed on the Input tensor and the Kernel tensor to ensure that the corresponding elements are correctly paired when multiplied, thus achieving correctness. IC is for the corresponding Figure 6 The writing in essentially refers to the same meaning as parameter C.

[0155] 5) A method for eliminating building conflicts in shared memory without padding and an automated method thereof: The present invention provides a method for eliminating building conflicts in shared memory without padding and an automated method thereof for a multi-core parallel architecture in a kernel function code. Padding is a commonly used means of eliminating building conflicts, but it will lead to a waste of shared memory space and reduce the number of thread bundles that can be executed in parallel on a computing unit, that is, reduce the unit occupancy rate of the program. Although adopting a no-padding strategy will increase the factors to be considered in the layout, the advantage is that the space of the shared memory will not be wasted, and a higher unit occupancy rate can be obtained, or more data can be cached. The operations on shared memory (data exchange and sharing) can be divided into two parts: 1. Storing the numbers taken from the global memory (temporarily stored in the register) in the specified location in the shared memory; 2. Each thread bundle takes out the data it needs from the specified location of the shared memory. The layout needs to ensure that there will be no shared memory building conflicts in these two processes, such as Figure 7 shown.

[0156] This module is divided into three submodules: input converter, controller and tuner. The input converter module is implemented and operated according to the following steps:

[0157] Step 1. Extract all 2D convolution parameters from user input or deep neural network and convert them into convolution descriptors; the parameters of 2D convolution include the input tensor shape (N, IC, IH, IW), the shape of the convolution kernel tensor (OC, IC, KH, KW), and (N, OC, OH, OW), as well as the size of stride and padding, and whether bias is enabled; record these parameters into a list, which is a convolution descriptor that describes the basic information about the convolution operator;

[0158] Step 2. Perform deduplication operation on the collected convolution descriptors to avoid repeated optimization, and then generate a list of convolution descriptors to be optimized; if the basic information of the convolution recorded in two convolution operator descriptors is exactly the same, the present invention considers that the two convolution descriptors are repeated, and deduplication is to eliminate this duplication, avoid repeated optimization, and save time.

[0159] The controller module operates as follows:

[0160] Step 3. The controller first reads the log file into the dictionary D. The log file stores the historical records of the convolution descriptors and the corresponding optimal optimization parameters. At the same time, an empty dictionary S is initialized to store the optimized convolution descriptors and their corresponding optimal optimization parameter combinations.

[0161] Step 4. The controller reads the convolution descriptors to be optimized from the descriptor list in turn, and determines whether they are in the dictionary D. If so, jump to step 7, otherwise continue to determine whether the descriptor is in the dictionary S. If so, execute step 4 again to continue to take the next descriptor, otherwise go to step 5.

[0162] Step 5. The controller submits the convolution descriptor to the tuner for tuning to obtain the best optimization parameter combination.

[0163] Step 6. This descriptor and its corresponding optimization parameter combination will be written into dictionary D, and go to step 8.

[0164] Step 7. Read the optimization parameters corresponding to the descriptor from D.

[0165] Step 8. Write this descriptor and its corresponding optimization parameter combination into the dictionary S, and determine whether all convolution descriptors have been optimized. If so, execute step 9, otherwise go to step 4.

[0166] Step 9. The controller numbers the descriptors in the dictionary S, and saves the correspondence between the numbers and the descriptors in the Model.log file, and saves the dictionary D in the Log file. Submit the dictionary S to the code generator (code generation module).

[0167] The process of the tuner module is as follows: Figure 8 As shown:

[0168] Step 51. Initialize an empty fitness set dictionary D to record the fitness of the tested optimization combination, and randomly generate a population as the initialization population. The population refers to a collection of multiple "optimization parameter combinations" generated. The population is a basic concept in the classical genetic algorithm, and the present invention follows this basic concept.

[0169] Step 52. At the beginning of each round of tuning, first traverse the individuals in the population, preliminarily check the legality of the configuration based on the hardware resource information of the neural network accelerator, and directly set the fitness of illegal configurations to 0. The neural network accelerator can be a GPU, NPU, etc.

[0170] Step 53. For the individuals in the population that pass the legitimacy check, first check whether the individual is in D. If so, directly read its fitness, which can avoid repeated detection and reduce testing time.

[0171] Step 54. For the individuals in the population that have not obtained fitness, these individuals are numbered and grouped to generate a parameter combination to be tested.

[0172] Step 55: Pass the parameter combination to be tested to the code generation module, and call it to generate code.

[0173] Step 56: Send the generated code to the kernel function execution module for compilation and testing, and return the fitness information.

[0174] Step 57: For the newly tested parameter combination and its fitness information, update it into the dictionary D.

[0175] Step 58. Use the method of optimal inheritance + "roulette" selection to select the parents of the sub-population: the parameter with the highest fitness is directly used as a parent, and the remaining parents are selected by the "roulette" method according to fitness. For example, if the fitness set of the current population is {1, 2, 3, 4, 5, 6, 7, 8, 9}, then the "child" with a fitness corresponding to 9 (that is, an optimized parameter combination) has the highest fitness. The present invention directly uses it as a parent, and then selects the remaining parents from the remaining "children". Assuming that there are 3 parents in total, then one parent has been determined, and the remaining two must be selected at this time. The selection method is the classic "roulette" method: that is, the higher the fitness of the child, the greater the chance of being selected, and the probability calculation formula is: (the fitness of this child) / (the sum of the fitness of all n children).

[0176] Step 59. The parents perform mutation and crossover of the parameter combination "gene" according to the pre-specified crossover rate and mutation rate to generate a new generation of population.

[0177] Step 510. After the specified number of rounds of tuning is reached, the fitness test process is performed again on the final population (steps 52 to 56), and the individual parameters with the best fitness are obtained and returned to the front-end module. Each iteration generates a series of "optimization scheme" combinations for the convolution descriptor to be tuned, and each "optimization scheme" is an optimization parameter combination. The optimization parameter combination is essentially a list, and each entry in the list corresponds to a tuning option in the optimization space.

[0178] Code generation module. This module is called by the tuner and controller and is responsible for generating HIP code according to the received optimization parameter combination and code template. In addition, for multiple sets of input codes, their corresponding executable codes will be numbered and packaged so that they can be submitted to the kernel function execution module for compilation and testing. Fig. 9 shown.

[0179] The code generation module receives a convolution descriptor or multiple convolution descriptor groups, and a set of optimization parameter combinations. If it is a convolution descriptor, then by default, this set of optimization parameter combinations is for this convolution descriptor, otherwise the convolution descriptor and the optimization parameter group correspond one to one. This module is implemented and operated according to the following steps:

[0180] Step 101. For each set of convolution descriptors and their optimization parameters, first select a pre-written function template based on whether to use pipeline technology. Whether to use the soft pipeline version of the code generation rule is determined by the soft pipeline entry in the optimization parameter combination. There is an optimization parameter in the optimization parameter combination that determines whether to use soft pipeline.

[0181] Step 102: There are several identifiers in the function template, and optimization code blocks need to be generated and added in the corresponding positions. The code generation module generates these code blocks according to the convolution parameters and optimization parameters in the convolution descriptor.

[0182] Step 103: Fill the code block into the corresponding identifier position in the template.

[0183] Step 104. After all codes are generated, the code generation module generates a peripheral encapsulation function according to the number, and the function calls the corresponding kernel function (kernel running on the GPU) through the id.

[0184] Step 105. According to the needs of the front-end module (controller or tuner), return the kernel function code set or write the code and encapsulation into a HIP file.

[0185] The HIP file is compiled by the HIPCC compiler, and the compiled program is bound to the front-end API through the interface provided by the front-end framework for calling. When calling, the corresponding information of the convolution parameters and numbers in Model.log will be read to start the corresponding kernel function. This accelerator can be applied to AMD Instinct MI100 accelerator, AMD InstinctMI200 series accelerator, and has the potential to be applied to AMD Instinct MI300 series accelerator and Haiguang K100DCU accelerator.

[0186] Kernel function execution module. This module is called by the tuner and is responsible for compiling and executing the generated code, testing the performance of each kernel function through the test framework, returning the corresponding fitness information by number and summarizing it as a list and returning it to the tuner module, such as Fig.10 shown.

[0187] The kernel function execution module receives the kernel function ID set to be tested and the convolution descriptor as input. This module is implemented and runs according to the following steps:

[0188] Step 561. Initialize an empty set F for storing fitness information.

[0189] Step 562: Call the compilation script, compile the generated kernel function HIP code using HIPCC and link it to the test framework.

[0190] Step 563. Start the test program, input the kernel function ID to be tested and the input parameters of the convolution, and the test framework will automatically test and return the fitness of each kernel function. The fitness represents the performance of the generated code generated according to this optimization parameter combination when running on the hardware, which is a basic concept in genetic algorithms.

[0191] Step 564. The kernel function execution module reads the fitness information returned by the test program, writes each fitness into the position corresponding to the ID in F, and returns F to the tuner.

[0192] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0193] The present invention also proposes a convolution operator compilation optimization device for a multi-core parallel architecture, which includes:

[0194] Module 1: Convert the input parameters of the convolution to be tuned into convolution descriptors;

[0195] Module 2: Perform deduplication operation on the collected convolution descriptors to generate a convolution descriptor list;

[0196] Module 3, read the log file into the dictionary D, the log file saves the historical record of the convolution descriptor and the corresponding best optimization parameter combination; initialize the dictionary S for saving the convolution descriptor and its corresponding best optimization parameter combination;

[0197] Module 4, read the convolution descriptors from the descriptor list in sequence, and determine whether they are in the dictionary D. If so, execute module 7. Otherwise, determine whether they are in the dictionary S. If so, execute module 4 again to continue to take the next convolution descriptor. Otherwise, execute module 5.

[0198] Module 5, the controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the best optimization parameter combination;

[0199] Module 6, write the convolution descriptor and its corresponding optimal optimization parameter combination into dictionary D, and execute module 8;

[0200] Module 7, reading the best optimization parameter combination corresponding to the convolution descriptor from the dictionary D;

[0201] Module 8, write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S, and determine whether all convolution descriptors in the convolution descriptor list are optimized. If yes, execute module 9, otherwise execute module 4;

[0202] Module 9, the controller numbers the convolution descriptors in the dictionary S, saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; generates a HIP code according to the dictionary S and the code template; obtains an execution file by compiling the HIP code, and the neural network accelerator runs the execution file according to the convolution descriptor and the Model.log file to obtain a convolution operation result.

[0203] The convolution operator compilation optimization device for multi-core parallel architecture, wherein the module 5 includes:

[0204] Module 51, initializing a dictionary D for recording the fitness of the optimized combination, and initializing the population;

[0205] Module 52, traverse the individuals in the population, check the legality of the configuration of the individual according to the hardware resource information of the backend hardware, and directly set the fitness of the illegal configuration to 0;

[0206] Module 53, check whether the individuals in the population that have passed the legality check are in the dictionary D, if so, directly read their fitness, otherwise execute module 54;

[0207] Module 54, grouping the individuals in the population that have not obtained fitness, and generating a parameter combination to be tested;

[0208] Module 55, a code generation module generates a kernel function code according to the parameter combination to be tested; and the generation process includes determining whether to generate a code using the soft pipelining technology or a code not using the soft pipelining technology according to the tuning parameters;

[0209] The soft pipelining technology includes parallel execution of global memory access, shared memory access and MatrixCore calculation; the data stored in the shared memory must wait for the global memory access to be completed before it can be obtained, and until all thread warps fill the data obtained from the global memory into the shared memory, data exchange and sharing between thread warps are performed; and according to the requirements of the MFMA instruction, each data is placed in the register of the specified thread;

[0210] The present invention provides a shared memory non-filled building conflict elimination method for implicit matrix multiplication and convolution of multi-core parallel architecture in shared memory, and the operation of shared memory includes two processes: storing the data taken from the global memory to the specified location in the shared memory; and each thread bundle takes out the data it needs from the specified location of the shared memory respectively; the arrangement needs to ensure that the two processes will not cause shared memory building conflicts;

[0211] The implicit matrix multiplication convolution kernel generated by module 55 adopts the NC*HW_X format; this data format uses the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as a tuning parameter to explore the optimal configuration;

[0212] Specifically, the module 55 converts the NCHW format to the Nc*HW_X format by the following method:

[0213] First, by reshaping the input tensor, a batch of matrices M of the same size to be transposed are generated. a ; Then perform matrix transposition on the Batch matrices to obtain the Batch matrices Matrix; finally, reshape the dimensions to get the output tensor in Nc*HW_X format;

[0214] The method of performing matrix transposition on each batch matrix is ​​NCHW = N·X·(C / X)·HW = N·X·c*·HW, that is, the matrix transposition of batch = N X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix;

[0215] Module 56: delivering the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and returning fitness information of the parameter combination to be tested;

[0216] Module 57, updating the fitness information into the dictionary D;

[0217] Module 58, select the optimization parameter combination with the highest fitness in the current population as the parent, and select at least one optimization parameter combination from the remaining multiple optimization parameter combinations in the current population except the optimization parameter combination with the highest fitness through the roulette algorithm as the parent, all parents perform mutation and crossover of parameter combination genes according to the preset crossover rate and mutation rate to update the population and execute module 52 again, until the tuning reaches the specified number of rounds, and select the individual parameter with the best fitness in the dictionary D as the best optimization parameter combination;

[0218] The module 56 comprises:

[0219] Module 561, initialization for saving fitness information set F;

[0220] Module 562, calling the compilation script, compiling the kernel function code and linking it into the test framework;

[0221] Module 563, start the test program, input the kernel function code to be tested and the convolution descriptor to the test framework, test and return the fitness of each kernel function code;

[0222] Module 564, write the measured fitness into the position corresponding to the kernel function code ID in the fitness information set F;

[0223] The module 9 generates a HIP code according to the dictionary S and the code template, specifically including:

[0224] For a convolution descriptor and its optimization parameter combination, a preset function template is selected; there are multiple identifiers in the function template; the code generator of the neural network accelerator generates a code block in the function template according to the convolution parameters in the convolution descriptor and the best optimization parameter combination; the code block is filled into the corresponding identifier position in the template to obtain a kernel function; the code generator generates a peripheral encapsulation function according to the number to call the corresponding kernel function through the number; the HIP file is generated according to the peripheral encapsulation function and the kernel function;

[0225] The tuner in module 5 is used to calculate the arithmetic strength AI of the operator in the convolution to be tuned by the following formula, and determine the result matrix block strategy of implicit matrix multiplication based on the arithmetic strength:

[0226]

[0227] Where TILE M and TILE N Respectively represent the block size of the result matrix in the M and N directions; FH and FW are the convolution kernel sizes, IC is the number of input channels, and the larger TILE is selected. M and TILE N Divide into blocks and TILE K Set to a size appropriate for the task assignment.

[0228] like Fig.11 As shown, the present invention further proposes a first electronic device A in another embodiment, comprising the convolution operator compilation optimization device.

[0229] like Fig.12 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain the input parameters of the convolution to be tuned, and the information display device D is used to display the convolution operation results obtained by the analysis of the present invention.

[0230] The information display device D can process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, such as the convolution operation results, Model.log file, HIP code, etc. The user can understand this information more timely without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0231] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the convolution operator compilation optimization method for the multi-core parallel architecture provided by the above methods.

[0232] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the convolution operator compilation optimization method of the multi-core parallel architecture. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0233] Fig.13 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0234] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0235] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0236] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0237] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A convolution operator compilation optimization method for multi-core parallel architecture, characterized in that: include: Step 1: Convert the input parameters of the convolution to be tuned into a convolution descriptor; Step 2: Perform a deduplication operation on the collected convolution descriptors to generate a convolution descriptor list; Step 3, read the log file into the dictionary D, which stores the historical records of the convolution descriptor and the corresponding best optimization parameter combination; initialize the dictionary S for storing the convolution descriptor and its corresponding best optimization parameter combination; Step 4, read the convolution descriptors from the descriptor list in sequence, and determine whether they are in the dictionary D. If so, execute step 7, otherwise determine whether they are in the dictionary S. If so, execute step 4 again to continue to take the next convolution descriptor, otherwise execute step 5; Step 5: The controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the best optimization parameter combination; Step 6: Write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary D, and execute step 8; Step 7: Read the best optimization parameter combination corresponding to the convolution descriptor from the dictionary D; Step 8: write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S, and determine whether all convolution descriptors in the convolution descriptor list have been optimized. If so, execute step 9; otherwise, execute step 4. Step 9, the controller numbers the convolution descriptors in the dictionary S, saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; generates a HIP code according to the dictionary S and the code template; obtains an execution file by compiling the HIP code, and the neural network accelerator runs the execution file according to the convolution descriptor and the Model.log file to obtain a convolution operation result.

2. The convolution operator compilation optimization method for multi-core parallel architecture according to claim 1, characterized in that: This step 5 includes: Step 51, initializing a dictionary D for recording the fitness of the optimized combination, and initializing the population; Step 52: traverse the individuals in the population, check the legality of the configuration of the individual according to the hardware resource information of the backend hardware, and directly set the fitness of the illegal configuration to 0; Step 53, check whether the individuals in the population that have passed the legality check are in the dictionary D, if so, directly read their fitness, otherwise execute step 54; Step 54, grouping the individuals in the population that have not obtained fitness, and generating a parameter combination to be tested; Step 55, the code generation module generates a kernel function code according to the parameter combination to be tested; and the generation process includes determining whether to generate a code using the soft pipelining technology or a code not using the soft pipelining technology according to the tuning parameters; The soft pipelining technology includes parallel execution of global memory access, shared memory access and Matrix Core calculation; the data stored in the shared memory must wait for the global memory access to be completed before it can be obtained, and until all thread warps fill the data obtained from the global memory into the shared memory, data exchange and sharing between thread warps are performed; and according to the requirements of the MFMA instruction, each data is placed in the register of the specified thread; The present invention provides a shared memory non-filled building conflict elimination method for implicit matrix multiplication and convolution of multi-core parallel architecture in shared memory, and the operation of shared memory includes two processes: storing the data taken from the global memory to the specified location in the shared memory; and each thread bundle takes out the data it needs from the specified location of the shared memory respectively; the arrangement needs to ensure that the two processes will not cause shared memory building conflicts; The implicit matrix multiplication convolution kernel generated in step 55 adopts the NC*HW_X format; this data format uses the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as a tuning parameter to explore the optimal configuration; Specifically, step 55 converts the NCHW format to the Nc*HW_X format by the following method: First, by reshaping the input tensor, a batch of matrices M of the same size to be transposed are generated. a ; Then perform matrix transposition on the Batch matrices to obtain the Batch matrices Matrix; finally, reshape the dimensions to get the output tensor in Nc*HW_X format; The method of performing matrix transposition on each batch matrix is ​​NCHW = N·X·(C / X)·HW = N·X·c*·HW, that is, the matrix transposition of batch = N X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix; Step 56: submit the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and return the fitness information of the parameter combination to be tested; Step 57: Update the fitness information into the dictionary D; Step 58, select the optimization parameter combination with the highest fitness in the current population as the parent, and select at least one optimization parameter combination from the remaining multiple optimization parameter combinations in the current population except the optimization parameter combination with the highest fitness through the roulette algorithm as the parent, all parents perform mutation and crossover of parameter combination genes according to the preset crossover rate and mutation rate to update the population and execute step 52 again until the tuning reaches the specified number of rounds, and then select the individual parameter with the best fitness in the dictionary D as the best optimization parameter combination.

3. The convolution operator compilation optimization method for multi-core parallel architecture according to claim 2, characterized in that: This step 56 comprises: Step 561, initializing a fitness information set F for saving; Step 562: Call the compilation script to compile and link the kernel function code into the test framework; Step 563: start the test program, input the kernel function code to be tested and the convolution descriptor into the test framework, test and return the fitness of each kernel function code; Step 564: Write the measured fitness into the position corresponding to the kernel function code ID in the fitness information set F.

4. The convolution operator compilation optimization method for multi-core parallel architecture according to any one of claims 1 to 3, characterized in that: In step 9, a HIP code is generated according to the dictionary S and the code template, specifically including: For a convolution descriptor and its optimization parameter combination, a preset function template is selected; there are multiple identifiers in the function template; the code generator of the neural network accelerator generates a code block in the function template according to the convolution parameters in the convolution descriptor and the best optimization parameter combination; the code block is filled into the corresponding identifier position in the template to obtain a kernel function; the code generator generates a peripheral encapsulation function according to the number to call the corresponding kernel function through the number; and the HIP file is generated according to the peripheral encapsulation function and the kernel function.

5. The convolution operator compilation optimization method for multi-core parallel architecture according to claim 1, characterized in that: The tuner in step 5 is used to calculate the arithmetic strength AI of the operator in the convolution to be tuned by the following formula, and determine the result matrix block strategy of implicit matrix multiplication based on the arithmetic strength: Where TILE M and TILE N Respectively represent the block size of the result matrix in the M and N directions; FH and FW are the convolution kernel sizes, IC is the number of input channels, and the larger TILE is selected. M and TILE N Divide into blocks and TILE K Set to a size appropriate for the task assignment.

6. A convolution operator compilation optimization device for a multi-core parallel architecture, characterized in that: include: Module 1: Convert the input parameters of the convolution to be tuned into convolution descriptors; Module 2: Perform deduplication operation on the collected convolution descriptors to generate a convolution descriptor list; Module 3, read the log file into the dictionary D, the log file saves the historical record of the convolution descriptor and the corresponding best optimization parameter combination; initialize the dictionary S for saving the convolution descriptor and its corresponding best optimization parameter combination; Module 4, read the convolution descriptors from the descriptor list in sequence, and determine whether they are in the dictionary D. If so, execute module 7. Otherwise, determine whether they are in the dictionary S. If so, execute module 4 again to continue to take the next convolution descriptor. Otherwise, execute module 5. Module 5, the controller submits the convolution descriptor to the tuner of the neural network accelerator to obtain the best optimization parameter combination; Module 6, write the convolution descriptor and its corresponding optimal optimization parameter combination into dictionary D, and execute module 8; Module 7, reading the best optimization parameter combination corresponding to the convolution descriptor from the dictionary D; Module 8, write the convolution descriptor and its corresponding optimal optimization parameter combination into the dictionary S, and determine whether all convolution descriptors in the convolution descriptor list are optimized. If yes, execute module 9, otherwise execute module 4; Module 9, the controller numbers the convolution descriptors in the dictionary S, saves the correspondence between the numbers and the convolution descriptors to the Model.log file, and saves the dictionary D to the Log file; generates a HIP code according to the dictionary S and the code template; obtains an execution file by compiling the HIP code, and the neural network accelerator runs the execution file according to the convolution descriptor and the Model.log file to obtain a convolution operation result.

7. The convolution operator compilation and optimization device for multi-core parallel architecture according to claim 6, characterized in that: This module 5 includes: Module 51, initializing a dictionary D for recording the fitness of the optimized combination, and initializing the population; Module 52, traverse the individuals in the population, check the legality of the configuration of the individual according to the hardware resource information of the backend hardware, and directly set the fitness of the illegal configuration to 0; Module 53, check whether the individuals in the population that have passed the legality check are in the dictionary D, if so, directly read their fitness, otherwise execute module 54; Module 54, grouping the individuals in the population that have not obtained fitness, and generating a parameter combination to be tested; Module 55, a code generation module generates a kernel function code according to the parameter combination to be tested; and the generation process includes determining whether to generate a code using the soft pipelining technology or a code not using the soft pipelining technology according to the tuning parameters; The soft pipelining technology includes parallel execution of global memory access, shared memory access and Matrix Core calculation; the data stored in the shared memory must wait for the global memory access to be completed before it can be obtained, and until all thread warps fill the data obtained from the global memory into the shared memory, data exchange and sharing between thread warps are performed; and according to the requirements of the MFMA instruction, each data is placed in the register of the specified thread; The present invention provides a shared memory non-filled building conflict elimination method for implicit matrix multiplication and convolution of multi-core parallel architecture in shared memory, and the operation of shared memory includes two processes: storing the data taken from the global memory to the specified location in the shared memory; and each thread bundle takes out the data it needs from the specified location of the shared memory respectively; the arrangement needs to ensure that the two processes will not cause shared memory building conflicts; The implicit matrix multiplication convolution kernel generated by module 55 adopts the NC*HW_X format; this data format uses the X input channel dimensions as the innermost dimension to achieve continuity in data arrangement, and uses the size of the X dimension as a tuning parameter to explore the optimal configuration; Specifically, the module 55 converts the NCHW format to the Nc*HW_X format by the following method: First, by reshaping the input tensor, a batch of matrices M of the same size to be transposed are generated. a ; Then perform matrix transposition on the Batch matrices to obtain the Batch matrices Matrix; finally, reshape the dimensions to get the output tensor in Nc*HW_X format; The method of performing matrix transposition on each batch matrix is ​​NCHW = N·X·(C / X)·HW = N·X·c*·HW, that is, the matrix transposition of batch = N X·c*HW, extracting the (C / X) dimension and appending it to the width of the matrix; Module 56: delivering the kernel function code to the kernel function execution module of the neural network accelerator for compilation and testing, and returning fitness information of the parameter combination to be tested; Module 57, updating the fitness information into the dictionary D; Module 58, select the optimization parameter combination with the highest fitness in the current population as the parent, and select at least one optimization parameter combination from the remaining multiple optimization parameter combinations in the current population except the optimization parameter combination with the highest fitness through the roulette algorithm as the parent, all parents perform mutation and crossover of parameter combination genes according to the preset crossover rate and mutation rate to update the population and execute module 52 again, until the tuning reaches the specified number of rounds, and select the individual parameter with the best fitness in the dictionary D as the best optimization parameter combination; The module 56 comprises: Module 561, initialization for saving fitness information set F; Module 562, calling the compilation script, compiling the kernel function code and linking it into the test framework; Module 563, start the test program, input the kernel function code to be tested and the convolution descriptor to the test framework, test and return the fitness of each kernel function code; Module 564, write the measured fitness into the position corresponding to the kernel function code ID in the fitness information set F; The module 9 generates a HIP code according to the dictionary S and the code template, specifically including: For a convolution descriptor and its optimization parameter combination, a preset function template is selected; there are multiple identifiers in the function template; the code generator of the neural network accelerator generates a code block in the function template according to the convolution parameters in the convolution descriptor and the best optimization parameter combination; the code block is filled into the corresponding identifier position in the template to obtain a kernel function; the code generator generates a peripheral encapsulation function according to the number to call the corresponding kernel function through the number; the HIP file is generated according to the peripheral encapsulation function and the kernel function; The tuner in module 5 is used to calculate the arithmetic strength AI of the operator in the convolution to be tuned by the following formula, and determine the result matrix block strategy of implicit matrix multiplication based on the arithmetic strength: Where TILE M and TILE N Respectively represent the block size of the result matrix in the M and N directions; FH and FW are the convolution kernel sizes, IC is the number of input channels, and the larger TILE is selected. M and TILE N Divide into blocks and TILE K Set to a size appropriate for the task assignment.

8. An electronic device, characterized in that: It includes a convolution operator compilation optimization device as described in claim 6 or 7, and the electronic device is connected to an information display device, and the information display device is used to display the evaluation result with display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the convolution operator compilation optimization method described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the convolution operator compilation optimization method described in any one of claims 1 to 5 are implemented.

Images