Method for accelerating AI operator library based on data stream architecture

Abstract

A method for accelerating an AI operator library based on a data stream architecture comprises the steps that S1, the operation type in operators is recognized, if the operation type is calculation-intensive operation, the step S2 is executed, and if the operation type is data-intensive operation, the step S3 is executed; s2, the step S2 comprises the following steps of S21 to S25: S21, generating a configuration file according to the matrix dimension and the hardware parameters; s22, generating a calculation task scheduling table according to the configuration file and the matrix dimension information; s23, generating an optimization assembly instruction; s24, constructing a data flow diagram and a PE communication relation diagram; s25, generating a master control code, a scheduling strategy and a DMA transmission strategy; s3, the step S3 comprises the following steps of S31 to S34: S31, defining a mapping relation between the PE and the loop unroll; s32, establishing and compiling a task folder; s33, establishing mapping among graph nodes, csv numbers and PE numbers; and S34, establishing an edge between the nodes.

Description

Technical Field

[0001] The present invention relates to the field of AI operator libraries, and more specifically, to a method for accelerating AI operator libraries based on a data flow architecture. Background Art

[0002] The current development of AI compiler operator libraries shows a trend toward diversification, high performance, ecosystem development, and standardization. The following is an analysis of the specific characteristics of AI compiler operator libraries:

[0003] 1. Diverse Operator Libraries Emerge: Currently, there are a variety of open-source, large-model general-purpose operator libraries for various AI chips. For example, the Zhiyuan large-model general-purpose operator library, FlagGems, developed in the Triton language, provides an open-source, unified, and efficient access solution for large-model operators. It supports over 120 large-model operators and is compatible with various AI chips from seven vendors. Furthermore, Huawei's Ascend CANN also offers components such as the Ascend C operator programming language and the AOL operator acceleration library, supporting a rich set of underlying interfaces to meet the needs of developers at all skill levels.

[0004] 2. High-Performance Optimization Technology: Operator libraries have made significant progress in performance optimization. For example, FlagGems, through code generation and runtime optimization, outperforms similar open-source projects in key metrics such as development efficiency and runtime performance. Its operators offer superior overall performance on Nvidia GPUs, surpassing the PyTorch CUDA version by over 90% on average. Huawei's Ascend CANN has launched the super operator MLAPO, which addresses the challenges of large-scale MoE model training scenarios by significantly reducing MLA pre-processing time and achieving end-to-end performance improvements.

[0005] 3. Standardization and Unification: Faced with the fragmented nature of hardware architecture, instruction sets, compilers, and operator libraries, the development of operator libraries is moving towards standardization and unification. In-depth research and practice is underway to explore a unified intermediate language, unified operator interface, and unified open-source operator library to lower the barrier to development for new algorithms for large models and accelerate chip architecture innovation.

[0006] The Zhiyuan Operator Library is oriented towards the computational needs of large models and is targeted at diverse chips. Leveraging the open-source and lightweight nature of the Triton compiler, it provides easily adaptable, high-performance operator implementations. Currently, various AI chips from seven manufacturers are integrated into FlagGems, providing more computing power options for large models. Leveraging the similarity of pointwise operations and the modularity of offset calculation code, automatic code generation is used to address discontinuous memory accesses for high-dimensional tensors. It can automatically handle tasks such as tensor broadcasting, index space calculations, parallel parameter calculations, and address offset calculations.

[0007] However, during the development and optimization of AI compiler operator libraries, not all operator types are simple pointwise operations. Due to their unique characteristics, pointwise operations can often be processed more efficiently using common compiler backend algorithms. These operations often operate independently at the data element level, and common compiler backend algorithms can, to a certain extent, adapt to their computational patterns, enabling relatively efficient code generation and execution. However, operations like Matrixmul are fundamentally different from pointwise operations. Matrix multiplication involves complex operations on large amounts of data, with close data dependencies and complex computational logic. Its computational process places extremely high demands on memory access patterns, parallel computing resource allocation, and optimized use of hardware instructions. Simply combining Matrixmul and pointwise operations with a common compiler backend algorithm for processing these operators may appear to achieve a unified optimization process at the code level, but in reality, it will not achieve the highest efficiency possible when executed on hardware. This is because general-purpose algorithms often fail to fully account for the unique hardware characteristics and optimization requirements of different operation types. They are unable to perform targeted optimizations for the unique characteristics of matrix multiplication operations. For example, they cannot effectively utilize the hardware's cache mechanisms, parallel computing units, and specific instruction sets, resulting in suboptimal performance when executed on the hardware. Furthermore, some current operator libraries are not tightly coupled with the hardware. During development, code-level optimizations are performed independently of the hardware chip to achieve code universality, which in reality significantly limits the full performance of the hardware. Different hardware chips exhibit significant differences in architecture design, instruction sets, cache structures, and parallel computing capabilities. These differences determine the optimal execution method for different types of operators. If operator library development fails to fully consider the characteristics of the hardware chip and is not tightly coupled and optimized with the hardware, it will be impossible to generate and optimize code specifically for the hardware's characteristics, and thus fail to fully utilize the various optimization mechanisms and resources provided by the hardware. Summary of the Invention

[0008] The present invention provides a method for accelerating an AI operator library based on a data flow architecture to solve the technical problems existing in the above-mentioned prior art.

[0009] To achieve the above objectives, the present invention provides a method for accelerating an AI operator library based on a data flow architecture, comprising:

[0010] S1: Identify the operation type in the operator. If the operation type is a computationally intensive operation, execute step S2; if the operation type is a data-intensive operation, execute step S3.

[0011] S2: Step S2 includes the following steps S21 to S25:

[0012] S21: Generate a configuration file based on the matrix dimensions and hardware parameters;

[0013] S22: Generate a computing task scheduling table based on the configuration file and matrix dimension information;

[0014] S23: Generate optimized assembly instructions;

[0015] S24: Construct data flow graph and PE communication relationship graph;

[0016] S25: Generate master control code, scheduling strategy and DMA transmission strategy;

[0017] S3: Step S3 includes the following steps S31 to S34:

[0018] S31: define the mapping relationship between PE and loop unroll;

[0019] S32: Create and compile the task folder;

[0020] S33: Mapping is established between graph nodes, CSV numbers and PE numbers;

[0021] S34: Establish edges between nodes.

[0022] In one embodiment of the present invention, in step S21 , the hardware parameters include the number of PEs and the DRAM bandwidth.

[0023] In one embodiment of the present invention, steps S21 to S25 are as follows:

[0024] S21: Get matrix dimensions and hardware parameters,

[0025] According to the matrix dimensions and hardware parameters, the optimal APP partitioning strategy and Task partitioning strategy are calculated. The APP partitioning strategy includes the input / output data size, computational load, and dependencies of the APP. The Task partitioning strategy includes the number of PEs and the tasks performed by each PE.

[0026] Store the APP partitioning strategy and Task partitioning strategy in structured data.

[0027] Generate a configuration file based on the structured data. The configuration file stores the app partitioning strategy, task partitioning strategy, and global parameters. The global parameters include the matrix dimension and the number of PEs.

[0028] S22: Calculate the matrix block information in each APP according to the configuration file.

[0029] Calculate the input / output data address of each computing task, including the on-chip storage address and DRAM address. The on-chip storage address is the storage location of each APP's input / output data in the on-chip SRAM, and the DRAM address is the storage location of each APP's input / output data in the off-chip DRAM.

[0030] Calculate the loop unrolling strategy, which includes the loop unrolling factor of each subtask and the SPM address of each Unroll.

[0031] Calculate data transfer information, which includes data that needs to be loaded from DRAM to SPM and data that needs to be transferred from SPM to other PEs.

[0032] Output computing task scheduling table, which contains the input / output data address of each Task, the computing parameters of each Subtask, and the data transmission path.

[0033] S23: Generate highly optimized assembly code according to the hardware instruction set. The assembly code includes calculation instructions, data transfer instructions, and synchronization instructions. The calculation instructions are matrix multiplication and addition operations, the data transfer instructions are DMA load / store instructions, and the synchronization instructions are inter-PE communication synchronization instructions.

[0034] Classify the assembly code by task and subtask, and generate corresponding CSV files. Each CSV file contains the optimized assembly instructions for the corresponding task.

[0035] S24: Construct a data flow graph, including nodes and edges, where nodes represent computing tasks or data transmission operations, and edges represent data dependencies.

[0036] Construct a PE communication relationship graph. The PE communication relationship graph includes intra-PE Copy edges and inter-PE Copy edges. For intra-PE Copy edges, data within the same PE can be reused. For inter-PE Copy edges, data between different PEs can be transmitted.

[0037] S25: Generate the master control code, which controls the execution process of the entire matrix multiplication.

[0038] Generate a scheduling strategy, which includes APP scheduling strategy and Task scheduling strategy. APP scheduling strategy determines the execution order of Tasks in each APP, and APP scheduling strategy determines the types of APPs that can be executed in parallel.

[0039] Generate DMA transfer strategies, including optimizing data handling order and using double buffering.

[0040] In one embodiment of the present invention, steps S31 to S34 are as follows:

[0041] S31: It is defined that a maximum of 4 loop unrolls can be mapped on a PE and the maximum number of unrolls mapped on a PE is the number of divided tasks;

[0042] S32: Derived from the number of unrolls on each PE, the corresponding number of tasks is created under the gpdpu_tensor folder, with four folders, task0 to task3, corresponding to all the information of unroll0 on each PE, task1 corresponding to all the information of unroll1 on each PE, task2 corresponding to all the information of unroll2 on each PE, and task3 corresponding to all the information of unroll3 on each PE.

[0043] According to the number of tasks determined, compile the csv file in the corresponding task folder to generate the file.

[0044] Among them, each PE can map up to 5 types of CSV. The function and distribution of each CSV are shown in the following table:

[0045]

[0046] According to the determined number of tasks, copy the corresponding task folder to the operator root directory and compile the node file.

[0047] S33: Define that LoadA has a maximum of 16 csv files, copyA has a maximum of 16 csv files, and loadB&cal has a maximum of 16 csv files. At this time, two types of csv numbers are recorded: the first type of csv numbering is in ascending order from 0, and the same number of numbers are generated according to the number of csv files; in the second type of csv numbering, loadA starts numbering from 0, copyA starts numbering from 16, and loadB&cal starts numbering from 32.

[0048] S34: For the edge between ld and cal in PE, if there is a load node in the PE, the edge is built;

[0049] Build an edge between ld and cp in PE, traverse each unroll, when (pe_id-1,pe_id) does not exist and (pe_id,pe_id+1) exists, then the current PE needs to build an edge from ld to cp in PE, pe_id is the coordinate of PE in the PE array inside the DFU processor,

[0050] Define an array int NodeinPE

[16] [3] and assign its initial value to 0. Use NodeinPE[][0] to count the ld node number of the current PE, use NodeinPE[][1] to count the cp node number of the current PE, and use NodeinPE[][2] to count the ld&cal node number of the current PE.

[0051] For the CP-CP edge between PEs, traverse each row of PEs in unroll units. If it is not the last PE in each row and the A array of the current PE is repeated with that of the next PE, then build the edge.

[0052] For the cp and cal edges between PEs, an edge is built if the current PE has a copy node.

[0053] The method for accelerating an AI operator library based on a dataflow architecture provided by the present invention can utilize a unique and efficient partitioning strategy to process matrix multiplication operators of any scale. This partitioning strategy takes into account the DMA transfer restrictions in the hardware architecture. By precisely analyzing the storage layout and computational requirements of the matrix data, the partitioning strategy rationally divides the matrix into multiple sub-blocks, ensuring the maximum possible transmission of continuous data during DMA transfer. Continuous data transmission means that large amounts of continuous memory data can be read or written at once, reducing the overhead associated with multiple small-scale data transfers. This significantly improves data transfer efficiency and reduces the impact of data transfer time on overall computing performance. Under the control of the scheduling strategy, the present invention can fully utilize hardware resources to ensure maximum data reuse between apps. In neural network computing, there is often a large amount of data sharing between different apps. For example, in convolution operations, multiple convolution kernels may need to process the same batch of input data. Through a reasonable scheduling strategy, the present invention enables apps to reuse data already loaded into memory as much as possible during processing, reducing data transfer between apps. This data reuse mechanism significantly reduces the number and amount of data transfers, reduces data transmission latency and bandwidth usage, and improves the efficiency of the entire computing system.

[0054] Through the inter-PE data flow mechanism of Task Parallelism and Copy, fast data transfer and efficient computation between PEs are achieved. This mechanism significantly reduces the need for repeated data loading by each PE, ensuring that data can be quickly transferred for computation between PEs, thereby improving data utilization and computational efficiency. As long as there are no dependencies between task blocks, tasks can be parallelized through Task. In neural network computations, many task blocks are independent of each other. For example, in convolution operations, convolution operations on different channels can be performed in parallel. Through the Task Parallelism mechanism, these independent task blocks can be assigned to different PEs for simultaneous execution, reducing waiting time between tasks and achieving fast execution. Specifically, the scheduling strategy dynamically allocates task blocks based on the characteristics of the task blocks and the resources available on the PEs, and coordinates the computational progress and data exchange between the various PEs to ensure the efficiency and stability of the entire computation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is the communication relationship graph of data flow edges;

[0057] Figure 2 Multiply the task partitioning strategy graph by 512*512 matrix;

[0058] Figure 3 Generate algorithm flow chart for csv type 2;

[0059] Figure 4 Generate algorithm flow chart for csv type 3;

[0060] Figure 5 For 512×512 matrix multiplication, subtask2 nodeMap number;

[0061] Figure 6 Flowchart for building edges between nodes. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0063] There are rich and varied types of operators in neural networks, and there are significant differences in computing modes, data access methods, and parallelism requirements. The original intention of designing general algorithms was to cover as many types of operators as possible and achieve the versatility and portability of the code. However, this processing method has led to its limitations in optimizing and accelerating each type of operator. For some operators with high computational complexity and strong data dependencies, it is difficult to perform targeted optimization and acceleration. The present invention proposes an innovative and efficient operator data processing solution on the DFU processor (DFU, Data Flow Unit data flow processor) based on the data flow architecture. The data flow architecture, with its unique parallel computing and data-driven characteristics, does not require traditional instruction sequence execution control, thereby making full use of the parallel computing resources of the hardware and improving the running speed of the operator.

[0064] The present invention first meticulously classifies and analyzes operators, identifying the key computational steps and data access patterns of different types of operators. Then, for each operator type, specialized optimization strategies and algorithms are designed, such as employing specific parallel computing modes, data caching strategies, and instruction scheduling methods to minimize computational latency and data transmission overhead. The solution also fully considers the hardware characteristics of the DFU processor, such as the number of parallel computing units, memory bandwidth, and cache structure, closely integrating operator optimization with these characteristics to ensure that the optimized operators can fully leverage their performance advantages on the DFU processor. Targeted optimization and acceleration strategies are designed for different types of operators based on the characteristics of the accelerator card. For example, for compute-intensive operators, multi-threaded parallel computing and instruction-level parallel optimization are employed to improve the computational parallelism of the operators. For data-intensive operators, cache optimization and other techniques are employed to reduce data access latency. Through the present invention, the operator library can provide the accelerator card with optimal operator implementation and acceleration strategies, significantly accelerating the network's execution speed and improving the overall performance of the neural network.

[0065] Based on this, the present invention provides a method for accelerating an AI operator library based on a data flow architecture, which includes:

[0066] S1: Identify the operation type in the operator. If the operation type is a computationally intensive operation, execute step S2; if the operation type is a data-intensive operation, execute step S3.

[0067] S2: Step S2 includes the following steps S21 to S25:

[0068] S21: Generate a configuration file based on the matrix dimensions and hardware parameters;

[0069] S22: Generate a computing task scheduling table based on the configuration file and matrix dimension information;

[0070] S23: Generate optimized assembly instructions;

[0071] S24: Construct data flow graph and PE communication relationship graph;

[0072] S25: Generate master control code, scheduling strategy and DMA transmission strategy;

[0073] S3: Step S3 includes the following steps S31 to S34:

[0074] S31: define the mapping relationship between PE and loop unroll;

[0075] S32: Create and compile the task folder;

[0076] S33: Mapping is established between graph nodes, CSV numbers and PE numbers;

[0077] S34: Establish edges between nodes.

[0078] In one embodiment of the present invention, steps S21 to S25 are as follows:

[0079] S21: Get matrix dimensions and hardware parameters,

[0080] According to the matrix dimensions and hardware parameters, the optimal APP partitioning strategy and Task partitioning strategy are calculated. The APP partitioning strategy includes the input / output data size, computational load, and dependencies of the APP. The Task partitioning strategy includes the number of PEs and the tasks performed by each PE.

[0081] This embodiment takes the matrix dimensions of both being 512×512 as an example, that is, two matrices of both being 512×512 are multiplied, and the hardware parameters include, for example, the number of PEs and the DRAM bandwidth.

[0082] Store the APP partitioning strategy and Task partitioning strategy in structured data.

[0083] Generate a configuration file based on the structured data. The configuration file stores the app partitioning strategy, task partitioning strategy, and global parameters. The global parameters include the matrix dimension and the number of PEs.

[0084] For example, a DFU with 16 PEs can be used to divide a 512×512 matrix into 16 sub-blocks, with each PE responsible for local operations within one of the sub-blocks. For example, PE0 is responsible for the lower 0-15 rows, and PE1 is responsible for rows 16-31.

[0085] S22: Calculate the matrix block information within each APP according to the configuration file, for example, each APP processes 16×16 sub-blocks.

[0086] Calculate the input / output data address of each computing task, including the on-chip storage (SPM) address and DRAM address. The on-chip storage address is the storage location of each APP's input / output data in the on-chip SRAM, and the DRAM address is the storage location of each APP's input / output data in the off-chip DRAM.

[0087] Calculate the loop unrolling strategy, which includes the loop unrolling factor of each subtask (e.g. 4×4 unrolling) and the SPM address of each Unroll (optimizing data locality).

[0088] Calculate data transfer information (copy pass), which includes data that needs to be loaded from DRAM to SPM (load) and data that needs to be transferred from SPM to other PEs (inter-PE communication).

[0089] Output computing task scheduling table, computing task scheduling table contains the input / output data address of each task, the computing parameters of each subtask (such as loop expansion factor) and the data transmission path (which data needs DMA transfer).

[0090] S23: Generate highly optimized assembly code based on the hardware instruction set. The assembly code includes calculation instructions, data transfer instructions, and synchronization instructions. The calculation instructions are matrix multiplication and addition (MADD) operations, the data transfer instructions are DMA load / store instructions, and the synchronization instructions are inter-PE communication synchronization instructions (such as COPY instructions).

[0091] Classify the assembly code by task and subtask, and generate corresponding CSV files (to facilitate subsequent code generation tool parsing). Each CSV file contains the optimized assembly instructions for the corresponding task.

[0092] S24: Figure 1 is the communication relationship diagram of the data flow edge,

[0093] Construct a data flow graph (DFG), which includes nodes and edges. Nodes represent computing tasks or data transfer operations, and edges represent data dependencies (e.g., the output of Task A is the input of Task B).

[0094] Construct a PE communication relationship graph. The PE communication relationship graph includes intra-PE Copy edges and inter-PE Copy edges. For intra-PE Copy edges, data within the same PE can be reused (e.g., SPM→PE register). For inter-PE Copy edges, data between different PEs can be transmitted (e.g., PE0→PE1).

[0095] S25: Generate the main control code, which controls the execution process of the entire matrix multiplication (such as initialization, task scheduling, and result collection).

[0096] Generate a scheduling strategy, which includes APP scheduling strategy and Task scheduling strategy. APP scheduling strategy determines the execution order of tasks within each APP (based on data dependency), and APP scheduling strategy determines the types of APPs that can be executed in parallel (based on the number of PEs).

[0097] Generate a DMA transfer strategy, including optimizing data transfer order and using double buffering. Optimizing data transfer order can reduce DRAM access conflicts (such as bank conflicts), while using double buffering can hide transfer delays.

[0098] For operators with high computational complexity and strong data dependencies, this paper proposes an innovative and efficient operator data processing solution on the DFU processor based on a dataflow architecture. With its unique parallel computing and data-driven nature, the dataflow architecture utilizes copy instructions for inter-PE data transfer, improving data interaction. It also utilizes task technology to increase parallelism, eliminating the need for traditional sequential instruction execution control. This allows full utilization of hardware parallel computing resources and increases operator execution speed.

[0099] The details are as follows:

[0100] Each SubKernel (task block) can be divided into up to four parallel tasks (hardware context switching). These tasks have no dependencies on each other, run independently, and share computing, storage, and communication resources, thus achieving efficient parallel processing and resource sharing. At the same time, each PE computing unit can share data through the copy instruction, eliminating the need for multiple transfers from SRAM to Register, reducing the computational delay caused by data transmission. The task division design scheme and the copy instruction design scheme of the present invention are as follows:

[0101] The number of tasks is determined by the maximum number of unrolls (loop unrolling) mapped on a PE. The number of unrolls mapped on a PE may be different. In this embodiment, a maximum of 4 unrolls can be mapped on a PE. For example, the task division of a 512×512 matrix multiplication is as follows: Figure 2 As shown, the red box represents task0, the blue box represents task1, the yellow box represents task2, and the green box represents task3.

[0102] In one embodiment of the present invention, steps S31 to S34 are as follows:

[0103] S31: It is defined that a maximum of 4 loop unrolls can be mapped on a PE and the maximum number of unrolls mapped on a PE is the number of divided tasks;

[0104] S32: Derived from the number of unrolls on each PE, the corresponding number of tasks is created under the gpdpu_tensor folder, with four folders, task0 to task3, corresponding to all the information of unroll0 on each PE, task1 corresponding to all the information of unroll1 on each PE, task2 corresponding to all the information of unroll2 on each PE, and task3 corresponding to all the information of unroll3 on each PE.

[0105] According to the number of tasks determined, compile the csv file in the corresponding task folder to generate the file.

[0106] Among them, each PE can map up to 5 types of CSV. The function and distribution of each CSV are shown in the following table:

[0107]

[0108] like Figure 3 、 Figure 4 The following are the flow charts of the algorithm for generating CSV types 2 and 3 respectively.

[0109] According to the determined number of tasks, copy the corresponding task folder to the operator root directory and compile the node file.

[0110] S33: Define that LoadA has a maximum of 16 csv files, copyA has a maximum of 16 csv files, and loadB&cal has a maximum of 16 csv files. At this time, two types of csv numbers are recorded: the first type of csv numbering is in ascending order from 0, and the same number of numbers are generated according to the number of csv files; in the second type of csv numbering, loadA starts numbering from 0, copyA starts numbering from 16, and loadB&cal starts numbering from 32.

[0111] For example, for 512×512 matrix multiplication, the subtask2 nodeMap number is as follows Figure 5 shown.

[0112] S34: Figure 6 The figure shows the flow chart of edge building between nodes. For edge building between ld and cal in a PE, if there is a load node in the PE, the edge will be built;

[0113] Establish an edge between ld and cp within the PE. Traverse each unroll. When (pe_id-1, pe_id) does not exist and (pe_id, pe_id+1) exists, the current PE needs to establish an edge from ld to cp within the PE. pe_id is the coordinate of the PE in the PE array inside the DFU processor. In this embodiment, the DFU processor is a 4×4 PE array, and the ld instruction only appears in the first column of the PE array. Therefore, when the above condition of judging the first column based on the PE coordinates is met, a data flow edge between ld and cp will be established within the PE in the first column.

[0114] Define an array int NodeinPE

[16] [3] and assign its initial value to 0. Use NodeinPE[][0] to count the ld node number of the current PE, use NodeinPE[][1] to count the cp node number of the current PE, and use NodeinPE[][2] to count the ld&cal node number of the current PE.

[0115] For the CP-CP edge between PEs, traverse each row of PEs in unroll units (one unroll is one task if the task is divided). If it is not the last PE in each row, and the A array of the current PE and the next PE is repeated, then build an edge.

[0116] For the cp and cal edges between PEs, an edge is built if the current PE has a copy node.

[0117] The method for accelerating an AI operator library based on a dataflow architecture provided by the present invention can utilize a unique and efficient partitioning strategy to process matrix multiplication operators of any scale. This partitioning strategy takes into account the DMA transfer restrictions in the hardware architecture. By precisely analyzing the storage layout and computational requirements of the matrix data, the partitioning strategy rationally divides the matrix into multiple sub-blocks, ensuring the maximum possible transmission of continuous data during DMA transfer. Continuous data transmission means that large amounts of continuous memory data can be read or written at once, reducing the overhead associated with multiple small-scale data transfers. This significantly improves data transfer efficiency and reduces the impact of data transfer time on overall computing performance. Under the control of the scheduling strategy, the present invention can fully utilize hardware resources to ensure maximum data reuse between apps. In neural network computing, there is often a large amount of data sharing between different apps. For example, in convolution operations, multiple convolution kernels may need to process the same batch of input data. Through a reasonable scheduling strategy, the present invention enables apps to reuse data already loaded into memory as much as possible during processing, reducing data transfer between apps. This data reuse mechanism significantly reduces the number and amount of data transfers, reduces data transmission latency and bandwidth usage, and improves the efficiency of the entire computing system.

[0118] Through the inter-PE data flow mechanism of Task Parallelism and Copy, fast data transfer and efficient computation between PEs are achieved. This mechanism significantly reduces the need for repeated data loading by each PE, ensuring that data can be quickly transferred for computation between PEs, thereby improving data utilization and computational efficiency. As long as there are no dependencies between task blocks, tasks can be parallelized through Task. In neural network computations, many task blocks are independent of each other. For example, in convolution operations, convolution operations on different channels can be performed in parallel. Through the Task Parallelism mechanism, these independent task blocks can be assigned to different PEs for simultaneous execution, reducing waiting time between tasks and achieving fast execution. Specifically, the scheduling strategy dynamically allocates task blocks based on the characteristics of the task blocks and the resources available on the PEs, and coordinates the computational progress and data exchange between the various PEs to ensure the efficiency and stability of the entire computation process.

[0119] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0120] Those skilled in the art will appreciate that the modules in the apparatuses of the embodiments may be distributed in the apparatuses of the embodiments as described in the embodiments, or may be located in one or more apparatuses different from the embodiments with corresponding changes. The modules in the above embodiments may be combined into one module or further divided into multiple sub-modules.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for accelerating an AI operator library based on a data flow architecture, characterized in that: include: S1: Identify the operation type in the operator. If the operation type is a computationally intensive operation, execute step S2; if the operation type is a data-intensive operation, execute step S3. S2: Step S2 includes the following steps S21 to S25: S21: Generate a configuration file based on the matrix dimensions and hardware parameters; S22: Generate a computing task scheduling table based on the configuration file and matrix dimension information; S23: Generate optimized assembly instructions; S24: Construct data flow graph and PE communication relationship graph; S25: Generate master control code, scheduling strategy and DMA transmission strategy; S3: Step S3 includes the following steps S31 to S34: S31: define the mapping relationship between PE and loop unroll; S32: Create and compile the task folder; S33: Mapping is established between graph nodes, CSV numbers and PE numbers; S34: Establish edges between nodes.

2. The method for accelerating an AI operator library based on a data flow architecture according to claim 1, characterized in that: In step S21 , the hardware parameters include the number of PEs and DRAM bandwidth.

3. The method for accelerating an AI operator library based on a data flow architecture according to claim 1, characterized in that: Steps S21 to S25 are as follows: S21: Get matrix dimensions and hardware parameters, According to the matrix dimensions and hardware parameters, the optimal APP partitioning strategy and Task partitioning strategy are calculated. The APP partitioning strategy includes the input / output data size, computational load, and dependencies of the APP. The Task partitioning strategy includes the number of PEs and the tasks performed by each PE. Store the APP partitioning strategy and Task partitioning strategy in structured data. Generate a configuration file based on the structured data. The configuration file stores the app partitioning strategy, task partitioning strategy, and global parameters. The global parameters include the matrix dimension and the number of PEs. S22: Calculate the matrix block information in each APP according to the configuration file. Calculate the input / output data address of each computing task, including the on-chip storage address and DRAM address. The on-chip storage address is the storage location of each APP's input / output data in the on-chip SRAM, and the DRAM address is the storage location of each APP's input / output data in the off-chip DRAM. Calculate the loop unrolling strategy, which includes the loop unrolling factor of each subtask and the SPM address of each Unroll. Calculate data transfer information, which includes data that needs to be loaded from DRAM to SPM and data that needs to be transferred from SPM to other PEs. Output computing task scheduling table, which contains the input / output data address of each Task, the computing parameters of each Subtask, and the data transmission path. S23: Generate highly optimized assembly code according to the hardware instruction set. The assembly code includes calculation instructions, data transfer instructions, and synchronization instructions. The calculation instructions are matrix multiplication and addition operations, the data transfer instructions are DMA load / store instructions, and the synchronization instructions are inter-PE communication synchronization instructions. Classify the assembly code by task and subtask, and generate corresponding CSV files. Each CSV file contains the optimized assembly instructions for the corresponding task. S24: Construct a data flow graph, including nodes and edges, where nodes represent computing tasks or data transmission operations, and edges represent data dependencies. Construct a PE communication relationship graph. The PE communication relationship graph includes intra-PE Copy edges and inter-PE Copy edges. For intra-PE Copy edges, data within the same PE can be reused. For inter-PE Copy edges, data between different PEs can be transmitted. S25: Generate the master control code, which controls the execution process of the entire matrix multiplication. Generate a scheduling strategy, which includes APP scheduling strategy and Task scheduling strategy. APP scheduling strategy determines the execution order of Tasks in each APP, and APP scheduling strategy determines the types of APPs that can be executed in parallel. Generate DMA transfer strategies, including optimizing data handling order and using double buffering.

4. The method for accelerating an AI operator library based on a data flow architecture according to claim 1, wherein: Steps S31 to S34 are as follows: S31: It is defined that a maximum of 4 loop unrolls can be mapped on a PE and the maximum number of unrolls mapped on a PE is the number of divided tasks; S32: Derived from the number of unrolls on each PE, the corresponding number of tasks is created under the gpdpu_tensor folder, with four folders, task0 to task3, corresponding to all the information of unroll0 on each PE, task1 corresponding to all the information of unroll1 on each PE, task2 corresponding to all the information of unroll2 on each PE, and task3 corresponding to all the information of unroll3 on each PE. According to the number of tasks determined, compile the csv file in the corresponding task folder to generate the file. Among them, each PE can map up to 5 types of CSV. The function and distribution of each CSV are shown in the following table: According to the determined number of tasks, copy the corresponding task folder to the operator root directory and compile the node file. S33: Define that LoadA has a maximum of 16 csv files, copyA has a maximum of 16 csv files, and loadB&cal has a maximum of 16 csv files. At this time, two types of csv numbers are recorded: the first type of csv numbering is in ascending order from 0, and the same number of numbers are generated according to the number of csv files; in the second type of csv numbering, loadA starts numbering from 0, copyA starts numbering from 16, and loadB&cal starts numbering from 32. S34: For the edge between ld and cal in PE, if there is a load node in the PE, the edge is built; Build an edge between ld and cp in PE, traverse each unroll, when (pe_id-1,pe_id) does not exist and (pe_id,pe_id+1) exists, then the current PE needs to build an edge from ld to cp in PE, pe_id is the coordinate of PE in the PE array inside the DFU processor, Define an array int NodeinPE[16][3] and assign its initial value to 0. Use NodeinPE[][0] to count the ld node number of the current PE, use NodeinPE[][1] to count the cp node number of the current PE, and use NodeinPE[][2] to count the ld&cal node number of the current PE. For the CP-CP edge between PEs, traverse each row of PEs in unroll units. If it is not the last PE in each row and the A array of the current PE is repeated with that of the next PE, then build the edge. For cp and cal edge building between PEs, an edge is built if the current PE has a copy node.

Images