AI acceleration engine device and equipment

Abstract

The invention discloses an AI acceleration engine device and equipment, and the device comprises an engine control module which is used for receiving an AI instruction sent by an instruction distribution part, and carrying out the analysis of the AI instruction; if the AI instruction is a first loading instruction for the first operand, the first operand is loaded from the local storage component, and the loaded first operand is sent to the dot product array; if the AI instruction is a second loading instruction for a second operand, the second operand is loaded from the local storage component, and the loaded second operand is sent to the dot product array; if the AI instruction is a write-back instruction, the calculation result in the post-processing module is written back to the local storage component; the dot product array is used for executing dot product operation on the first operand and the second operand and sending a calculation result to the post-processing module; and the post-processing module is used for executing post-processing operation on the calculation result of the dot product array. According to the scheme, an AI acceleration calculation scheme can be optimized, and the advantages of AI calculation and HPC calculation are combined.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to an AI acceleration engine device and equipment. Background Art

[0002] Artificial Intelligence (AI) technology is increasingly being applied to various fields. However, AI algorithms, such as convolutional neural networks, require high-speed computing of large amounts of data. Traditional computing architectures are usually used in high-performance computing (HPC) scenarios and are lacking in neural network processing. Its basic operations are arithmetic and logical operations, and it often takes hundreds of instructions to complete the processing of a neuron, making the execution of AI algorithms inefficient.

[0003] In order to meet the high-speed computing requirements of AI algorithms, existing technologies usually adopt a CPU+NPU structure. The traditional CPU processor is used as the main core for control, and the NPU (Neural Processing Unit) is used as the secondary core to accelerate AI computing. This structure can give full play to the performance of the CPU and NPU, but the existing NPU usually only supports FP16, BF16 and FP8 data formats, and lacks support for high-performance computing. Summary of the invention

[0004] Embodiments of the present invention provide an AI acceleration engine device and equipment to optimize the AI ​​acceleration computing solution and combine the advantages of AI computing and HPC computing.

[0005] According to one aspect of the present invention, an AI acceleration engine device is provided, the device comprising:

[0006] The engine control module is used to receive the AI ​​instruction sent by the instruction distribution component and parse the AI ​​instruction; if the AI ​​instruction is a first load instruction for a first operand, the first operand is loaded from the local storage component and the loaded first operand is sent to the dot product array; if the AI ​​instruction is a second load instruction for a second operand, the second operand is loaded from the local storage component and the loaded second operand is sent to the dot product array; if the AI ​​instruction is a write-back instruction, the calculation result in the post-processing module is written back to the local storage component;

[0007] A dot product array, used to perform a dot product operation on the first operand and the second operand, and send the calculation result to a post-processing module;

[0008] The post-processing module is used to perform post-processing operations on the calculation results of the dot product array to obtain the calculation results after post-processing.

[0009] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor and an AI unit externally mounted on the processor; the AI ​​unit comprising: a local storage component, an internal register, an instruction distribution component and an AI acceleration engine device according to any embodiment of the present invention;

[0010] A processor, configured to send instructions to an instruction distribution component;

[0011] An instruction distribution component, used to integrate and parse instruction information obtained from the processor and internal registers, and if the instruction is determined to be an AI instruction according to the parsing result, the AI ​​instruction is sent to the AI ​​acceleration engine device;

[0012] A local storage component for storing data;

[0013] Internal registers are used to store data and instruction information.

[0014] The embodiments of the above invention have the following advantages or beneficial effects: the engine control module can accurately receive and parse AI instructions, and then quickly load operands from the local storage component to the dot product array according to the instruction type, avoiding data transmission delays, reducing waiting time, and effectively accelerating the calculation process. After that, the dot product array is used specifically for dot product operations, which can perform operations with higher efficiency, thereby significantly accelerating the overall AI calculation process. Finally, the calculation results of the dot product array are post-processed by the post-processing module to obtain the post-processed calculation results, combining the advantages of AI calculations and HPC calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the structure of an AI unit provided in an embodiment of the present invention;

[0016] Figure 2 is a schematic diagram of the structure of an AIE provided by an embodiment of the present invention;

[0017] Figure 3 is a structural schematic diagram of an engine control module provided by an embodiment of the present invention;

[0018] Figure 4 is a structural diagram of an instruction parsing module provided by an embodiment of the present invention;

[0019] Figure 5 is a schematic diagram of the structure of a first operand loading module or a second operand loading module provided by an embodiment of the present invention;

[0020] Figure 6 This is an example diagram of a four-in-one access provided by an embodiment of the present invention;

[0021] Figure 7is a structural diagram of a write-back module provided by an embodiment of the present invention;

[0022] Figure 8 This is an example diagram of an eight-in-one access provided by an embodiment of the present invention;

[0023] Fig. 9 is a schematic diagram of the structure of a dot product array provided by an embodiment of the present invention;

[0024] Fig.10 is a structural schematic diagram of a post-processing module provided by an embodiment of the present invention;

[0025] Fig.11 is an example diagram of a weight vector for loading a convolutional neural network provided by an embodiment of the present invention;

[0026] Fig.12 is an example diagram of a loading eigenvalue vector provided by an embodiment of the present invention;

[0027] Fig.13 is an example diagram of an accumulated dot product result provided by an embodiment of the present invention;

[0028] Fig.14 This is an example diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] The purpose of an embodiment of the present invention is to realize an AI acceleration engine device that supports both HPC application scenarios and AI application scenarios. The AI ​​acceleration engine device will be referred to as AIE in the following. The application advantage of HPC lies in accuracy, and most of its research objects are concentrated in the field of scientific computing, so the requirements for data accuracy are very high. AI's requirements for computing speed take precedence over its requirements for data accuracy, but AI is an empirical research model, so its data accuracy needs to be improved. The embodiment of the present invention realizes the integration of AI and HPC at the hardware design level, allowing users to perform verification of AI algorithms through HPC algorithms, while ensuring computing speed and improving accuracy.

[0031] The AIE designed in the embodiment of the present invention is based on the fifth-generation Reduced Instruction Set Computing-V (RISC-V) and uses a custom extended AI instruction set. The instruction set is fully integrated with the AI ​​algorithm and has good readability and flexibility. The embodiment of the present invention supports the instruction set in hardware implementation, and ensures the pipeline performance of the hardware by fully exploring the parallelism between instructions. The embodiment of the present invention supports the sparse acceleration function in the AI ​​algorithm, and adopts the means of memory access merging to reduce the frequency of interaction between the AIE and the local storage component, so the AIE implemented by the present invention has a higher data handling efficiency.

[0032] In order to achieve the above objectives, the embodiments of the present invention have the following solutions.

[0033] The general RISC-V instruction set is usually only 32 bits, and the configurable instruction information is limited. The present invention stores the specific information of the AI ​​instructions used by AIE in the general registers and internal registers provided by the local CPU core. The user can directly configure these two registers through the general instructions of the register configuration type. The embodiment of the present invention stores the addressing information of these two registers in a 32-bit instruction. After the user sends the instruction, the hardware can obtain the specific configuration of the AI ​​instruction by addressing the two registers, and execute the corresponding instruction function in AIE.

[0034] The AIE designed by the present invention adopts an asynchronous mechanism to run. Figure 1 As shown, AIE is part of the AI ​​unit, which is a computing acceleration unit external to the CPU. The instruction distribution component in the AI ​​unit obtains instruction information from the local CPU core and internal registers. If the instruction parsed by the instruction distribution component is an AI instruction, the instruction is distributed to the AIE. The data of AIE comes from the local storage component, and the calculation results are also stored in the local storage component. If AIE executes the current instruction, it initiates a write reply word request to the internal register or local storage component to inform that the asynchronous event is completed. The status information of AIE execution instructions can be obtained through the IO module of the AI ​​unit, namely the AI ​​unit_IO module. It should be noted that the AI ​​unit also includes a module responsible for transcendental function calculation, a network interface for remote access, etc., which are not shown in the figure because they are less relevant to the embodiments of the present invention.

[0035] The embodiment of the present invention adopts an extended AI instruction set based on the RISC-V instruction set, and defines five types of instructions, namely, configuration instruction (CFG), load instruction of operand A (LDA), load instruction of operand B (LDB), write back instruction of calculation result (ST), and automatic load instruction of operand A / B (CTN). The functions corresponding to the five instructions are as follows:

[0036] (1) CFG: By sending this instruction, the configuration of AIE can be initialized, including whether to enable the sparse acceleration function, the input data format, whether to enable the fast format conversion function, and the rounding mode of floating-point calculations.

[0037] (2) LDA: By sending this instruction, operand A for calculation can be loaded from the local storage component and sent to the dot product array of AIE. This instruction supports multiple loading mode configurations, including but not limited to the initial address of the local storage component, address stride data, PADDING, write answer word, accumulation mode, data mask, OPBREG0 / 1 (register for storing operand B) selection, etc.

[0038] (3) LDB: By sending this instruction, operand B for calculation can be loaded from the local storage component and sent to the dot product array of AIE. This instruction supports multiple loading mode configurations, including but not limited to the initial address of the storage component, address stride mode, write answer word, data mask, OPB REG0 / 1 (register for storing operand B), whether operand B is transposed, etc.

[0039] (4) ST: By sending this instruction, the user can write the calculation results of AIE back to the local storage component. This instruction supports the configuration of multiple write-back methods, including but not limited to the data format of the write-back data, whether to write back the maximum absolute value, whether to perform scaling offset operations on the calculation results, the initial address of the write-back local storage component, the address stride mode, the selection of the calculation result buffer 0 / 1, the length of the write-back data, whether to clear the buffer, etc.

[0040] (5) CTN: By sending this instruction, AIE can be commanded to automatically execute multiple rounds of LDB→LDA instructions.

[0041] AIE needs to support the execution of the above five instructions in hardware implementation. Its overall architecture is as follows: Figure 2 As shown, AIE specifically includes: engine control module (EC), dot product array, post-processing module (POSP) and AIE input and output (IO) module, namely AIE_IO module. Among them:

[0042] The engine control module (EC) is used to receive the AI ​​instruction sent by the instruction distribution component and parse the AI ​​instruction; if the AI ​​instruction is a first load instruction for the first operand, the first operand is loaded from the local storage component and the loaded first operand is sent to the dot product array; if the AI ​​instruction is a second load instruction for the second operand, the second operand is loaded from the local storage component and the loaded second operand is sent to the dot product array; if the AI ​​instruction is a write-back instruction, the calculation result in the post-processing module is written back to the local storage component; if the AI ​​instruction is a configuration instruction, the configuration instruction is written back to the local storage component. The configuration information in the configuration register is written, and the configuration information is classified and forwarded to the dot product array and the post-processing module, etc.; if the AI ​​instruction is a compound load instruction for automatically executing multiple rounds of the first load instruction and the second load instruction, the compound load instruction is disassembled into multiple first load instructions and multiple second load instructions, and for each disassembled first load instruction, the first operand is loaded from the local storage component, and the loaded first operand is sent to the dot product array, and for each disassembled second load instruction, the second operand is loaded from the local storage component, and the loaded second operand is sent to the dot product array. Among them, the first operand is operand A, the second operand is operand B, the first load instruction is LDA instruction, the second load instruction is LDB instruction, the write back instruction is ST instruction, the configuration instruction is CFG instruction, and the compound load instruction is CTN instruction.

[0043] A dot product array (DPA), configured to perform a dot product operation on the first operand and the second operand, and send the calculation result to a post-processing module;

[0044] The post-processing module (POSP) is used to perform post-processing operations on the calculation results of the dot product array to obtain the calculation results after post-processing. The post-processing operations include: at least one of accumulation operations, cache operations, scaling and offset operations, absolute value maximum statistics operations and data format conversion operations.

[0045] Functionally, the EC in the embodiment of the present invention can reduce the number of accesses to the local storage component by means of memory access merging, improve the utilization rate of the interface between the AIE and the local storage component, and improve the data handling efficiency of the AIE. That is, the above EC loads the first operand from the local storage component, specifically including: calculating the read address of each piece of data of the first operand to be obtained in the local storage component based on the starting address and address stride data in the first load instruction; determining whether the data access requests of each piece of data can be merged according to the read address of each piece of data in the local storage component; if so, merging the data access requests of each piece of data, and sending the merged data access request to the local storage component to read each piece of data of the first operand from the local storage component. Similarly, the EC loads the second operand from the local storage component, specifically including: calculating the read address of each piece of data of the second operand to be obtained in the local storage component based on the starting address and address stride data in the second load instruction; determining whether the data fetching requests of each piece of data can be merged according to the read address of each piece of data in the local storage component; if so, merging the data fetching requests of each piece of data, and sending the merged data fetching request to the local storage component to read each piece of data of the second operand from the local storage component. The EC writes the calculation result in the post-processing module back to the local storage component, including: calculating the write address of each piece of data of the calculation result to be written in the local storage component based on the starting address and address stride data in the write-back instruction; determining whether the write request of each piece of data can be merged according to the write address of each piece of data in the local storage component; if so, merging the write request of each piece of data, and sending the merged write request to the local storage component to write each piece of data of the calculation result into the local storage component. The specific merging rules and methods will be introduced in detail later.

[0046] Functionally, the embodiments of the present invention can add support for high-performance computing (FP64 and FP32) on the basis of the traditional AI acceleration engine (supporting calculations in multiple data formats such as FP16, BF16, FP8, BF8, etc.) by multiplexing the dot product array and the computing module in the post-processing module (i.e., the accumulation unit introduced later). That is, the above-mentioned post-processing module performs post-processing operations on the calculation results of the dot product array, specifically including: when the accumulation mode is configured to be turned on in the first load instruction, the calculation results sent into the dot product array are accumulated with the calculation results cached in the buffer unit of the post-processing module, and the accumulated calculation results are written into the buffer unit; or when the input data format is a preset data format, multiple rounds of accumulation operations are performed, and the accumulation operation is to accumulate the calculation results sent into the dot product array with the calculation results of the previous round of the post-processing module.

[0047] Specifically, Figure 3As shown, the engine control module (EC) includes: an instruction parsing module, a first operand loading (OAL) module, a second operand loading (OBL) module, a write back (OCW) module and a configuration module. Among them:

[0048] an instruction parsing module, configured to receive an AI instruction sent by the instruction distribution component and parse the AI ​​instruction; if the AI ​​instruction is a first load instruction for a first operand, the first load instruction is sent to a first operand load module; if the AI ​​instruction is a second load instruction for a second operand, the second load instruction is sent to a second operand load module; if the AI ​​instruction is a write-back instruction, the write-back instruction is sent to a write-back module; if the AI ​​instruction is a configuration instruction, the configuration instruction is sent to a configuration module;

[0049] a first operand loading (OAL) module, configured to load a first operand from a local storage component based on a first load instruction, and to send the loaded first operand to a dot product array;

[0050] a second operand loading (OBL) module, configured to load a second operand from a local storage component based on a second load instruction, and send the loaded second operand to a dot product array;

[0051] A write-back (OCW) module, used to write the calculation results of the post-processing module back to the local storage component based on the write-back instruction;

[0052] The configuration module is used to write the configuration information in the configuration instruction into the configuration register, and classify and forward the configuration information to the first operand loading module, the second operand loading module, the write-back module, the dot product array and the post-processing module.

[0053] Among them, the configuration instruction may include at least one of the following configuration information: whether to enable the configuration instruction of the sparse acceleration function, the input data format, and whether to enable the fast format conversion function. The sparse acceleration function refers to the engine control module performing sparse loading on the first operand according to the configured sparse multiple, the dot product array extracts the valid first operand from the first operand loaded by the engine control module according to the configured sparse index, and performs dot product operation on the valid first operand and the second operand. The sparse index is used to indicate the position of the valid first operand in the loaded data. Sparse loading means that when the first operand is loaded from the local storage component, the loaded data bit width is N times the original preset bit width, and N is the sparse multiple. For example, if the value of N is 2 and the original preset bit width is 64B, the data bit width to be loaded when sparse loading is 128B; for another example, if the value of N is 4 and the original preset bit width is 64B, the data bit width to be loaded when sparse loading is 256B. The input data formats may include: FP8, INT8, FP16, BF16, FP32, and FP64; when the fast format conversion function is turned on, the second operand may not be loaded. Through the first load instruction, the first operand may be stored in a buffer of the post-processing module in a specific data format. Through the write-back instruction, the first operand may be written back to the local storage component in a specified format. Therefore, the function of directly converting the operand into a specified output format may be realized.

[0054] Specifically, Figure 4 As shown, the instruction parsing module includes: a first instruction cache module, an instruction processing module (IND), a second instruction cache module and an instruction distribution module (INI).

[0055] A first instruction cache module, used for caching AI instructions sent by the instruction distribution component;

[0056] The instruction processing module (IND) is used to read the AI ​​instruction cached in the first instruction cache module. If the AI ​​instruction read is a compound load instruction, i.e., a CTN instruction, for automatically executing multiple rounds of first load instructions and second load instructions, multiple first load instructions and multiple second load instructions are disassembled based on the compound load instruction, and each disassembled first load instruction and each second load instruction is sent to the second instruction cache module; if the AI ​​instruction read is not a compound load instruction, the AI ​​instruction is directly sent to the second instruction cache module; wherein, after reading the AI ​​instruction cached in the first instruction cache module, the instruction processing module can first perform an exception check on the AI ​​instruction read, including checks for illegal instruction sequence, out-of-bounds address, conflicting configuration, etc., and then perform subsequent operations after the exception check passes. If the AI ​​instruction fails the exception check, the instruction will be intercepted and not sent to the second instruction cache module. At the same time, the abnormal instruction information will be sent to the AIE_IO module for recording.

[0057] A second instruction cache module, used for caching AI instructions sent by the instruction processing module;

[0058] The instruction dispatch module (INI) is used to read the AI ​​instruction cached in the second instruction cache module, identify the instruction type of the read AI instruction, and send the read AI instruction to the first operand loading module, the second operand loading module, the write back module or the configuration module according to the instruction type.

[0059] By disassembling the CTN instruction in IND, OBL will only receive LDB instructions and OAL will only receive LDA instructions, which can simplify the design of the two and make the hardware structure clearer.

[0060] Furthermore, in order to enable the first operand loading module, the second operand loading module and the write-back module to receive and process new instructions only when the software and hardware resources allow, thereby ensuring the orderliness and correctness of the overall instruction distribution and processing, the instruction distribution module (INI) in this embodiment is specifically used to: read the AI ​​instruction cached in the second instruction cache module, and identify the instruction type of the read AI instruction; if the instruction type is the first load instruction, then determine whether the first operand loading module can receive the new first load instruction based on the instruction execution status data fed back by the first operand loading module, the second operand loading module and the write-back module, and if so, send the first load instruction to the first operand loading module. Specifically, two conditions need to be judged. Condition 1 is to determine whether the first operand loading module can receive a new first loading instruction according to the instruction execution data fed back by the first operand loading module; Condition 2 is to determine whether the first loading instruction can occupy the corresponding OPBREG and buffer according to the instruction execution data fed back by the second operand loading module and the write-back module; If both of the above conditions are met, the first loading instruction is sent to the first operand loading module; If the instruction type is the second loading instruction, the second operand loading module is determined to receive a new second loading instruction according to the instruction execution data fed back by the second operand loading module and the first operand loading module, and if so, the second loading instruction is sent to the second operand loading module. Specifically, two conditions need to be judged. Condition 1 is to determine whether the second operand loading module can receive a new second loading instruction according to the instruction execution data fed back by the second operand loading module; Condition 2 is to determine whether the instruction can occupy the corresponding OPB REG according to the instruction execution data fed back by the first operand loading module; If both of the above conditions are met, the second loading instruction is sent to the second operand loading module. If the instruction type is a write-back instruction, then the write-back module and the first operand loading module feedback instruction execution status data to determine whether the write-back module can receive a new write-back instruction, and if so, the write-back instruction is sent to the write-back module. Specifically, two conditions need to be judged. Condition one is to determine whether the write-back module can receive a new write-back instruction based on the instruction execution status data fed back by the write-back module; condition two is to determine whether the instruction can occupy the corresponding buffer based on the instruction execution status data fed back by the first operand loading module; if both of the above two conditions are met, the write-back instruction is sent to the write-back module.

[0061] Specifically, Figure 5 As shown, the first operand loading (OAL) module or the second operand loading (OBL) module includes: a first storage interaction module (RS), a first data cache module, an array interaction module (DD) and a first scoreboard module; wherein,

[0062] A first storage interaction module (RS), configured to send a data fetch request for obtaining an operand to a local storage component based on a received load instruction; wherein, if the RS belongs to the OAL, the load instruction received by the RS is a first load instruction, and the data fetch request sent is for obtaining the first operand; if the RS belongs to the OBL, the load instruction received by the RS is a second load instruction, and the data fetch request sent is for obtaining the second operand;

[0063] A first data cache module, used for caching the operand returned by the local storage component; wherein, if the first data cache module belongs to the OAL, the first data cache module caches the first operand returned by the local storage component; if the first data cache module belongs to the OBL, the first data cache module caches the second operand returned by the local storage component;

[0064] An array interaction module (DD) is used to send the operand cached in the first data cache module to a dot product array (DPA); wherein, if the DD belongs to the OAL, the DD sends the first operand cached in the first data cache module in the OAL to the dot product array; if the DD belongs to the OBL, the DD sends the second operand cached in the first data cache module in the OBL to the dot product array;

[0065] The first scoreboard module is used to record instruction execution data and send the instruction execution data to the instruction parsing module (such as the instruction distribution module in the instruction parsing module), so that the instruction parsing module determines whether to send a load instruction to the first operand loading module or the second operand loading module according to the instruction execution data. Wherein, if the first scoreboard module belongs to OAL, the first scoreboard module records the instruction execution data of OAL, and sends the instruction execution data of OAL to the instruction parsing module, so that the instruction parsing module determines whether to send the first load instruction to OAL according to the instruction execution data of OAL, combined with the occupancy of OBL REG and the occupancy of buffer obtained by other scoreboard modules; if the first scoreboard module belongs to OBL, the first scoreboard module records the instruction execution data of OBL, and sends the instruction execution data of OBL to the instruction parsing module, so that the instruction parsing module determines whether to send the second load instruction to OBL according to the instruction execution data of OBL, combined with the occupancy of OBL REG obtained by other scoreboard modules.

[0066] Specifically, the instruction execution status data of the OAL includes: the idle state data of the RS in the OAL, the overflow state data of the FIFO queue in the DD in the OAL, and the occupancy data of the buffer in the post-processing module required for the first load instruction; the INI in the instruction parsing module determines that the OAL can receive a new first load instruction when judging that the RS of the OAL is idle, the FIFO in the DD of the OAL is not full, and no instruction currently occupies the buffer in the post-processing module required for the first load instruction according to the instruction execution status data of the OAL. The instruction execution status data of the OBL includes: the idle state data of the RS in the OBL, the overflow state data of the FIFO queue in the DD in the OBL, and the occupancy data of the operand register unit in the dot product array required for the second load instruction; the INI in the instruction parsing module determines that the OBL can receive a new second load instruction when judging that the RS of the OBL is idle, the FIFO in the DD of the OBL is not full, and no instruction currently occupies the operand register unit in the dot product array required for the second load instruction according to the instruction execution status data of the OBL. For OAL, if the current first load instruction sends all the data access requests to the local storage component, its RS is considered to be idle; if DD sends all the data to the dot product array, its DD is considered to be idle, and the occupation of the corresponding operand register unit is completed; if all the data in the operation module (including the dot product array and the post-processing module) is calculated, it is considered that the occupation of the buffer is completed. For OBL, if the current second load instruction sends all the data access requests to the local storage component, its RS is considered to be idle; if DD sends all the data to the dot product array, its DD is considered to be idle, and the occupation of the corresponding operand register unit is completed; OBL does not occupy the buffer.

[0067] Further, as described above, after the AIE parses the instruction through EC, it will initiate an access request to the local storage component to move data. The data handling efficiency often determines the data loading efficiency of the computing component, thereby affecting the overall performance of the AIE. The embodiment of the present invention can support the sparsification acceleration function required by the AI ​​algorithm. In order to ensure the data handling efficiency of the AIE when the sparsification acceleration function of the maximum sparsification multiple (for example, 4 times) is turned on, the embodiment of the present invention widens the data bit width between the AIE and the local storage component to the maximum sparsification multiple of the original bit width (for example, widened to 4 times the original bit width). However, when the sparsification acceleration function of the maximum sparsification multiple is not turned on, the widened data bit width is not fully utilized, which will cause a waste of hardware resources. For this problem, the embodiment of the present invention merges multiple non-conflicting access requests into one request by implementing memory access merging inside the AIE.

[0068] Specifically, for OAL and OBL, the widened data read bit width between AIE and the local storage component is widened, and the widened data read bit width can be the product of the data bit width that the dot product array can consume at a time and the preset maximum sparsification multiple. Correspondingly, in order to realize the above-mentioned memory access merging, the first storage interaction module (RS) is specifically used to: calculate the read address of each piece of data of the operand to be acquired in the local storage component based on the starting address and address stride data in the load instruction when the sparsification acceleration function is not enabled; determine whether the data fetching requests of each piece of data can be merged according to the read address of each piece of data in the local storage component; if so, merge the data fetching requests of each piece of data, and send the merged data fetching request to the local storage component to read each piece of data of the operand from the local storage component; if not, send the data fetching request of each piece of data to the local storage component separately. Specifically, determining whether the data fetching requests for each data entry can be merged is performed based on the read addresses of each data entry in the local storage component, including: determining the column (bank) number corresponding to the read addresses of each data entry in the local storage component, if the bank numbers corresponding to the read addresses are different, then determining that the data fetching requests for each data entry can be merged into one data fetching request, otherwise, determining that the data fetching requests for each data entry cannot be merged.

[0069] Take the case of loading data from the local storage component in FP8 computing mode as an example: Assuming that the data bit width that the DPU can consume at a time in the dot product array is 64B, when the 4x sparse loading function is enabled, 256B of data needs to be read from the local storage component, so the read data bit width of the AIE and the local storage component is defined as 256B. However, when the sparse loading function is not enabled, only 64B of data needs to be read from the local storage component, and 192B of bandwidth is wasted. In order to avoid bandwidth waste, the following merging method can be used: Taking the case of loading operand B with the LDB instruction and not enabling the sparse loading function as an example, the OAL in the EC uses the address calculation formula to calculate the read address of each piece of data in the local storage component according to the starting address and address stride data configured in the LDB instruction:

[0070] Address 0 = starting address + address granularity * address stride

[0071] Address 1 = Address 0 + Address Granularity * Address Stride

[0072] Address 2 = Address 1 + Address Granularity * Address Stride

[0073] Address 3 = Address 2 + Address Granularity * Address Stride

[0074] like Figure 6As shown in the figure, when the sparse loading function is not enabled, the address granularity is 64B, assuming that the stride is 1, and the starting address is 0x000. Then the read address of the first data is 0x000, the read address of the second data is 0x040, the read address of the third data is 0x080, and the read address of the fourth data is 0x0c0. For the local storage component, when the lower 8 bits of the address are 0x00, it means bank0; when the lower 8 bits of the address are 0x40, it means bank1; when the lower 8 bits of the address are 0x80, it means bank2; when the lower 8 bits of the address are 0xc0, it means bank3; when the high bits of the addresses are the same, they are the same row, and when the high bits of the addresses are different, they are different rows. Regardless of whether the row numbers of multiple requests are the same, as long as the bank numbers are different, it is considered that these requests can be merged for access. Therefore, these four requests can be merged into one request. It can be seen that by supporting memory access merging in hardware, as long as the configured address stride is reasonable, the efficiency of AIE reading data from the local storage component can be effectively improved.

[0075] Furthermore, the array interaction module (DD) includes: a first-in-first-out storage unit (FIFO) and a data placement unit.

[0076] A first-in-first-out storage unit (FIFO); used to store the instruction configuration information in the load instruction put in by the first storage interaction module; similarly, if the current DD belongs to OAL, the FIFO stores the instruction configuration information in the first load instruction put in by the RS in the OAL; if the current DD belongs to OBL, the FIFO stores the instruction configuration information in the second load instruction put in by the RS in the OBL; a number placing unit, used to determine whether the operand cached in the first data cache module can be sent based on the instruction configuration information in the FIFO, and if so, the operand cached in the first data cache module is sent to the dot product array. Similarly, if the current DD belongs to OAL, the number placing unit is to send the first operand cached in the first data cache module in the OAL to the dot product array; the method for determining whether the first operand can be sent is to determine the cache address of the current calculation result in the buffer in the post-processing module according to the buffer identifier and the initial address in the instruction configuration information, determine whether there is an address conflict with the previous operation according to the cache address, and send the first operand cached in the first data cache module to the dot product array when it is determined that there is no address conflict. If the current DD belongs to OBL, the data release unit sends the second operand cached in the first data cache module in OBL to the dot product array. The method for judging whether the second operand can be sent is to determine whether the bit width of the currently collected data to be sent is consistent with the preset maximum data bit width. If so, it is determined that the second operand can be sent, otherwise, the second operand cannot be sent. Specifically, it can be determined whether there is a data merge at present based on the address stride data in the configuration information. Assuming that there is no data merge, the effective granularity of a data is 64B, and 4 data need to be collected before the second operand can be sent; then if there is a data merge, for example, the effective granularity of a data is 256B, only 1 data needs to be collected to send the second operand.

[0077] For the same instruction, RS must be idle earlier than DD. Therefore, in the embodiment of the present invention, a FIFO for storing instruction configuration information is set in DD, so that after RS ​​sends a data fetch request of the current instruction to the local storage component and puts the instruction configuration information into the FIFO in DD, it can continue to process the next instruction without waiting for the local storage component to return the operand. This structure can make full use of the hardware resources of the computing module, tap the parallelism between instructions, and improve the performance of AIE.

[0078] If DD belongs to the first operand loading module, in DD, if and only if there is no address conflict, DD is allowed to send data to the dot product array. After the data is sent, the subsequent calculation is automatically completed by DPA and ACCUM according to the configuration after a fixed period. The specific method for determining the address conflict is as follows: First, DD determines which buffer (represented by the buffer identifier) ​​in the post-processing module the current operand should be stored in after the dot product and accumulation operations, as well as the cache address in the buffer. The buffer identifier is consistent with the bank number given by the LDA instruction. The cache address is obtained based on the initial address given by the LDA instruction and the number of operands sent. Specifically, the initial address is increased by one each time the operand is sent. Secondly, DD controls the number of clock cycles after sending the data to access the cache address according to the calculation mode (whether to accumulate) given by the LDA instruction and the calculation accuracy given by the CFG instruction, so that the access time of the cache address can be determined. Then, it is determined whether the preset buffer access principle is met at the access time. If it is met, it is determined that there is no address conflict with the previous operation. If not, it is determined that there is an address conflict. The buffer access principle may include at least one of the following: two addresses that are both odd / even cannot be read or written at the same time, an address that has an uncompleted read operation cannot be written, and an address that has an uncompleted write operation cannot be read. Based on the above information, the DD can determine whether the current data to be sent has an address conflict with the previous operation, thereby controlling whether the data is actually sent to the DPA.

[0079] Further, such as Figure 7 As shown, the write-back (OCW) module in the engine control module (EC) includes a post-processing interaction module (RA), a second data cache module, a second storage interaction module (DS) and a second scoreboard module.

[0080] A post-processing interaction module (RA), configured to send a data fetch request to the post-processing module based on a write-back instruction;

[0081] A second data cache module, used for caching the calculation results returned by the post-processing module;

[0082] A second storage interaction module (DS), configured to write the calculation results cached in the second data cache module into the local storage component;

[0083] The second scoreboard module is used to record instruction execution status data and send the instruction execution status data to the instruction parsing module, so that the instruction parsing module determines whether to send a write-back instruction to the write-back module based on the instruction execution status data and the buffer occupancy status obtained by other scoreboard modules.

[0084] Specifically, the instruction execution status data of OCW includes: the idle state data of RA and DS in OCW, and the occupancy data of the buffer in the post-processing module required for the write-back instruction. When the INI in the instruction parsing module determines that RA and DS are both idle according to the instruction execution status data of OCW, and no instruction occupies the buffer required for the current write-back instruction, it determines that OCW can receive new write-back instructions. For OCW, if the current ST instruction sends all the data fetch requests to the buffer in POSP, its RA is considered idle and the occupancy of the corresponding buffer ends. If DS sends all the data and writes the answer word, its DS is considered idle. OCW does not occupy the operand register unit in the dot product array.

[0085] Similarly, the embodiment of the present invention realizes memory access merging by OCW inside AIE, merging multiple access requests without conflicts into one request. Specifically, for OCW, the write data bit width between AIE and the local storage component is widened, and the widened read data bit width can be the product of the number of data that the dot product array can provide at a time and the preset maximum data bit width. The preset maximum data bit width refers to the maximum data bit width among the data bit widths of various data formats, such as the data format FP64 has a maximum data bit width of 64. Accordingly, in order to realize the memory access merging, the second storage interaction module (DS) is specifically used to: calculate the write address of each data of the calculation result to be written in the local storage component based on the starting address and address stride data in the write-back instruction; determine whether the write request of each data can be merged according to the write address of each data in the local storage component; if so, merge the write request of each data, and send the merged write request to the local storage component to write each data of the calculation result into the local storage component; otherwise, send the write request of each data to the local storage component separately. Among them, determining whether the write requests of each data can be merged according to the write address of each data in the local storage component includes: determining the bank number corresponding to the write address of each data according to the write address of each data in the local storage component, if the bank numbers corresponding to the write addresses are different, determining that the write requests of each data can be merged into one write request; if there is a write address with the same bank number and row number, determining that the write requests of the upper and lower half areas in the same bank with the write address with the same bank number and row number can be merged.

[0086] Assume that the DPU of the dot product array provides 32 data at a time. If the data format is FP64, 256B of data needs to be written back at a time. In order to ensure the write-back efficiency when the output data format is FP64, the write data bit width between AIE and the local storage component is defined as 256B. However, if the write-back data format is FP8, only 32B of data needs to be written back at a time, and 224B of bandwidth is wasted. In order to avoid bandwidth waste, the following merging method can be used: Taking the case where the ST instruction writes back the result and the write-back accuracy is FP8 as an example, the OCW module in the EC uses the address calculation formula to calculate the write address of each piece of data in the local storage component according to the starting address and address stride data configured by the ST instruction:

[0087] Address 0 = starting address + address granularity * address stride

[0088] Address 1 = Address 0 + Address Granularity * Address Stride

[0089] Address 2 = Address 1 + Address Granularity * Address Stride

[0090] Address 3 = Address 2 + Address Granularity * Address Stride ...

[0092] Address 7 = Address 6 + Address Granularity * Address Stride

[0093] like Figure 8As shown in the figure, when the write-back precision is FP8, the address granularity is 32B, assuming the stride is 1, and the starting address is 0x000. Then the write address of the first data is 0x000, the write address of the second data is 0x020, the write address of the third data is 0x040, and so on, the write address of the eighth data is 0x0e0. For the local storage component, when the lower 8 bits of the address are 0x00 and 0x20, it means bank0; when the lower 8 bits of the address are 0x40 and 0x60, it means bank1; when the lower 8 bits of the address are 0x80 and 0xa0, it means bank2; when the lower 8 bits of the address are 0x0e0 and 0x0c0, it means bank3; when the high bits of the address are the same, it is the same row, and when the high bits of the address are different, it is different rows. Regardless of whether the row numbers of multiple requests are the same, as long as the bank numbers are different, the local storage component considers that these requests can be accessed in a merged manner. In particular, when the lower 6 bits of the address are 0x40, it means the upper half of a bank; when the lower 6 bits of the address are 0x20, it means the lower half of a bank. When the bank number and the row number are the same, the local storage component believes that the upper and lower halves of the same bank can also be merged. Therefore, these eight requests can be merged into one request. It can be seen that by supporting memory access merging in hardware, as long as the user-configured starting address and address stride are reasonable, the efficiency of AIE writing data to the local storage component can be effectively improved.

[0094] In order to meet the requirements of both general high-performance computing power and dedicated AI computing power, the embodiment of the present invention supports input data formats including FP64 / FP32 / FP16 / BF16 / FP8 / INT8. The computing operations in AIE are completed by DPA and POSP.

[0095] Specifically, Fig. 9 As shown, the dot product array (DPA) includes: an operand register unit (OPB ERG), multiple operand selection units (OPA SEL) and multiple data processing units (DPU), and the operand selection unit (OPA SEL) corresponds to the data processing unit (DPU) one by one; illustratively, the DPA includes 32 OPA SELs OPA SEL0-OPA SEL31, including 32 OPA SELs DPU0-DPU31. Among them:

[0096] An operand register unit (OPB ERG), used for registering a second operand sent by an engine control module;

[0097] An operand selection unit (OPA SEL), configured to receive a first operand sent by an engine control module, and send the first operand to a corresponding DPU;

[0098] A data processing unit (DPU) is used to perform a dot product operation on the second operand in the operand register unit and the first operand sent by the operand selection unit.

[0099] Furthermore, in order to realize the sparsification acceleration function in DPA, the operand register unit (OPB ERG) is also used to:

[0100] The sparse index is stored in the case where the sparse acceleration function is turned on; wherein the sparse index is read from the local storage module according to the index address and then written into the operand register unit after the engine control module determines the index address according to the instruction configuration information in the second load instruction;

[0101] The operand selection unit (OPA SEL) is also used to extract a valid first operand from the first operand according to the sparsification index in the operand register unit when the sparsification acceleration function is turned on, and send the extracted first operand to the corresponding DPU.

[0102] Furthermore, the embodiment of the present application uses two operand registers in DPA for ping-pong operation when loading the second operand B and the first operand. Ping-pong operation is a data flow control technology, which is mainly used to achieve seamless buffering and processing in data flow processing. The core idea is to use two data buffer units, through alternating read and write operations, so that the data flow does not pause during the processing process, thereby achieving efficient data processing.

[0103] Specifically, Fig. 9 As shown, the operand register unit (OPB ERG) includes a first register (OPB REG0) and a second register (OPB REG1); accordingly, the engine control module determines whether to send the second operand and the sparse index to the first register or the second register according to the instruction configuration information in the second load instruction; and the operand selection unit determines whether to read the sparse index from the first register or the second register according to the instruction configuration information in the first load instruction.

[0104] Specifically, Fig.10 As shown, the post-processing module (POSP) includes: a buffer unit, a scaling offset unit, a maximum value statistics unit, a general format conversion unit and a special format conversion unit; wherein:

[0105] A buffer unit, used for caching calculation results;

[0106] A scaling offset unit is used to perform a scaling offset operation on the calculation result cached in the buffer unit to obtain a scaling offset result; the input of the scaling offset unit is an internal custom data format, and the output is FP32 / INT32;

[0107] A maximum value statistics unit, used to perform a statistical operation of the absolute maximum value on the scaling offset result and then output the calculation result;

[0108] A general format conversion unit is used to convert the data format (INT32 / FP32) of the scaling offset result into a first data format and then output the calculation result; exemplarily, the first data format may include FP32, FP16, BF16, FP8, INT32 and INT16, etc.

[0109] A dedicated format conversion unit is used to convert the calculation result cached in the buffer unit into a second data format and then output the calculation result; the data bit width of the second data format is greater than the data bit width of the first data format. Exemplarily, the second data format can be FP64, thereby realizing a dedicated conversion function of the FP64 format.

[0110] Further, such as Fig.10 As shown, the post-processing module also includes: multiple accumulation units (ACCUM), and the accumulation units correspond one-to-one to the DPUs in the dot product array; exemplarily, the post-processing module includes 32 ACCUMs, ACCUM0-ACCUM31. Specifically, the accumulation unit (ACCUM) is used to accumulate the calculation result sent by the corresponding DPU with the calculation result cached in the buffer unit when the accumulation mode is configured to be turned on in the first load instruction, and write the accumulated calculation result into the buffer unit; or to perform multiple rounds of accumulation operations when the input data format is a preset data format, and the accumulation operation is to accumulate the calculation result sent by the corresponding DPU with the previous round of calculation results of the accumulation unit. Exemplarily, the preset data format may include FP64 and FP32.

[0111] DPA uses DPU as a computing unit. The input bit width of each DPU is the same and fixed. The number of data contained in a vector is determined by the DPU input bit width and the input data format. The number of vector operations that can be completed by the dot product array at one time is determined by the number of DPUs and the format of the input data. When the input data format is FP64 or FP32, the single data bit width is long. In order to save hardware resources, vector dot product operations in this format need to reuse the logic of DPA and ACCUM to complete them together.

[0112] Taking FP32 mode as an example, the reuse principle is as follows: Assuming that a dot product operation is performed on two vectors, and each vector is composed of 32 FP16-precision data, 32 multipliers with a width of 12 times 12 are required in the dot product calculation operation. Assuming that a dot product operation is performed on two vectors, and each vector is composed of 16 FP32-precision data, 32 multipliers with a width of 13 times 12, 16 multipliers with a width of 13 times 13, and 16 multipliers with a width of 12 times 12 are required in the dot product calculation operation. In order to avoid adding too many multiplier resources, the DPU obtains the dot product calculation results of the first 8 data and the dot product calculation results of the last 8 data through two rounds of operations, and then completes the addition of the above two rounds of calculation results through the addition logic in ACCUM.

[0113] Through the above method, the DPU uses 16 multipliers with a width of 13 times 12, 8 multipliers with a width of 12 times 12, and 8 multipliers with a width of 13 times 13 to support dot product calculations in FP32 mode. Compared with the DPU that only supports FP16 precision, there is no increase in excessive hardware resources due to the increase in support for FP32. Similarly, if support for dot product calculations in FP64 mode is added, a similar multiplexing principle is adopted. At this time, the DPU obtains four results through four rounds of dot product calculations and adds them in ACCUM.

[0114] Furthermore, the embodiment of the present invention uses two buffers in the post-processing module for ping-pong operation when storing and reading the accumulated results. Specifically, the buffer unit includes a first buffer (buffer 0) and a second buffer (buffer 1). The post-processing module determines whether to write the calculation result to the first buffer or the second buffer according to the instruction configuration information in the first load instruction. Specifically, the OAL obtains the buffer identifier and the cache address according to the first load instruction and sends them to the post-processing module. The post-processing module determines whether to write the calculation result to the first buffer or the second buffer according to the buffer identifier, and determines the specific address to write to the first buffer or the second buffer according to the cache address. The engine control module determines whether to read the calculation result from the first buffer or the second buffer according to the instruction configuration information in the write-back instruction.

[0115] POSP implements data addition operations through ACCUM. The calculation results are stored in the buffer, and two buffers 0 / 1 are set for ping-pong pipes. In addition, POSP is equipped with a special scaling offset module, a maximum value statistics module, a general format conversion unit, and a special format conversion unit to realize the output processing of the calculation results. Since the data format conversion when the calculation mode is FP64 is different from the data format conversion when it is in other formats, it is difficult to implement module reuse in hardware, so a format conversion module dedicated to FP64 is set. If AIE receives the ST instruction, it takes out the calculation result from the buffer, and according to the instruction configuration information, performs scaling offset and absolute value maximum statistics on the calculation result, and then converts it into the corresponding data format for output.

[0116] Further, such as Figure 2 As shown, the AIE in the embodiment of the present invention further includes: an input and output (IO) module in the AIE, namely an AIE_IO module, which is used to record error information occurring during operation and record performance data.

[0117] Specifically, the AIE_IO module outputs error information and performance data that occur during the execution of instructions. First, the AIE_IO module has an exception register, a correctable and uncorrectable register, a fault register, a write-set register, and a mask register. By accessing the space address of the AIE_IO module, you can query the abnormal, correctable, uncorrectable and fault conditions inside the AIE. Among them, the exceptions are mainly divided into two categories: configured instruction exceptions (the instruction does not comply with the AIE specification), and accessing a non-existent address when accessing the AIE_IO space, or a write operation request for an unwritable address. Among them, both correctable and uncorrectable errors are generated in the SRAM inside the AIE for storing calculation results (i.e., the buffer as described above). Among them, the fault is generated by the FIFO overflow inside the AIE. Once these errors occur in the AIE, the corresponding registers in the AIE_IO module will be pulled up, and the information when the error occurs will be stored and reported for user debugging. In addition, the low-priority errors can be shielded by accessing the mask register in the AIE_IO module, and errors can be made by configuring the write-set register.

[0118] Secondly, every time AIE receives an instruction, it will send a pulse signal to the outside through the AIE_IO module, indicating that the instruction of this type has been executed once. If AIE is in a non-idle state, it will send a pulse signal to the outside through the AIE_IO module, indicating that AIE has worked for one clock cycle. If the DPA in AIE is in a non-idle state, it will send a pulse signal to the outside through the AIE_IO module, indicating that DPA has worked for one clock cycle. The performance of using AIE can be counted accordingly.

[0119] In summary, the data flow of the operation in AIE is as follows: OPB and OPA enter DPA for dot product operation, and the calculation results are output after a fixed number of delay beats. The number of calculation results is determined by the number of DPUs in DPA. The calculation results can be directly saved to the buffer of POSP, or they can enter ACCUM of POSP and be accumulated with the previous accumulation results saved in the buffer before being saved in the buffer. When data needs to be written back, the data is read from the buffer. The number of data read each time is fixed and determined by the number of DPUs in DPA. According to the write-back instruction configuration information, you can choose whether to perform offset scaling operation on the read data and select the format of the output data. In FP64 mode, the data offset scaling operation is not performed, and the data is directly converted to FP64 format; in other modes, after the optional offset scaling operation, the data is converted to FP32 / FP16 / BF16 / FP8 / INT32 / INT16 format according to the configuration, and you can also choose whether to count the absolute maximum value in the output data according to the configuration.

[0120] The input of each DPU is two fixed-width OPA and OPB vectors, which are divided into 8 / 16 / 32 / 64 groups of OPA and OPB data according to the different data bit widths. The DPU calculates the dot product of these divided data. Since a single shot cannot fully consume all data in FP64 and FP32 modes (for example, a single shot can only consume 2 groups of data in FP64), it is necessary to divide the calculation into multiple shots to complete the dot product operation of all inputs, and then use the accumulator in the post-processing module to realize the addition of multiple partial calculation results.

[0121] In a typical usage scenario (calculating the dot product of OPA and OPB), the order of the instruction flow is CFG→LDB→LDA→ST or CFG→CTN→ST. The instruction flow of LDB→LDA in the above instruction flow can be executed multiple times to complete the accumulation of multiple dot product data. The CTN instruction will be automatically split into multiple LDB→LDA instruction pairs in AIE.

[0122] In fast format conversion and transcendental function mode, the order of the typical instruction flow is CFG→LDA→ST. At this time, there is no need to load OPB, and DPA automatically assigns OPB.

[0123] like Fig.11 , Fig.12 and Fig.13 As shown, a specific working method of an embodiment of the present invention implementing a single convolution operation in a typical application scenario.

[0124] Before calculation, you need to use the LDB instruction to load the weight vector of the convolutional neural network and enter the DPA in pre-loaded form, such as Fig.11As shown in the figure, the DPA consists of 32 DPUs. AIE loads a vector per beat, and loads the data of the same position of 32 convolution kernels 32 times (the position corresponding to 1 in the figure). Finally, the weight vectors of different convolution kernels are deployed to the 32 DPUs respectively.

[0125] After the weight vector is loaded, use the LDA instruction to load the eigenvalue vector and map it to the input feature map as follows Fig.12 As shown, each square corresponds to a vector. The LDA instruction loads eigenvalue vectors from left to right and from top to bottom, and supports two-dimensional stride loading. One LDA can load multiple data. Each time a data is loaded, the data can be sparsely processed to obtain 32 eigenvalue vectors, which are deployed to 32 DPUs respectively. After the eigenvalue vector enters the DPU, the dot product calculation is performed with the preloaded weight value vector. After a fixed period, 32 dot product results are obtained. If the LDA instruction turns on the accumulation mode, the calculation result of the previous instruction is taken out from the buffer, accumulated with the current dot product result, and then stored in the buffer, such as Fig.13 shown.

[0126] The ST instruction is responsible for writing back the calculation results stored in the accumulator buffer. Since the storage format is a custom data format, it needs to undergo format conversion when writing back and finally outputs the data format required by the instruction.

[0127] See also Fig.14 , an embodiment of the present invention provides an electronic device, which includes: a processor (i.e., a local CPU core) and an AI unit external to the processor; wherein the AI ​​unit includes: a local storage component, an internal register, an instruction distribution component, and an AI acceleration engine device in the embodiment of the present application, namely, AIE. Among them, the processor is used to send instructions to the instruction distribution component; the instruction distribution component is used to integrate and parse the instruction information obtained from the processor and the internal register, and if the instruction is determined to be an AI instruction based on the parsing result, the AI ​​instruction is sent to the AI ​​acceleration engine device; the local storage component is used to store data; the internal register is used to store data and instruction information. The internal architecture and operation mode of AIE can refer to the above description.

[0128] In summary, the embodiment of the present invention implements a customized AI extended instruction set based on the RISC-V instruction set, and the instructions are highly flexible and configurable. The AIE of the embodiment of the present invention supports the implementation of this instruction set in hardware, and fully exploits the parallelism between instructions, so as to realize high-performance AIE.

[0129] The embodiment of the present invention pre-decomposes the CTN instruction into a single LDB and LDA instruction during the instruction parsing stage, and then sends the single instruction to the component that executes the corresponding function, thereby simplifying the hardware design and making the hardware structure clearer.

[0130] The embodiment of the present invention reduces the number of accesses to the local storage component by means of memory access merging, improves the utilization rate of the interface between the AIE and the local storage component, and improves the data handling efficiency of the AIE.

[0131] The embodiment of the present invention multiplexes the dot product array and the accumulation unit to increase support for high-performance computing (FP64 / FP32) on the basis of the traditional AI acceleration engine (supporting calculations in multiple data formats such as FP16 / BF16 / FP8 / BF8).

[0132] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0133] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An AI acceleration engine device, characterized in that: The device comprises: The engine control module is configured to receive the AI ​​instruction sent by the instruction distribution component and parse the AI ​​instruction; if the AI ​​instruction is a first load instruction for a first operand, the first operand is loaded from the local storage component and the loaded first operand is sent to the dot product array; if the AI ​​instruction is a second load instruction for a second operand, the second operand is loaded from the local storage component and the loaded second operand is sent to the dot product array; if the AI ​​instruction is a write-back instruction, the calculation result in the post-processing module is written back to the local storage component; A dot product array, used to perform a dot product operation on the first operand and the second operand, and send the calculation result to a post-processing module; The post-processing module is used to perform a post-processing operation on the calculation result of the dot product array to obtain a post-processed calculation result.

2. The device according to claim 1, characterized in that The engine control module includes: an instruction parsing module, configured to receive an AI instruction sent by an instruction distribution component, and parse the AI ​​instruction; if the AI ​​instruction is a first load instruction for a first operand, the first load instruction is sent to a first operand load module; if the AI ​​instruction is a second load instruction for a second operand, the second load instruction is sent to a second operand load module; if the AI ​​instruction is a write-back instruction, the write-back instruction is sent to a write-back module; if the AI ​​instruction is a configuration instruction, the configuration instruction is sent to a configuration module; a first operand loading module, configured to load the first operand from a local storage component based on the first loading instruction, and send the loaded first operand into a dot product array; a second operand loading module, configured to load the second operand from the local storage component based on the second loading instruction, and send the loaded second operand into the dot product array; A write-back module, used for writing the calculation result of the post-processing module back to the local storage component based on the write-back instruction; A configuration module is used to write the configuration information in the configuration instruction into a configuration register, and to classify and forward the configuration information to the first operand loading module, the second operand loading module, the write-back module, the dot product array, and the post-processing module.

3. The device according to claim 2, characterized in that The configuration instruction includes at least one of the following configuration information: Configuration instructions for whether to enable the sparse acceleration function, the input data format, and whether to enable the fast format conversion function.

4. The device according to claim 2, characterized in that The instruction parsing module includes: A first instruction cache module, used for caching AI instructions sent by the instruction distribution component; an instruction processing module, configured to read the AI ​​instruction cached in the first instruction cache module, and if the AI ​​instruction read is a composite load instruction for automatically executing multiple rounds of first load instructions and second load instructions, then disassemble the composite load instruction into multiple first load instructions and multiple second load instructions, and send the disassembled first load instructions and second load instructions to the second instruction cache module; if the AI ​​instruction read is not the composite load instruction, then directly send the AI ​​instruction to the second instruction cache module; A second instruction cache module, used for caching the AI ​​instructions sent by the instruction processing module; An instruction distribution module is used to read the AI ​​instruction cached in the second instruction cache module, identify the instruction type of the read AI instruction, and send the read AI instruction to the first operand loading module, the second operand loading module, the write back module or the configuration module according to the instruction type.

5. The device according to claim 4, characterized in that The instruction distribution module is specifically used for: Reading the AI ​​instruction cached in the second instruction cache module, and identifying the instruction type of the read AI instruction; If the instruction type is a first load instruction, determining whether the first operand load module can receive a new first load instruction according to the instruction execution status data fed back by the first operand load module, the second operand load module and the write-back module, and if so, sending the first load instruction to the first operand load module; If the instruction type is a second load instruction, determining whether the second operand load module can receive a new second load instruction based on the instruction execution status data fed back by the second operand load module and the first operand load module, and if so, sending the second load instruction to the second operand load module; If the instruction type is a write-back instruction, it is determined whether the write-back module can receive a new write-back instruction based on the instruction execution status data fed back by the write-back module and the first operand loading module. If so, the write-back instruction is sent to the write-back module.

6. The device according to claim 2, characterized in that The first operand loading module or the second operand loading module comprises: A first storage interaction module, configured to send a data fetch request for obtaining an operand to a local storage component based on a received load instruction; A first data cache module, used for caching the operands returned by the local storage component; An array interaction module, used for sending the operands cached in the first data cache module into the dot product array; The first scoreboard module is used to record instruction execution status data and send the instruction execution status data to the instruction parsing module, so that the instruction parsing module determines whether to send a load instruction to the first operand loading module or the second operand loading module according to the instruction execution status data.

7. The device according to claim 6, characterized in that The first storage interaction module is specifically used to: When the sparse acceleration function is not enabled, the read address of each piece of data of the operand to be acquired in the local storage component is calculated based on the start address and the address stride data in the load instruction; Determine whether the data fetching requests of each data can be merged according to the read address of each data in the local storage component; If so, the data fetching requests for each piece of data are merged, and the merged data fetching requests are sent to the local storage component to read each piece of data of the operand from the local storage component.

8. The device according to claim 6, characterized in that The array interaction module comprises: A first-in-first-out storage unit FIFO; used to store instruction configuration information in the load instruction put by the first storage interaction module; A data placement unit is used to determine whether the operands cached in the first data cache module can be sent based on the instruction configuration information, and if so, send the operands cached in the first data cache module into the dot product array.

9. The device according to claim 2, characterized in that The write-back module comprises: A post-processing interaction module, used for sending a data fetch request to the post-processing module based on the write-back instruction; A second data cache module, used for caching the calculation results returned by the post-processing module; A second storage interaction module, used for writing the calculation results cached in the second data cache module into the local storage component; The second scoreboard module is used to record instruction execution status data and send the instruction execution status data to the instruction parsing module, so that the instruction parsing module determines whether to send a write-back instruction to the write-back module according to the instruction execution status data.

10. The device according to claim 9, characterized in that The second storage interaction module is specifically used to: Calculate the write address of each piece of data of the calculation result to be written in the local storage component based on the starting address and the address stride data in the write-back instruction; determine whether the write request of each piece of data can be merged according to the write address of each piece of data in the local storage component; If so, the write requests for each piece of data are merged, and the merged write request is sent to the local storage component to write each piece of data of the calculation result into the local storage component.

11. The device according to claim 1, characterized in that The dot product array comprises: an operand register unit, a plurality of operand selection units and a plurality of data processing units DPU, wherein the operand selection units correspond to the DPUs one by one; wherein: The operand register unit is used to register the second operand sent by the engine control module; The operand selection unit is used to receive the first operand sent by the engine control module and send the first operand to the corresponding DPU; The DPU is used to perform a dot product operation on the second operand in the operand register unit and the first operand sent by the operand selection unit.

12. The device according to claim 11, characterized in that The operand register unit is further used for: Storing a sparse index when the sparse acceleration function is turned on; wherein the sparse index is read from the local storage module according to the index address after the engine control module determines the index address according to the instruction configuration information in the second load instruction, and then written into the operand register unit; The operand selection unit is also used to extract a valid first operand from the first operand according to the sparsification index in the operand storage unit when the sparsification acceleration function is turned on, and send the extracted first operand to the corresponding DPU.

13. The device according to claim 12, characterized in that The operand register unit includes a first register and a second register; The engine control module determines to send the second operand and the sparse index to the first register or the second register according to the instruction configuration information in the second load instruction; and the operand selection unit determines to read the sparse index from the first register or the second register according to the instruction configuration information in the first load instruction.

14. The device according to claim 1, characterized in that The post-processing module includes: a buffer unit, a scaling offset unit, a maximum value statistics unit, a general format conversion unit and a special format conversion unit; wherein: The buffer unit is used to cache the calculation results; The scaling offset unit is used to perform a scaling offset operation on the calculation result cached in the buffer unit to obtain a scaling offset result; The maximum value statistical unit is used to perform a statistical operation of the absolute maximum value on the scaling offset result and then output a calculation result; The general format conversion unit is used to convert the data format of the scaling offset result into a first data format and then output the calculation result; The dedicated format conversion unit is used to convert the calculation result cached in the buffer unit into a second data format and then output the calculation result; the data bit width of the second data format is greater than the data bit width of the first data format.

15. The device according to claim 14, characterized in that The post-processing module further includes: a plurality of accumulation units, the accumulation units corresponding one-to-one to the DPUs in the dot product array; The accumulation unit is used to accumulate the calculation result sent by the corresponding DPU and the calculation result cached in the buffer unit when the accumulation mode is configured to be turned on in the first load instruction, and write the accumulated calculation result into the buffer unit; or to perform multiple rounds of accumulation operations when the input data format is a preset data format, and the accumulation operation is to accumulate the calculation result sent by the corresponding DPU and the previous round of calculation results of the accumulation unit.

16. The device according to claim 14, characterized in that The buffer unit includes a first buffer zone and a second buffer zone; The post-processing module determines whether to write the calculation result to the first buffer or the second buffer according to the instruction configuration information in the first load instruction; The engine control module determines whether to read the calculation result from the first buffer or the second buffer according to the instruction configuration information in the write-back instruction.

17. The device according to claim 1, characterized in that The device also includes: The input and output IO module is used to record error information that occurs during operation and record performance data.

18. The device according to any one of claims 1 to 17, characterized in that When executing the convolution algorithm, the first operand is the eigenvalue vector of the feature map, and the second operand is the weight vector of the convolutional neural network.

19. An electronic device, characterized in that: The electronic device comprises: a processor and an AI unit external to the processor; the AI ​​unit comprises: a local storage component, an internal register, an instruction distribution component, and an AI acceleration engine device according to any one of claims 1 to 18; The processor is used to send instructions to the instruction distribution component; The instruction distribution component is used to integrate and analyze the instruction information obtained from the processor and the internal register, and if the instruction is determined to be an AI instruction according to the analysis result, the AI ​​instruction is sent to the AI ​​acceleration engine device; The local storage component is used to store data; The internal register is used to store data and instruction information.

Images