AI acceleration system based on multi-core acceleration card

Abstract

The invention discloses an AI acceleration system based on a multi-core acceleration card, and relates to the field of model acceleration. Comprising a scalar calculation core mounted on an NOC, N matrix calculation cores, a calculation management core, an HBM and a Host CPU. The scalar calculation core is in routing connection with the N matrix calculation cores through NOC; the N matrix calculation cores are connected end to end through NOC; the computing management core receives a task of the Host CPU, sends an instruction and data to the NOC, and sends the instruction and the data to the target computing core through a routing link; the matrix calculation core performs matrix operation based on the received instruction data and outputs intermediate data to the scalar calculation core; and the scalar calculation core performs scalar calculation and data aggregation based on the received instruction and the intermediate data, and returns an acceleration result. According to the scheme, through an NOC routing link and a pipeline cascade communication mechanism, the utilization rate of computing resources can be improved, the inter-core communication efficiency is optimized, and the supporting capacity of an ultra-long text sequence is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of model acceleration, in particular to an AI acceleration system based on a multi-core acceleration card. BACKGROUND

[0002] With the development and progress of large language models, model deployment is also facing new challenges. All models are developing towards larger scale and longer sequence support. For example, the model length has increased from 8K to 160K. These all put higher requirements on computing resources and storage resources, especially the traditional large language model architecture in terms of computing power and delay.

[0003] The related art large model is mainly divided into a multi-core hardware architecture directly connected to a module and a multi-core hardware architecture using a multi-level cache. The first one is a multi-core hardware scheme directly connected to a module. The computing cores interact directly through port signals, and the high-speed memory is mounted on the network on chip. With the increase of computing density, the interface data bit width increases exponentially, resulting in a dramatic increase in wiring pressure, chip area and power consumption. The scalability and portability of the multi-core architecture are severely dependent on specific hardware resources. The second one is a multi-core scheme with multi-level cache, such as GPU architecture. The computing cores interact through the cache hierarchy, but there is a problem of frequent data transfer. The delay is uncontrollable, and it is highly dependent on the performance of the bus bandwidth, and the hardware cost is high.

[0004] These technical solutions generally have the following problems: First, the utilization rate of computing resources is low, and it is difficult to fully exert the advantages of multi-core parallel computing; Second, the communication efficiency between cores is low, the data interaction path is complex, the timing is difficult to converge, and the working clock frequency is limited; Third, the support capability for ultra-long text sequences is insufficient, and it is difficult to meet the needs of modern large language models.

[0005] In addition, these architectures lack efficient mechanisms in terms of task scheduling, data partitioning and result aggregation, resulting in limited overall system performance.

[0006] In view of the above problems, the prior art needs to be improved. SUMMARY

[0007] The embodiments of the present application provide an AI acceleration system based on a multi-core acceleration card, which has the advantages of improving the utilization rate of computing resources, optimizing the communication efficiency between cores, and enhancing the support capability for ultra-long text sequences.

[0008] The system includes a scalar computing core, N matrix computing cores, a computing management core, a high-bandwidth memory HBM, and an external Host CPU. The Host CPU, the computing management core and the HBM are mounted on a network on chip NOC. The scalar calculation core is connected with N matrix calculation cores through the NOC respectively; and the N matrix calculation cores are connected in sequence through the NOC to form a routing link. The calculation management core receives a task sent by a Host CPU, sends instructions and data to the NOC, and sends the instructions and data to a target calculation core through the routing link. The matrix calculation core performs matrix operation based on the received instructions and data, and outputs intermediate data to the scalar calculation core; the scalar calculation core performs scalar calculation and data aggregation based on the received instructions and intermediate data, and returns an acceleration result.

[0009] Specifically, the one-way pipeline cascade communication is performed between the N matrix calculation cores, and a rear-stage matrix calculation core receives output of a front-stage matrix calculation core. The management calculation core inputs a control instruction through an initial matrix calculation core, and transmits the control instruction to the N matrix calculation cores through the one-way pipeline link. The scalar calculation core sends received to-be-calculated data to the initial matrix calculation core, transmits the to-be-calculated data to the N matrix calculation cores through the one-way pipeline link, and receives intermediate data output by the matrix calculation core.

[0010] Specifically, the calculation management core, the scalar calculation core, and the matrix calculation core comprise a pack module and an unpack module. The pack module is used for packaging to-be-calculated data according to an interface type to generate a standard data packet; and the unpack module is used for unpackaging the standard data packet output by a front stage to parse out to-be-calculated data / intermediate data.

[0011] Specifically, the calculation management core further comprises a first gate module, the unpack module parses instructions, data, and corresponding address / port information sent by a Host CPU, and the first gate module filters out data and control instructions for the scalar calculation core and the matrix calculation core respectively. The pack module performs data packaging according to the address / port information respectively, and is mounted on the NOC.

[0012] Specifically, the matrix calculation core further comprises a second gate module, the unpack module parses packaged data output by a front stage, and the second gate module matches address / port information of the calculation core with address / port information corresponding to the front-stage output data; when the matching is successful, the data is directly sent to a core for matrix operation, and is transmitted to a lower stage gradually; when the matching is unsuccessful, the data is directly discarded.

[0013] Specifically, the matrix calculation core is built-in with a DMA module, which reads the cached KV data and model weight from the HBM through the NOC and sends them into the core for matrix operation when receiving a control instruction; the intermediate data generated by the matrix operation is converged to the scalar calculation core.

[0014] Specifically, the calculation management core sends scalar calculation instructions to the scalar calculation core and matrix calculation instructions to the matrix calculation core according to the analysis result, and monitors the data information returned by each calculation core, analyzes the state information and result information in real time; The calculation management core triggers the instructions and data of the next task according to the feedback calculation completion signal; when receiving the result data, the result data is uploaded to the Host side cache area through PCIE for the Host side to use, so that the next group of to-be-calculated vectors are issued until the model inference / generation is completed.

[0015] Specifically, the scalar calculation core is built-in with an operator module and a data reorganization vec2col module, and the operator module includes at least one of a root mean square layer normalization RMSNorm operator, a rotation position encoding ROPE operator, and an activation function SILU operator; The scalar calculation core analyzes and operates the data issued by the calculation management core or the converged intermediate data based on the operator module; and the vec2col module is used to perform vector transposition and reorganization on the converged intermediate data, and cache the KV data generated in the calculation process to the HBM.

[0016] Specifically, the data received by the calculation management core includes to-be-calculated data and scale data; When the received instruction is a reading task for the to-be-calculated data, stop issuing instructions, and actively read the to-be-calculated data and send them to the scalar calculation core in a packaged manner; When the received instruction is a scale data reading task, detect the state of the scalar calculation core, and when it is confirmed to be in an idle state, package and issue the scale data to the matrix calculation core.

[0017] Specifically, the matrix calculation core is built-in with an operator module, including at least one of an RMSNorm operator, an ROPE operator, and an SILU operator; and the scalar calculation core is built-in with a vec2col module; The calculation management core directly inputs data into the initial matrix calculation core, and transmits the data to N matrix calculation cores through a unidirectional flow link; the operator module is activated by the matrix calculation core to perform operation, and the intermediate data generated is converged to the scalar calculation core; and the vec2col module is used to perform vector transposition on the converged intermediate data, and cache the KV data generated in the calculation process to the HBM.

[0018] The technical scheme provided by the embodiment of the application has at least the following beneficial effects: the separation design between the computing cores forms a pipeline division of labor, and the links are all based on the NOC network topology to realize resource distribution and decoupling of the inter-core data packets, so that the occupation of system bandwidth resources can be maximally reduced. In actual operation, only the NOC network needs to be configured, thereby reducing the wiring resource consumption and the wiring difficulty of the EDA tool. The routing links of the matrix computing cores in cascade support the splitting processing of the super-long sequence, thereby reducing the data carrying times. The computing management core realizes the dynamic allocation of task instructions and data, so that the matrix computing core focuses on the parallel matrix operation, the scalar computing core is responsible for data aggregation and nonlinear calculation, and a hierarchical and distributed computing task processing is formed, thereby breaking through the limitation of the single-core memory capacity. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is an architecture diagram of a multi-core hardware scheme in the related art in which modules are directly connected; Figure 2 is an architecture diagram of a multi-core hardware scheme in the related art in which multi-level caches are used; Figure 3 is an architecture diagram of an AI acceleration system based on a multi-core acceleration card provided by the embodiment of the application; Figure 4 shows an AI acceleration system structure diagram in which a transmit unit is added; Figure 5 shows a NOC wiring schematic diagram between computing cores; Figure 6 is an operation block diagram of a typical hybrid expert MoE model; Figure 7 is an instruction processing flowchart written according to the operation sequence of the MoE model; Figure 8 is a multi-core collaborative work schematic diagram according to the instruction processing flowchart and the instruction running; Figure 7 The instruction processing flowchart cooperates with the instruction running. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the application more clear, the embodiments of the application will be further described in detail below with reference to the drawings.

[0021] In this document, “multiple” refers to two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character “ / ” generally represents an “or” relationship between the associated objects before and after it.

[0022] Figure 1is an architecture diagram of a multi-core hardware scheme of a module direct connection in the related art, (matrix) computing core 0 to (matrix) computing core N, and high bandwidth memory (HBM) is mounted on a network-on-chip (NoC), and the scalar computing core is connected to the N computing cores through a high-speed bus. Each computing core accesses the weight data in the HBM and the KV Cache data through the NOC; as the computing density of the computing core increases, the data bit width between the interfaces will also increase by a multiple, and the layout and wiring pressure between the computing cores will also increase by a multiple, resulting in a corresponding increase in the area and power consumption of the chip, and the scalability and portability of the multi-core architecture will be highly dependent on specific hardware resources / frameworks.

[0023] Figure 2 is an architecture diagram of a multi-core hardware scheme using a multi-level cache in the related art, N CUDA cores are mounted on a GPU BUS high-speed bus through respective L1 caches and L2 caches, and data interaction with the HOST side. There is no specific port for interconnection between the GPU CUDA cores, and the data interaction between the cores is through the L1 / L2 cache. However, this architecture will frequently perform DMA data transfer, and the delay is uncontrollable. Each core needs to transfer the data required for calculation from the host bus to the L2 cache first, and then transfer the data required for the corresponding computing density to the L1 cache for calculation. The entire architecture is highly dependent on the bandwidth performance of the GPU BUS, and the entire GPU architecture is difficult to reproduce on other hardware, is highly dependent on the existing GPU architecture, and the hardware cost of the scheme is also relatively high.

[0024] Based on the problems in the related art, the present application provides an AI acceleration system based on a multi-core acceleration card, as shown in Figure 3 The system architecture includes a scalar computing core, N matrix computing cores, a computing management core, a high-bandwidth memory HBM, and an external Host CPU. The Host CPU, the computing management core, and the HBM are directly mounted on the network-on-chip NOC. Considering that the bus connection method is used for inter-core interaction in the traditional scheme, the system bandwidth is occupied, so the present application will make the most of the NOC network, and the scalar computing core is connected to the N matrix computing cores through the NOC link. The connection method of the N matrix computing cores is also improved, and the NOC network is used for connection in sequence to form a routing link.

[0025] The external Host CPU is not within the protection scope of the system, and only serves as a central controller for issuing tasks, responsible for initiating computing tasks and receiving final results.

[0026] The computing management core is a control unit for task scheduling and instruction distribution in the architecture, responsible for receiving various inference / acceleration tasks issued by the external Host CPU, parsing host instructions, scheduling and monitoring global acceleration cores, dynamically allocating computing resources, and feeding back task processing progress and results to the Host CPU. It can be implemented using a reduced instruction set processor combined with a state machine.

[0027] In some embodiments, each computing unit within the system also has a transmit unit / module, which enables data transmission between the computing core and the NOC network. As shown in Figure 4 The AI acceleration system structure diagram with a transmit unit, Figure 4 and Figure 3 In terms of structure, the routing links between the matrix computing cores, and the links with the HBM are all part of the NOC network.

[0028] When the computing management core receives a task issued by the Host CPU, it needs to send instructions and data to the NOC, and through the routing link, send the instructions and data to the target computing core, so that the corresponding computing core can perform the corresponding operation. The data here can be the data after task partitioning, and the partitioned data can be of various types. There are multiple data partitioning methods for large model computing, such as the commonly used DP (Data Parallelism) partitioning method, which partitions vector data into different computing cores and performs matrix operations with the same weight data.

[0029] The matrix computing core and the scalar computing core are used to perform typical matrix operations and scalar operations, respectively, and are the key computing units for model acceleration / inference. Their specific structures are not within the scope of protection of the present application, and will not be described here. As a more fine-grained execution system, the N matrix computing cores perform matrix operations based on the received instructions and data, and output intermediate data. The scalar computing core, as a small control core, is responsible for receiving and integrating the intermediate data output by the N matrix computing cores, such as aggregating the intermediate data and performing necessary scalar operations, and outputting the result data to the Host CPU level by level.

[0030] In summary, the separation design between the computing cores forms a pipelined division of labor, and the links are all based on the NOC network topology to realize the resource distribution and decoupling of the inter-core data packets, so that the occupation of the system bandwidth resources can be minimized. In actual operation, only the NOC network needs to be configured, thereby reducing the wiring resource consumption and the wiring difficulty of the EDA tool. The routing link of the matrix computing core cascaded at the head and tail supports the splitting processing of the super-long sequence, thereby reducing the data carrying times. The computing management core realizes the dynamic allocation of the task instructions and data, so that the matrix computing core focuses on the parallel matrix operation, the scalar computing core is responsible for the data aggregation and nonlinear calculation, a hierarchical and distributed computing task processing is formed, and the limitation of the single-core memory capacity is broken through.

[0031] In order to reduce the wiring resource pressure and system complexity of all computing cores end to end, in some embodiments, the application can adopt a one-way pipelined cascading communication wiring mode between the N matrix computing cores. As shown in FIG. 6, the matrix computing core 0 is connected to the matrix computing core 1 through the routing link 0, the matrix computing core 1 is connected to the matrix computing core 2 through the routing link 1, and so on, until the matrix computing core N is connected to the scalar computing core through the routing link N-1. Figure 5 The NOC wiring schematic diagram between the computing cores is shown in FIG. 6. Between the matrix computing cores, the rear matrix computing core receives the output of the front matrix computing core. For the management computing core, it can input the control instructions to the chain structure through the initial matrix computing core (matrix computing core 0), and the instructions are transmitted to the N matrix computing cores through the one-way pipelined link. The separate scalar computing core directly inputs the data and instructions based on the straight chain.

[0032] Because the scalar computing core and the N matrix computing cores form a star connection, in terms of data flow, a plurality of modes can be used for transmission, including: 1. The management computing core distributes the data in two ways The data for the scalar computing core is directly sent to the scalar computing core, and the data for the matrix computing core is input through the matrix computing core 0, which is sequentially transmitted to the rear through the one-way pipelined cascading path.

[0033] 2. The management computing core distributes the data in one way All data is sent to the scalar computing core, and the data related to the matrix operation is distributed through the matrix computing core 0, which is sequentially transmitted to the rear through the one-way pipelined cascading path.

[0034] The above two schemes are the distribution modes of the management computing core for the data. The two modes are determined based on the deployment position of the model operator. When the model operator is deployed in the scalar computing core, the related to-be-calculated data needs to be preliminarily operated through the scalar computing core and then sequentially distributed. When the model operator is deployed in the matrix computing core, it does not need to be distributed to the scalar computing core first, but can be directly distributed through the one-way pipelined cascading path.

[0035] In the dual-path mode, a one-way transmission mode is adopted between the scalar calculation core and the N matrix calculation cores, that is, the scalar calculation core receives the intermediate data calculated and output by the N matrix calculation cores. In the single-path mode, a bidirectional transmission mode is adopted between the scalar calculation core and the initial matrix calculation core, and a unidirectional pipeline transmission mode is adopted between the remaining cores. When performing the inference task, the scalar calculation core receives the intermediate data calculated and output by the N matrix calculation cores.

[0036] In some embodiments, in order to realize data routing and forwarding between the cores, a pack module and an unpack module are included in the computation management core, the scalar calculation core, and the matrix calculation cores. The pack module is used to encapsulate the to-be-calculated data according to the interface type to generate a standard data packet, and the unpack module is used to unpack the standard data packet output by the previous stage to parse the to-be-calculated data / intermediate data.

[0037] The address / port information can include a specific numerical identifier. Whenever the computation management core or the matrix calculation core receives data, the data is unpacked by the pack module and subjected to data screening and operation. For data that needs to be passed back or distributed, the data is encapsulated according to the interface type, such as an AXI Stream interface form data packet, and then mounted on the NOC as a standard data packet. For example, the computation management core sends instruction data and initial vector data to the NOC, and the pack module fills the routing information into the port ID signal according to the instruction content. The NOC forwards the instruction data and vector / sequence data to the corresponding calculation core according to the ID of the interface. The processing process of the matrix calculation core and the scalar calculation core is similar.

[0038] In some schemes of the present application, the instruction and data sent by the Host CPU are mixed and transmitted, and the control instruction and the to-be-processed data for the scalar calculation core and the matrix calculation core cannot be effectively distinguished, resulting in the problems of unclear routing information and low data classification efficiency when the NOC network forwards. The first gate module is further built in the computation management core. The unpack module parses the instruction, data, and corresponding address / port information sent by the Host CPU, and the first gate module screens the data and control instruction for the scalar calculation core and the matrix calculation core, respectively. The corresponding pack module encapsulates the data according to the address / port information, and mounts the data on the NOC, respectively.

[0039] When the Host CPU issues a mixed data stream, the unpack module first extracts the address field in the data packet header for analysis. The first gate module activates the corresponding data channel based on the address resolution result, for example, routing the control instruction data packet carrying the A address to the scalar calculation core processing queue, routing the control instruction data packet carrying the B address to the matrix calculation core processing queue, and routing the to-be-calculated data packet carrying the C address to the scalar calculation core input buffer. When the pack module re-encapsulates the classified data, the routing label recognizable by the NOC is added to the packet header. The NOC directs the control instruction packet to the initial matrix calculation core physical port through the crossbar matrix according to the target port information in the routing label, and the to-be-calculated data packet is transmitted to the scalar calculation core through an independent channel.

[0040] This classification transmission mechanism based on address / port information realizes accurate classification and directional transmission of instructions and data, so that the control instruction and data are isolated in the physical transmission layer, and network congestion caused by mixed data streams is avoided.

[0041] In the foregoing embodiment, for the matrix calculation core, the unpacked data may contain information that does not belong to the processing range of the current matrix calculation core. If such irrelevant data is directly processed or transmitted, it will cause waste of calculation core resources, reduction of pipeline communication efficiency, and even calculation errors caused by data misprocessing.

[0042] Therefore, the second gate module is further arranged in the matrix calculation core, the unpack module analyzes the encapsulated data output by the previous stage, and the second gate module matches the address / port information of the calculation core with the address / port information corresponding to the data output by the previous stage; when the matching is successful, the data is directly sent to the core for matrix operation and transmitted to the next stage; when the matching is unsuccessful, the data is directly discarded.

[0043] The address / port matching mechanism of the second gate module can be realized by a hardware logic circuit, for example, by configuring a register to store the physical address or virtual port identifier of the current calculation core, and performing bit-by-bit comparison with the target address field in the data packet header extracted by the unpack module. The matching logic can support exact matching or mask matching mode, and the mask bit width can be set to 8 to 32 bits, depending on the demand of the address coding space. The transmission mechanism is realized by data path bypass design, when the matching is successful, the data is copied into two paths, one path enters the calculation core, and the other path directly passes through the pipeline register to the next calculation core, and the transmission delay can be controlled within 1 to 3 clock cycles. The data discard mechanism cuts off the invalid data flow to the calculation core and the next stage link through the gate switch, and the discard operation can be completed within the first clock cycle after the unpack stage, avoiding the occupation of cache resources by invalid data.

[0044] Specifically, after the unpacking module completes data analysis, the second gating module immediately starts address / port matching detection. The preset address information of the current computing core and the target address information of the data packet are compared in real time by a comparator. If the two are consistent, an enable signal is generated to open the data input channel of the computing core, and a transparent channel is triggered to push a copy of the data to the next-level computing core. If the address is not matched, all data channels are immediately closed and the current data packet is marked as invalid. The hardware controller clears the relevant registers and cache content in the next clock cycle. This process cooperates with the packaging / unpacking module in the pre-claimed right. The packaging module has embedded target address information in the data packaging stage, and the unpacking module extracts key routing parameters in real time during link transmission, so that the second gating module can quickly complete the validity judgment before the data enters the computing core. By integrating address matching judgment and data transparent operation inside the computing core, the defect of needing additional routing nodes for data screening in the traditional scheme is avoided, the communication efficiency of the multi-core pipeline is greatly improved, and the calculation error rate caused by misprocessed data is greatly reduced.

[0045] When the matrix computing core performs matrix operation, KV cache and model weight parameters and other data can be extracted from the HBM according to the control instruction. For this purpose, a DMA module can also be built in the matrix computing core. When receiving the control instruction, the DMA module reads the cached KV data and model weight from the HBM through the NOC link and sends them to the core for matrix operation. The intermediate data generated by the matrix operation is then aggregated to the scalar computing core.

[0046] Although the foregoing embodiment proposes a scheme in which the computing management core transmits instructions and data to the matrix computing core through a unidirectional flow link to realize task scheduling. However, in this process, if there is a lack of real-time monitoring mechanism for the state of the computing core, the task issuing rhythm cannot be dynamically adjusted, resulting in poor task synchronization between computing cores, low result data return efficiency, difficulty in realizing task-level flow control and dynamic resource allocation, and affecting the throughput and response delay of the overall acceleration system.

[0047] Therefore, the computing management core needs to send scalar computing instructions to the scalar computing core and matrix computing instructions to the matrix computing core according to the analysis result, and monitor the data information returned by each computing core in real time to analyze the state information and result information.

[0048] The computing management core triggers the instructions and data of the next task according to the feedback of the computing completion signal; when receiving the result data, the result data is uploaded to the Host-side cache area through PCIE for use by the Host side, so that the next group of vectors to be calculated is issued until the model inference / generation is completed.

[0049] In a possible implementation, after parsing the task instructions issued by the Host side, the computing management core classifies the instructions into scalar computing instructions and matrix computing instructions according to the instruction types. The scalar computing instructions are transmitted through the NOC routing path configured as the scalar computing core exclusive address, and the matrix computing instructions are transmitted through the unidirectional pipeline link of the initial matrix computing core. The real-time monitoring mechanism is implemented by parsing the state field in the data returned by the computing core, for example, embedding a state code containing the current load state of the computing core in the header of the data packet. When the state information indicates that the scalar computing core is idle, the computing management core can immediately issue the scalar computing task to be processed; if the matrix computing core feeds back a busy state, the issue time of the subsequent matrix task is delayed. The feedback computing completion signal triggers the task scheduler to extract the next group of computing vectors from the queue and transmit them to the corresponding computing core through the NOC routing link.

[0050] When the result data is uploaded through the PCIE interface, the zero-copy technology is used to directly map the data from the output buffer of the computing core to the Host side memory space, avoiding the delay caused by the intermediate cache. The task-level pipeline control is realized by dynamically adjusting the instruction issue rhythm of the computing management core, for example, after detecting that the scalar computing core completes data aggregation, the matrix computing core is triggered to start the operation of the next stage. The dynamic allocation of resources is embodied in adjusting the resource proportion of different types of computing tasks according to the real-time state information, for example, when the load of the matrix computing core is too high, the scalar computing task is preferentially allocated to balance the system resource utilization.

[0051] It should be noted that after the scalar computing core performs data aggregation, data restoration needs to be performed according to the task parallelism, so that the data format can be calculated in the next step. For example, two matrix multiplications A(1x16)*B(16*32), the vector A is the initial vector, and the vector B is divided into 16 parts in the row direction according to the parallelism 16, and is sent into the matrix core for calculation, and the calculated data is aggregated to the scalar computing core for recombination.

[0052] This feedback-based task triggering mechanism and optimized data return path jointly construct an efficient task-level pipeline control system, realize dynamic allocation and optimized utilization of computing resources. Therefore, the application significantly improves the throughput of the overall acceleration system and reduces the system response delay, and is particularly suitable for inference and generation scenarios of super-long sequence models.

[0053] The embodiment takes the built-in operator module in the scalar calculation core as an example. The operator module can include at least one of RMSNorm (Root Mean Square Layer Normalization) root mean square layer normalization operator, ROPE (Rotary Position Embedding) rotary position embedding operator, and SILU (Sigmoid-Weighted Linear Unit) activation function operator. In addition, the scalar calculation core also has built-in data gathering gather module and data reorganization vec2col module, which are respectively used to gather the intermediate data output by each matrix calculation core and transpose and reorganize the intermediate data (the specific reorganization can be realized based on the remap unit therein). The vec2col module converts a one-dimensional vector into a two-dimensional matrix through a row and column permutation operation, for example, converts a vector of length L into an MxN matrix form, where the values of M and N are determined according to the input interface bit width of the matrix calculation core.

[0054] After receiving the instruction data and the vector (sequence) data to be calculated issued by the calculation management core, the scalar calculation core analyzes the instruction content issued and activates the related operator module for operator processing and operation, and then distributes the vector data that needs to be multiplied by the matrix to the initial matrix calculation core after processing; or analyzes and operates the gathered intermediate data.

[0055] In some other embodiments, the data received by the calculation management core includes to-be-calculated data and scale data; when it is detected that the received instruction is a reading task of the to-be-calculated data, the instruction issuing is stopped, the to-be-calculated data is actively read and packaged and sent to the scalar calculation core through the pack module; When it is detected that the received instruction is a scale data reading task, the state (Busy / Idel) of the scalar calculation core is detected, and when it is confirmed that the state is idle, the scale data is packaged and issued to the matrix calculation core through the pack module.

[0056] In model inference calculation, The scale data generally refers to a normalization parameter or a scaling scale parameter used to adjust the output of a model, used to control the scaling ratio of the output result, ensure numerical stability and prediction accuracy. When the calculation management core receives a to-be-calculated data reading instruction, the instruction issuing channel is immediately closed, for example, the instruction transmission path is cut off by a gating circuit, while the DMA starts reading the to-be-calculated data from the HBM. After the data reading is completed, the data is packaged into a 512bit / 64Byte data packet through the AXI Stream interface, and is transmitted to the input queue of the scalar calculation core through the routing link of the NOC. When the scale data reading instruction is received, the state register of the scalar calculation core is first queried, if the state identifier is "Idle", the scale data is packaged in a 64-byte block structure, and the target port information of the matrix calculation core is attached, and is sent to the configuration interface of the matrix calculation core through the NOC. If the state identifier is "Busy", a retry mechanism is started, for example, the state is detected again every 10 clock cycles until the scalar calculation core releases the resource and then the data issuing is performed. By distinguishing the processing logic of the to-be-calculated data and the scale data, the path conflict of the two data types in the transmission process is avoided, for example, to prevent the input buffer overflow of the scalar calculation core when receiving the to-be-calculated data and the scale data at the same time. At the same time, the state detection mechanism ensures that the scale data is transmitted only when the system resource is idle, reducing the instruction queuing delay of the matrix calculation core caused by resource competition. The target address identifier added in the data packaging process matches the routing strategy of the NOC, ensuring that the data packet can be accurately delivered to the specified interface of the target calculation core. Therefore, the system can maintain stable data distribution efficiency in a high-concurrency task environment.

[0057] When the operator module of model inference is deployed in the matrix calculation core, the calculation management core can directly input the (split) data into the initial matrix calculation core, pass it to the N matrix calculation cores through the one-way flow link, and directly activate the operator module for operation by the matrix calculation core, and the generated intermediate data is aggregated to the scalar calculation core. Correspondingly, the scalar calculation core aggregates, vector transposes, and reorganizes the result data of each matrix calculation core, and caches the KV data generated in the calculation process to the HBM. The difference between the two modes is mainly the deployment position of the instruction, and two different data transmission schemes can be designed according to the demand and fine-grained division, and the rest of the model acceleration and intermediate data aggregation process remains unchanged.

[0058] The following is illustrated by a specific embodiment, Figure 6 is a typical mixed expert MoE model operation block diagram, Figure 7is an instruction processing flowchart written according to the MoE model operation sequence, the content and sequence of the executed instructions are coded and implemented according to the large model calculation framework, and the instruction content can be modified according to the deployment model, the modification of the calculation core operator module, and the inter-module connection relationship to adapt to the operation process of the corresponding model. For the above-mentioned model instruction processing flowchart, the AI acceleration system of the present application is used for execution, taking 16 matrix cores and 8 batch support in the matrix core as an example, Figure 8 is according to Figure 7 The instruction processing flow cooperates with the multi-core collaborative work schematic diagram of the instruction running, including the following steps: Data line ①: First, the calculation management core parses the instruction information, sends the parsed instruction data to the pack module for packaging, transmits it to the scalar calculation core through the NOC, and then reads out the initial token vector (sequence) data, packages it through the pack module, and routes it to the scalar calculation core through the NOC.

[0059] After receiving the instruction data, the scalar calculation core parses the content, activates the corresponding operator module to enter the calculation state, and performs operator module calculation on the vector data. The calculation result is directly transmitted to the matrix calculation core for caching according to the instruction routing information, or the final result is uploaded to the Host side by the calculation management core.

[0060] Data line ②: First, the calculation management core parses the instruction information, sends the parsed instruction data to the pack module for packaging, and routes it to the matrix calculation core through the NOC. The matrix calculation core parses the instruction and starts DMA for weight data reading after receiving the DMA start signal. The read weight data is synchronized with the vector data in the core cache unit, and is sent to the matrix calculation module in the core for matrix multiplication operation. The calculation result is also routed to the scalar calculation core according to the instruction content.

[0061] Data line ③: One-way path, the matrix calculation cores are connected in series according to the flow structure, so as to realize the sharing of instruction data and vector / sequence data among the matrix calculation cores. The matrix calculation core receives the instruction data issued by the management core and the calculation result of the scalar calculation core, and transmits it to the next level matrix calculation core in the flow structure after verification.

[0062] Data line ④: Same as data line ③, the vector / sequence data and instruction data sharing path between the upper and lower matrix cores.

[0063] Data line ⑦: bidirectional path; direction 1: matrix calculation core -> scalar calculation core, the scalar calculation core receives the calculation results (intermediate data) of each matrix calculation core one by one, and rearranges the calculation results according to the batch number supported by the matrix calculation core, and then outputs to the corresponding operator module for calculation according to the instruction routing content, in addition to the above data rearrangement function, the scalar calculation core will also perform row-column transposition on the vector (sequence) data, for example, when performing the S*V process in Figure 7 , the data V needs to be transposed, and the transposed data is sent to the operator module for corresponding processing, and the calculation result is then transmitted to the matrix calculation core / computation management core according to the instruction routing information; Direction 2: scalar calculation core -> matrix calculation core, before performing the matrix multiplication process, the scalar calculation core transmits the intermediate data and to-be-calculated data to the matrix calculation core through the NOC, and then writes the corresponding batch of cache units according to the instruction content for the matrix calculation core to read and use; a plurality of matrix calculation cores and scalar calculation cores form a star-shaped calculation structure as shown in Figure 5 , the calculation result of the scalar calculation core is output to the matrix calculation core 0 through the NOC, and then passed down in turn through the Figure 5 pipeline structure between matrix calculation cores.

[0064] In summary, the technical effects brought by the AI acceleration system based on the multi-core acceleration card of the present application include the following: 1. Flexible architecture, convenient for transplantation and expansion: each calculation core encapsulates a standard interface, and the various cores are interconnected through the NOC, and the single calculation core resource and the number of calculation cores can be flexibly matched according to the resources of the hardware device; it is conducive to the UT / ST design and verification of the entire system architecture; 2. Improve hardware resource utilization: connect each calculation core and memory together through the NOC, reduce the number of signals and signal interaction between each calculation core, and the wiring resources brought by the signal interaction, each calculation core can share the memory, and the calculation core access memory bandwidth can be improved through the NOC dedicated path, make full use of the number of nodes provided by the NOC, so as to improve the layout and routing flexibility of EDA tools, so that the running frequency of the entire scheme and the performance of the hardware product are higher.

[0065] 3. High model calculation performance: the wiring of the entire architecture is simple, so the working clock frequency will be higher, the model calculation time will be correspondingly reduced, and the hardware can achieve higher calculation performance (throughputs).

[0066] 4. Support for deployment of multiple large models: according to the calculation resources of the hardware, different numbers of calculation cores can be matched to expand the computing power scale of the entire system; according to the different model calculation processes, corresponding hardware instructions can be written to complete the deployment of different large models on hardware.

[0067] The specific embodiments are only illustrative of the present application, and are not intended to limit the present application. Those skilled in the art can make modifications to the embodiments without creative contribution after reading the specification, and the modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

Claims

1. An AI acceleration system based on a multi-core accelerator card, characterized in that, It includes a scalar computation core, N matrix computation cores, a computation management core, a high-bandwidth memory (HBM), and an external host CPU; the host CPU, the computation management core, and the HBM are mounted on an on-chip network (NOC). The scalar computation core is routed to N matrix computation cores via the NOC; and the N matrix computation cores are connected end-to-end via the NOC to form a routing link; The computing management core receives tasks sent by the Host CPU, sends instructions and data to the NOC, and then sends them to the target computing core through a routing link; The matrix computation core performs matrix operations based on the received instructions and data, and outputs intermediate data to the scalar computation core; The scalar computation core performs scalar computation and data aggregation based on the received instructions and intermediate data, and returns accelerated results.

2. The AI ​​acceleration system based on a multi-core accelerator card according to claim 1, characterized in that, Unidirectional pipelined cascaded communication between the N matrix computation cores, with the subsequent matrix computation core receiving the output of the preceding matrix computation core; The management computing core inputs control commands through the initial matrix computing core, which are then transmitted to the N matrix computing cores via a one-way pipeline. The scalar computation core sends the received data to be computed to the initial matrix computation core, transmits it to N matrix computation cores through a unidirectional pipeline, and receives intermediate data output by the matrix computation cores.

3. The AI ​​acceleration system based on a multi-core accelerator card according to claim 1, characterized in that, The computation management core, the scalar computation core, and the matrix computation core each contain a pack module and an unpack module. The pack module is used to encapsulate the data to be calculated according to the interface type and generate a standard data packet; the unpack module is used to decapsulate the standard data packet output from the previous stage and parse out the data to be calculated / intermediate data.

4. The AI ​​acceleration system based on a multi-core accelerator card according to claim 3, characterized in that, The computing management core also has a built-in first gating module. The unpack module parses the instructions, data and corresponding address / port information sent by the host CPU, and filters out the data and control instructions for the scalar computing core and the matrix computing core through the first gating module respectively. The pack module encapsulates data according to address / port information and mounts it onto the NOC.

5. The AI ​​acceleration system based on a multi-core accelerator card according to claim 3, characterized in that, The matrix calculation core also has a built-in second gate module. The unpack module parses the encapsulated data output from the previous stage, and the second gate module matches the address / port information of the calculation core with the address / port information corresponding to the output data of the previous stage. When the match is successful, the data is directly sent into the core for matrix operation and passed through to the next stage. When the match is unsuccessful, the data is discarded.

6. The AI ​​acceleration system based on a multi-core accelerator card according to any one of claims 1-5, characterized in that, The matrix calculation core has a built-in DMA module. When receiving control commands, the DMA module reads the cached KV data and model weights from the HBM through the NOC and sends them to the core for matrix operations. The intermediate data generated by the matrix operations is aggregated to the scalar calculation core.

7. The AI ​​acceleration system based on a multi-core accelerator card according to claim 2, characterized in that, Based on the parsing results, the computation management core sends scalar computation instructions to the scalar computation core and matrix computation instructions to the matrix computation core, and monitors the data information returned by each computation core in real time, parsing status information and result information. The computing management core triggers the instructions and data for the next task based on the feedback signal indicating that the computing is complete. When the result data is received, it is uploaded to the host-side cache via PCIe for the host to use, so that the next set of vectors to be calculated can be sent out, until the model inference / generation is completed.

8. The AI ​​acceleration system based on a multi-core accelerator card according to claim 2, characterized in that, The scalar computation kernel has built-in operator modules and data recombination vec2col modules. The operator modules include at least one of the following: root mean square layer normalization (RMSNorm) operator, rotation position encoding (ROPE) operator, and activation function (SILU) operator. The scalar computation core performs parsing and computation on the data issued by the computation management core based on the operator module, or on the aggregated intermediate data; it performs vector transposition and recombination on the aggregated intermediate data based on the vec2col module, and caches the KV data generated during the computation process to the HBM.

9. The AI ​​acceleration system based on a multi-core accelerator card according to claim 8, characterized in that, The data received by the computing management core includes data to be computed and scale data; When the received instruction is a task to read data to be computed, stop issuing instructions, actively read the data to be computed, package it and send it to the scalar computing core; When the received instruction is a scalar data read task, the status of the scalar computation core is checked. If the idle state is confirmed, the scale data is packaged and sent to the matrix computation core.

10. The AI ​​acceleration system based on a multi-core accelerator card according to claim 8, characterized in that, The matrix computation kernel has a built-in operator module, including at least one of the RMSNorm operator, ROPE operator, and SILU operator; the scalar computation kernel has a built-in vec2col module. The computation management core directly inputs data into the initial matrix computation core, which then transmits it to N matrix computation cores via a unidirectional pipeline. The matrix computation cores directly activate the operator modules to perform calculations, and the resulting intermediate data is aggregated into the scalar computation core. The scalar computation core performs vector transposition and recombination on the aggregated intermediate data based on the vec2col module, and caches the KV data generated during the computation process into the HBM.

Images