Single-core processing core and control method therefor, many-core communication architecture, and chip

Abstract

Disclosed in the present invention are a single-core processing core and a control method therefor, a many-core communication architecture, and a chip. Each single-core processing core comprises a network-on-chip interface for external data transmission and a memory access interface; the inter-core data transmission is carried out on the network-on-chip interface; the single-core processing core comprises a compute engine, a control unit, a data exchange unit, a parameter RAM, an NI unit, a DMA unit, a queue transmission unit, a weight gradient transmission unit, a data exchange bus, and a control bus; the queue transmission unit is used for realizing capacity expansion and acceleration of a hardware queue between forward and backward directions; the weight gradient transmission unit is used for providing transmission acceleration for the frequent reading and writing of weight gradients of a deep neural network (DNN) in a DRAM; and the weight gradients are stored in the DRAM so as to accelerate gradient read / write access of weight gradient calculation. Therefore, the embodiments of the present invention can improve the transmission performance and the energy efficiency of inter-core communication, and reduce the DNN training data transmission time delay.

Description

Single-core processing core and its control method, many-core communication architecture, chip Technical Field

[0001] Embodiments of the present invention relate to the technical field of chip design, and in particular to a single-core processing core and a control method thereof, a many-core communication architecture, and a chip. Background Art

[0002] Among related technologies, the current mainstream DNN (Deep Neural Networks) training platform uses GPU (Graphics Processing Unit) chips. The advantages of GPU are mainly reflected in the acceleration and high parallelism of calculations, and it has strong versatility in operators and communication. However, the GPU is based on a shared memory transmission architecture, and uses multi-level cache and cache consistency protocols to achieve inter-core communication. It cannot adapt well to the characteristics of DNN training data flow and static model deployment. Its transmission latency and energy efficiency still have a lot of room for optimization.

[0003] Summary of the Invention

[0004] Embodiments of the present invention provide a single-core processing core and its control method, a many-core communication architecture, and a chip, which can improve the transmission performance and energy efficiency of inter-core communication and reduce the data transmission delay of DNN training.

[0005] In the first aspect, an embodiment of the present invention provides a many-core communication architecture, wherein the single-core processing core has an on-chip network interface and a memory access interface for external data transmission, and the single-core processing core internally includes a computing engine, a control unit, a data exchange unit, a parameter RAM, an NI unit, a DMA unit, a queue transmission unit, a weight gradient transmission unit, a data exchange bus and a control bus, and the control unit is responsible for the activation of computing instructions, memory transfer instructions and network transmission instructions, the computing instructions are pre-arranged and stored in the computing engine, the memory transfer instructions are pre-arranged and stored in the DMA unit, the memory transfer instructions include DMA write instructions and DMA read instructions, the network transmission instructions are pre-arranged and stored in the NI unit, and the network transmission instructions include NI send instructions and NI receive instructions The control unit is electrically connected to the computing engine, the NI unit, the DMA unit, and the queue transmission unit through the control bus. The data exchange unit is electrically connected to the parameter RAM, the NI unit, the DMA unit, the queue transmission unit, and the weight gradient transmission unit through the data exchange bus. The queue transmission unit is used to achieve hardware queue expansion and acceleration between forward and backward directions. The weight gradient transmission unit is used to provide transmission acceleration for the frequent reading and writing of DRAM memory for weight gradients of deep neural networks. The weight gradients are stored in the DRAM memory. The control unit triggers the queue transmission unit to generate queue transmission instructions through a control signal, and the computing engine triggers the weight gradient transmission unit to generate weight gradient transmission instructions through a control signal.

[0006] In some embodiments, the NI unit has the on-chip network interface, and the NI unit sends the data of the parameter RAM in the source single-core processing core to the destination single-core processing core according to the NI send instruction, and writes the data received from the destination single-core processing core to the parameter RAM in the source single-core processing core according to the NI receive instruction. The paired NI send instructions and the NI receive instructions are associated by the same tag number.

[0007] In some embodiments, the DMA unit is used to implement data transmission between the parameter RAM and the DRAM memory, the DMA write instruction and the DMA read instruction contain the data type code, source address information and target address information corresponding to the parameter RAM, the source address information of the DMA write instruction and the target address information of the DMA read instruction are used to indicate the internal offset of the parameter RAM, and the data exchange unit accesses the parameter RAM according to the data type code, the source address information and the target address information.

[0008] In some embodiments, the queue transfer unit provides a first-in-first-out hardware queue mechanism between forward and backward directions of training to accelerate data transfer and provides a configurable queue depth in DRAM memory.

[0009] In some embodiments, the computing engine forms an interlocking cooperation with the weight gradient transmission unit.

[0010] In some embodiments, the data exchange unit is used to provide data exchange between the parameter RAM and outside the core. The data exchange unit adopts ID addressing in the single-core processing core to unify data scheduling of network and memory access.

[0011] In a second aspect, an embodiment of the present invention provides a many-core communication architecture having at least two single-core processing cores as described in the first aspect, wherein the on-chip network interface is used for data transmission between any two of the single-core processing cores, the on-chip network interface is electrically connected to an on-chip router, and the on-chip router is used for data forwarding between any two of the single-core processing cores, the single-core processing cores exclusively occupy a dynamic random access memory DRAM space, and the memory logic between any two of the single-core processing cores is isolated.

[0012] In a third aspect, an embodiment of the present invention provides a chip comprising the many-core communication architecture as described in the second aspect.

[0013] In a fourth aspect, an embodiment of the present invention provides a control method for a single-core processing core, which is applied to the single-core processing core as described in the first aspect, and the method includes:

[0014] When it is determined that the execution of the previous round of instruction group is completed, the control unit activates the NI receive instruction corresponding to the NI unit and the DMA read instruction corresponding to the DMA unit, and starts the data input transmission to the parameter RAM;

[0015] The control unit sends a control signal to the queue transmission unit, so that the queue transmission unit generates a queue transmission instruction according to the control signal, and starts the transmission of queue data to the parameter RAM input according to the queue transmission instruction;

[0016] When the checking of the input data is completed, the control unit activates the calculation engine to start the calculation;

[0017] When it is determined that the calculation engine has completed the calculation, the control unit activates the NI send instruction corresponding to the NI unit and the DMA write instruction corresponding to the DMA unit, wherein the NI send instruction includes queue data between the forward and backward directions;

[0018] When the output of the inspection data is completed, the control unit determines that the execution of the current round of instruction group is completed.

[0019] In some embodiments, the method further comprises:

[0020] The queue transmission unit receives the queue message from the NI unit;

[0021] Locating the hardware queue ID according to the data type code in the message header of the queue message;

[0022] Maintain the tail pointer and head pointer of the corresponding hardware queue ID;

[0023] When the input data associated with the computing instruction requires queue data, the control unit generates a queue_load instruction and distributes it to the queue transmission unit;

[0024] The queue transfer unit reads the data block according to the head pointer and transfers the data block to the parameter RAM.

[0025] In some embodiments, the method further comprises:

[0026] When the calculation engine completes the calculation of a single weight gradient element, it notifies the weight gradient transmission unit to start by setting the lock signal to 1;

[0027] After the weight gradient transmission unit completes the transmission of the current element store and the next weight gradient element load, the calculation engine is notified to start the calculation of the next weight gradient element by clearing the lock signal;

[0028] Provide the address of the weight gradient in DRAM;

[0029] Execute DMA write instructions to transfer weight gradient elements from SRAM to DRAM;

[0030] Execute a DMA read instruction to transfer the weight gradient elements from DRAM to SRAM.

[0031] According to the single-core processing core and its control method, many-core communication architecture, and chip provided by the embodiments of the present invention, the single-core processing core of one embodiment has an on-chip network interface and a memory access interface for external data transmission. The single-core processing core internally includes a computing engine, a control unit, a data exchange unit, a parameter RAM, an NI unit, a DMA unit, a queue transmission unit, a weight gradient transmission unit, a data exchange bus, and a control bus. The control unit is responsible for activating computing instructions, memory transmission instructions, and network transmission instructions. The computing instructions are pre-arranged and stored in the computing engine, the memory transmission instructions are pre-arranged and stored in the DMA unit, the memory transmission instructions include DMA write instructions and DMA read instructions, and the network transmission instructions are pre-arranged and stored in the NI Unit, network transmission instructions include NI send instructions and NI receive instructions. The control unit is electrically connected to the computing engine, NI unit, DMA unit, and queue transmission unit through a control bus. The data exchange unit is electrically connected to the parameter RAM, NI unit, DMA unit, queue transmission unit, and weight gradient transmission unit through a data exchange bus. The queue transmission unit is used to achieve hardware queue expansion and acceleration between forward and backward directions. The weight gradient transmission unit is used to provide transmission acceleration for the frequent reading and writing of DRAM memory by the weight gradient of the deep neural network. The weight gradient is stored in the DRAM memory. The control unit triggers the queue transmission unit to generate queue transmission instructions through a control signal, and the computing engine triggers the weight gradient transmission unit to generate weight gradient transmission instructions through a control signal. One embodiment of the many-core communication architecture has at least two of the above-mentioned single-core processing cores. The single-core processing core has an on-chip network interface and a memory access interface for external data transmission. The on-chip network interface is used for data transmission between any two single-core processing cores. The on-chip network interface is electrically connected to an on-chip router. The on-chip router is used for data forwarding between any two single-core processing cores. The single-core processing core exclusively occupies a dynamic random access memory DRAM space. The memory logic between any two single-core processing cores is isolated. A chip according to one embodiment includes the aforementioned many-core communication architecture.One embodiment of the control method of a single-core processing core is applied to the above-mentioned single-core processing core, and the method includes: when determining that the execution of the previous round of instruction group is completed, the control unit activates the NI receive instruction corresponding to the NI unit and the DMA read instruction corresponding to the DMA unit to start the data input transmission to the parameter RAM; the control unit sends a control signal to the queue transmission unit so that the queue transmission unit generates a queue transmission instruction according to the control signal, and starts the queue data input transmission to the parameter RAM according to the queue transmission instruction; when checking that the input data is completed, the control unit activates the calculation engine to start the calculation; when determining that the calculation engine calculation is completed, the control unit activates the NI send instruction corresponding to the NI unit and the DMA write instruction corresponding to the DMA unit, wherein the NI send instruction contains the queue data between the forward and backward directions; when checking that the data output is completed, the control unit determines that the execution of this round of instruction group is completed. Based on this, each single-core processing core has an on-chip network interface and a memory access interface for external data transmission, which supports two sets of data transmission modes, namely on-chip network access and memory access, and inter-core data transmission uses the on-chip network interface. The single-core processing core includes a compute engine, a control unit, a data exchange unit, parameter RAM, a NI unit, a DMA unit, a queue transfer unit, a weight gradient transfer unit, a data exchange bus, and a control bus. The single-core processing core provides programmable transfer instructions, including memory transfer instructions and network transfer instructions. Memory transfer instructions are pre-programmed and stored in the DMA unit, while network transfer instructions are pre-programmed and stored in the NI unit, adapting to a variety of models and deployment scenarios. Queue transfer instructions and weight gradient transfer instructions are generated in real time based on control signals from the control unit, eliminating the need for software pre-programming. The control unit activates the compute engine's computational instructions, which then completes the corresponding computation and notifies the control unit upon completion, accelerating the transition between transmission and computation and enabling rapid switching between processing transmission and computation. The queue transfer unit and weight gradient transfer unit within the single-core processing core are dedicated hardware transfer units for DNN training. The queue transfer unit is used to expand and accelerate hardware queues between forward and backward passes. The weight gradient transfer unit provides transfer acceleration for the frequent read and write operations of DRAM memory used by deep neural networks. Weight gradients are stored in DRAM memory to accelerate gradient read and write access for weight gradient calculations. A dedicated hardware transmission unit for DNN training can offload software control and management overhead, reducing DNN training data transmission latency. Therefore, embodiments of the present invention can improve the transmission performance and energy efficiency of inter-core communication and reduce DNN training data transmission latency.

[0032] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation to the technical solution of the present invention.

[0034] FIG1 is a schematic diagram of the structure of a many-core communication architecture provided by an embodiment of the present invention;

[0035] FIG2 is a schematic diagram of the internal structure of a single-core processing core provided by one embodiment of the present invention;

[0036] FIG3A is a flow chart of a control method for a single-core processing core according to an embodiment of the present invention;

[0037] FIG3B is a control flow chart of a control unit provided by one embodiment of the present invention;

[0038] FIG4A is a flow chart of a control method for a single-core processing core according to another embodiment of the present invention;

[0039] FIG4B is a control flow chart of a queue transmission unit according to an embodiment of the present invention;

[0040] FIG5A is a flow chart of a control method for a single-core processing core according to another embodiment of the present invention;

[0041] FIG5B is a control flow chart of a weight gradient transmission unit according to an embodiment of the present invention;

[0042] FIG6 is a schematic diagram of a calculation instruction and a transmission instruction provided by an embodiment of the present invention cooperating to implement state machines of different models. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] It should be understood that in the description of the embodiments of the present invention, "multiple" (or multiple) means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, and "above," "below," and "within" are understood to include the number itself. The terms "first," "second," and so on are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or implicitly indicating the number of the indicated technical features, or implicitly indicating the order of the indicated technical features.

[0045] Deep neural networks (DNNs) are multi-layered computational models that mimic the workings of the human nervous system and have been widely used in fields such as automatic language translation and image recognition. DNN training is typically accomplished through supervised learning, consisting of two main steps: forward propagation and backpropagation. Forward propagation transfers input data from the input layer to the output layer, while backpropagation propagates errors from the output layer back to the input layer. Based on these errors, the weights and offsets between neurons in each layer are adjusted so that the network's output accurately reflects the input data. Backpropagation can be broken down into two substeps: error propagation and weight gradients.

[0046] Research and development for DNN training acceleration primarily focuses on the three steps of FP, EP, and WG. While neural network models continue to evolve, and the connections between operators and data are constantly changing, the functions and inter-unit connections of FP, EP, and WG remain essentially unchanged in the training domain. Therefore, chip design for training acceleration can form a reusable and scalable architecture around these three units, with relatively high reuse of the transmission part.

[0047] Taking a convolutional neural network (CNN) as an example, the forward pass (FP) primarily computes the convolution between the activation function A and the weight W, proceeding layer by layer in the forward direction. The backward pass (EP) primarily computes the convolution between the error E and the weight W, proceeding layer by layer in the backward direction. The workpiece (WG) primarily handles the convolution between the activation function A (provided in the forward direction) and the error E, proceeding synchronously with the backward pass (EP). As the number of DNN layers increases, the neural network regularization method BN (based on batched samples) has been introduced. BN increases the data transfer from the forward forward pass (FP) to the backward pass (EP), including intermediate data such as the mean and variance. BN also adds learning and updating of the scaling parameters γ and β, which can be viewed as extensions of the weight W, requiring additional gradient calculations for γ and β in the workpiece (WG). The introduction of residuals changes the inter-layer connectivity, but the inter-layer correspondence and data connections between the forward and backward directions remain unchanged. Spiking neural networks (SNNs) primarily transform the activation function A into a pulse S form and introduce operators in the time dimension, increasing the model's orchestration dimension T.

[0048] There are two main approaches to improving model training and deployment performance: data parallelism and model parallelism. Data parallelism requires each node to contain all model parameters and weights. Different nodes run different sample data simultaneously, and finally, the cumulative results of each node's WG are batch-reduced to update the weights W. Data parallelism places high demands on the memory capacity of a single node and is therefore more suitable for cluster deployment solutions, where a single node corresponds to a system consisting of multiple interconnected chips. Model parallelism divides the model into multiple regions, with pipelined parallel execution between regions. A single region is suitable for chip-level or single-core deployment. Model parallelism requires sample data to flow continuously between cores. Since data flow does not require inter-core traversal, it is more suitable for chip-on-chip (NOC) and many-core architectures. GPGPUs can also achieve model parallelism. Based on a shared memory architecture, memory addressing is unified, making them suitable for address-contiguous memory access.

[0049] One challenge in chip design supporting model parallelism is the large amount of memory required for pipelining. Since neural network training requires data caching between forward and backward passes, and forward and backward passes must be performed layer by layer, latency increases linearly with the number of DNN layers, and the required cache capacity also scales linearly. While there are other ways to reduce cache requirements, such as reducing data precision, alternating forward and backward pipelining, and recalculating at breakpoints, chip design, even with a NoC architecture, still requires maintaining high-bandwidth channels and efficient transmission methods between the many cores and DRAM memory to reduce training time costs.

[0050] Among related technologies, the current mainstream DNN (Deep Neural Networks) training platform uses GPU (Graphics Processing Unit) chips. The advantages of GPU are mainly reflected in the acceleration and high parallelism of calculations, and it has strong versatility in operators and communication. However, the GPU is based on a shared memory transmission architecture, and uses multi-level cache and cache consistency protocols to achieve inter-core communication. It cannot adapt well to the characteristics of DNN training data flow and static model deployment. Its transmission latency and energy efficiency still have a lot of room for optimization.

[0051] In response to the above technical problems, embodiments of the present invention provide a single-core and its control method, a multi-core communication architecture, and a chip. In one embodiment, the single-core processing core has an on-chip network interface and a memory access interface for external data transmission. The single-core processing core internally includes a computing engine, a control unit, a data exchange unit, a parameter RAM, an NI unit, a DMA unit, a queue transmission unit, a weight gradient transmission unit, a data exchange bus, and a control bus. The control unit is responsible for activating computing instructions, memory transfer instructions, and network transmission instructions. The computing instructions are pre-arranged and stored in the computing engine, the memory transfer instructions are pre-arranged and stored in the DMA unit, the memory transfer instructions include DMA write instructions and DMA read instructions, and the network transmission instructions are pre-arranged and stored in NI unit, network transmission instructions include NI send instructions and NI receive instructions. The control unit is electrically connected to the computing engine, NI unit, DMA unit, and queue transmission unit via a control bus. The data exchange unit is electrically connected to the parameter RAM, NI unit, DMA unit, queue transmission unit, and weight gradient transmission unit via a data exchange bus. The queue transmission unit is used to achieve hardware queue expansion and acceleration between forward and backward directions. The weight gradient transmission unit is used to provide transmission acceleration for the frequent reading and writing of DRAM memory by the weight gradient of the deep neural network. The weight gradient is stored in the DRAM memory. The control unit triggers the queue transmission unit to generate queue transmission instructions via a control signal, and the computing engine triggers the weight gradient transmission unit to generate weight gradient transmission instructions via a control signal. One embodiment of the many-core communication architecture has at least two of the above-mentioned single-core processing cores. The single-core processing cores have an on-chip network interface and a memory access interface for external data transmission. The on-chip network interface is used for data transmission between any two single-core processing cores. The on-chip network interface is electrically connected to an on-chip router. The on-chip router is used for data forwarding between any two single-core processing cores. The single-core processing cores exclusively occupy a dynamic random access memory (DRAM) space, and the memory logic between any two single-core processing cores is isolated. A chip according to one embodiment includes the aforementioned many-core communication architecture.One embodiment of the control method of a single-core processing core is applied to the above-mentioned single-core processing core, and the method includes: when determining that the execution of the previous round of instruction group is completed, the control unit activates the NI receive instruction corresponding to the NI unit and the DMA read instruction corresponding to the DMA unit to start the data input transmission to the parameter RAM; the control unit sends a control signal to the queue transmission unit so that the queue transmission unit generates a queue transmission instruction according to the control signal, and starts the queue data input transmission to the parameter RAM according to the queue transmission instruction; when checking that the input data is completed, the control unit activates the calculation engine to start the calculation; when determining that the calculation engine calculation is completed, the control unit activates the NI send instruction corresponding to the NI unit and the DMA write instruction corresponding to the DMA unit, wherein the NI send instruction contains the queue data between the forward and backward directions; when checking that the data output is completed, the control unit determines that the execution of this round of instruction group is completed. Based on this, each single-core processing core has an on-chip network interface and a memory access interface for external data transmission, which supports two sets of data transmission modes, namely on-chip network access and memory access, and inter-core data transmission uses the on-chip network interface. The single-core processing core includes a compute engine, a control unit, a data exchange unit, parameter RAM, a NI unit, a DMA unit, a queue transfer unit, a weight gradient transfer unit, a data exchange bus, and a control bus. The single-core processing core provides programmable transfer instructions, including memory transfer instructions and network transfer instructions. Memory transfer instructions are pre-programmed and stored in the DMA unit, while network transfer instructions are pre-programmed and stored in the NI unit, adapting to a variety of models and deployment scenarios. Queue transfer instructions and weight gradient transfer instructions are generated in real time based on control signals from the control unit, eliminating the need for software pre-programming. The control unit activates the compute engine's computational instructions, which then completes the corresponding computation and notifies the control unit upon completion, accelerating the transition between transmission and computation and enabling rapid switching between processing transmission and computation. The queue transfer unit and weight gradient transfer unit within the single-core processing core are dedicated hardware transfer units for DNN training. The queue transfer unit is used to expand and accelerate hardware queues between forward and backward passes. The weight gradient transfer unit provides transfer acceleration for the frequent read and write operations of DRAM memory used by deep neural networks. Weight gradients are stored in DRAM memory to accelerate gradient read and write access for weight gradient calculations. A dedicated hardware transmission unit for DNN training can offload software control and management overhead, reducing DNN training data transmission latency. Therefore, embodiments of the present invention can improve the transmission performance and energy efficiency of inter-core communication and reduce DNN training data transmission latency.

[0052] As shown in Figures 1 and 2, Figure 1 is a schematic diagram of a multi-core communication architecture structure provided by an embodiment of the present invention, and Figure 2 is a schematic diagram of the internal structure of a single-core processing core provided by an embodiment of the present invention. Each single-core processing core has an on-chip network interface and a memory access interface for external data transmission, which supports two sets of data transmission modes, namely on-chip network access and memory access, and inter-core data transmission uses the on-chip network interface. Each single-core processing core is mainly composed of a computing engine 201, a control unit 202, a data exchange unit 203, a parameter RAM (Random Access Memory) 204, an NI unit 205, a DMA (Direct Memory Access) unit 206, a queue transmission unit 207, a WG (Weight Gradients) transmission unit 208, a data exchange bus 209 and a control bus 210. The control unit 202 is electrically connected to the computing engine 201, the NI unit 205, the DMA unit 206, and the queue transfer unit 207 via the control bus 210. The data exchange unit 203 is electrically connected to the parameter RAM 204, the NI unit 205, the DMA unit 206, the queue transfer unit 207, and the WG transfer unit 208 via the data exchange bus 209. In-core instructions are distributed across the various units. For example, computing instructions are pre-arranged and stored in the computing engine 201, memory transfer instructions are pre-arranged and stored in the DMA unit 206, and network transfer instructions are pre-arranged and stored in the NI unit 205, adapting to various models and deployment scenarios. Queue transfer instructions and weight gradient transfer instructions are generated in real time based on control signals from the control unit 202, without the need for pre-arrangement by software. It should be pointed out that the queue transmission unit and the weight gradient transmission unit are dedicated hardware transmission units for DNN training. The queue transmission unit is used to achieve hardware queue expansion and acceleration between the forward and backward directions. The weight gradient transmission unit is used to provide transmission acceleration for the frequent reading and writing of DRAM memory for the weight gradient of the deep neural network. The weight gradient is stored in the DRAM memory to accelerate the gradient read and write access of the weight gradient calculation. The dedicated hardware transmission unit for DNN training can offload the control and management overhead of the software and reduce the data transmission latency of DNN training. Therefore, the embodiments of the present invention can improve the transmission performance and energy efficiency of inter-core communication and reduce the data transmission latency of DNN training.

[0053] It is understood that the control unit 202 is responsible for activating computational instructions, memory transfer instructions, and network transfer instructions, controlling the overall execution order. Instructions are pre-organized and stored in various units within the core. For example, computational instructions are pre-organized and stored in the computation engine 201, memory transfer instructions are pre-organized and stored in the DMA unit 206, and network transfer instructions are pre-organized and stored in the NI unit 205. Based on the model segmentation and tasks deployed to a single core, the control unit 202 activates a set of related computational and transfer instructions in a round-robin fashion. Execution between groups is serial, similar to the effect of memory barriers in a processor. Control within a group follows the order of input data -> computation -> output data.

[0054] It is understood that upon receiving the calculation instruction activation information from the control unit 202, the calculation engine 201 initiates the corresponding calculation and notifies the control unit 202 upon completion, thereby accelerating the connection between transmission and calculation and enabling rapid switching between processing transmission and calculation. The calculation instruction includes information such as the operator type, the internal RAM offset of the input and output parameters, and the parameter size.

[0055] It is understood that the parameter RAM 204 provides a buffer for input data required for calculations and a buffer for result output.

[0056] It will be appreciated that the NI unit 205 has an on-chip network interface, and network transmission instructions include NI send instructions and NI receive instructions. The NI unit sends data from the parameter RAM of the source single-core processing core to the destination single-core processing core according to the NI send instruction, and writes data received from the destination single-core processing core to the parameter RAM of the source single-core processing core according to the NI receive instruction. Pairs of NI send instructions and NI receive instructions are associated by the same tag number.

[0057] As will be appreciated, the NI unit 205 provides interface processing for network transmission. NI Send transmits data from the parameter RAM to other cores, while NI Receive writes data received from other cores to the corresponding parameter RAM. NI instructions provide routing information, directing network traffic to the destination core. NI instructions also specify parameter type and size, indicating the connection between the NI and the parameter RAM. See Table 1 below for NI Send instructions, which support both unicast and multicast.

[0058] TABLE1

[0059] See Table 2 below for NI receive commands, which support 1-to-1 and many-to-1 aggregate communication models.

[0060] TABLE2

[0061] NI instructions are passive. After receiving a network message, the device must transmit data to the destination according to the NI instruction. NI sending and receiving are associated through the same tag number. The network message header carries the tag information.

[0062] It can be understood that the DMA unit 206 is used to realize data transmission between the parameter RAM and the DRAM memory. The memory transfer instructions include DMA write instructions and DMA read instructions. The DMA write instructions and DMA read instructions contain the data type code, source address information and target address information of the corresponding parameter RAM. The source address information and target address information are used to indicate the internal offset of the parameter RAM. The data exchange unit accesses the parameter RAM according to the data type code, source address information and target address information.

[0063] As can be understood, DMA unit 206 implements data transfer between general parameter RAM and DRAM memory. DMA writes transfer data from parameter RAM to DRAM. DMA reads transfer data from DRAM to the corresponding parameter RAM. To achieve integrated DMA and NI transfers within the core, DMA instructions are specially designed. See Table 3 below for details.

[0064] [Corrected 08.12.2023 according to Rule 91] TABLE 3

[0065] The DMA instruction provides a dcode data type encoding corresponding to a specific parameter RAM. DMA write src_addr and DMA read dst_addr indicate the internal offset of the parameter RAM. Therefore, the data exchange unit 203 can uniformly access the parameter RAM based on the data type dcode and the internal offset of the RAM.

[0066] It can be understood that the queue transmission unit 207 realizes the expansion and acceleration of the hardware queue between the forward and backward directions. The queue transmission unit 207 provides a hardware queue mechanism to accelerate data transmission between the forward and backward directions of training, and provides a configurable queue depth in DRAM. The data is managed as a whole according to the queue FIFO (First Input First Output), and the size of the processed data block and the internal data format are pre-set. When the backward core receives the data encoding of the queue type from the network interface NI, the queue transmission unit 207 automatically generates a queue_store instruction and stores it in the queue cache according to the queue tail pointer. When the input data associated with the calculation instruction needs to use the queue data, the control unit 202 generates a queue_load instruction and distributes it to the queue transmission unit 207, reads the data block according to the queue head pointer and transfers it to the parameter RAM.

[0067] It's understood that the WG transfer unit 208 is a transmission acceleration unit for the frequent WG read and write operations into DRAM memory. In a neural network model (DNN), weights W account for the largest portion of memory storage. The gradients involved in the WG require batch reduction, occupying the same cache capacity as W and requiring frequent memory read and write access. The WG transfer unit 208 supports storing gradients in DRAM and accelerates the gradient read and write accesses for WG calculations.

[0068] As can be understood, the calculation engine 201 and the WG transmission unit 208 form an interlocked collaboration. The current WG calculation result is written back to the memory, and the next round of WG data is transferred from the memory to the parameter RAM. Once both are completed, the WG calculation starts. The WG transmission unit is automatically triggered by the interlock control signal without the need for dynamic software intervention, achieving low-latency WG transmission.

[0069] It is understood that the data exchange unit 203 provides data exchange between the parameter RAM and external cores. A single core presents a continuous address space to the outside world, while internal parameter RAM addressing is accomplished using the data type DCODE. Specifically, the data exchange unit 203 uses ID addressing within the core, unifying data scheduling for network and memory accesses. Addressing is unified within the core using the ID + RAM internal offset address method.

[0070] It is understandable that the data exchange bus 209 carries a unified message transmission format within the core, based on ID addressing.

[0071] It can be understood that the control bus 210 carries trigger signals and instruction activation information between the control unit 202 and the computing engine 201, the NI unit 205, the DMA unit 206, and the queue transmission unit 207.

[0072] Based on this, the embodiments of the present invention can improve the transmission performance and energy efficiency of inter-core communication and reduce the data transmission delay of DNN training.

[0073] An embodiment of the present invention also provides a many-core communication architecture, as shown in FIG1 . The many-core communication architecture has at least two of the aforementioned single-core processing cores. Each single-core processing core has an on-chip network interface and a memory access interface for external data transmission. Inter-core data transmission uses the on-chip network interface, which is used for data transmission between any two single-core processing cores. The on-chip network interface is electrically connected to an on-chip router, which is used for data forwarding between any two single-core processing cores. The single-core processing cores exclusively occupy a block of dynamic random access memory (DRAM) space. The memory between any two single-core processing cores is logically isolated, making system deployment and expansion easier. The many-core communication architecture is expanded through the on-chip network, and the DRAM memory is primarily used to expand the core's storage capacity.

[0074] In one embodiment, because the many-core communication architecture utilizes at least two of the aforementioned single-core processing cores, the many-core communication architecture can achieve the same technical effects as the aforementioned single-core processing cores. Each single-core processing core in the many-core communication architecture has an on-chip network interface and a memory access interface for external data transmission. It supports two data transmission modes: on-chip network access and memory access. Inter-core data transmission uses the on-chip network interface. The single-core processing core internally comprises a computing engine, a control unit, a data exchange unit, a parameter RAM, an NI unit, a DMA unit, a queue transfer unit, a weight gradient transfer unit, a data exchange bus, and a control bus. The single-core processing core provides programmable transfer instructions, including memory transfer instructions and network transfer instructions. Memory transfer instructions are pre-programmed and stored in the DMA unit, and network transfer instructions are pre-programmed and stored in the NI unit, adapting to various models and deployment scenarios. The queue transfer instructions and weight gradient transfer instructions are generated in real time based on control signals from the control unit, without the need for pre-programming by software. The control unit activates the computing engine's computational instructions, and the computing engine notifies the control unit upon completion of the corresponding computation, accelerating the connection between transmission and computation, enabling rapid switching between processing transmission and computation. The queue transmission unit and weight gradient transmission unit inside the single-core processing core are dedicated hardware transmission units for DNN training. The queue transmission unit is used to realize the expansion and acceleration of the hardware queue between the forward and backward directions. The weight gradient transmission unit is used to provide transmission acceleration for the frequent reading and writing of the DRAM memory for the weight gradient of the deep neural network. The weight gradient is stored in the DRAM memory to accelerate the gradient read and write access of the weight gradient calculation. The dedicated hardware transmission unit for DNN training can offload the control and management overhead of the software and reduce the data transmission delay of DNN training. It should be pointed out that the present invention is oriented towards the characteristics of DNN training acceleration, integration of NOC and shared memory. The multi-core communication architecture of the present invention has higher energy efficiency, the advantages of reusability and inheritance, and is more suitable for supporting coarse-grained computing instructions. The coarse-grained computing instructions have many types of parameters and a large amount of data, which can give play to the parallel and continuous high-efficiency characteristics of transmission. Therefore, the multi-core communication architecture of the present invention can improve the transmission performance and energy efficiency of inter-core communication and reduce the data transmission delay of DNN training.

[0075] An embodiment of the present invention also provides a chip that includes the aforementioned many-core communication architecture. Thus, the chip can achieve the same technical effects as the aforementioned many-core communication architecture. The chip includes the aforementioned many-core communication architecture, wherein each single-core processing core in the many-core communication architecture has an on-chip network interface and a memory access interface for external data transmission. The chip supports two data transmission modes: on-chip network access and memory access. Inter-core data transmission uses the on-chip network interface. The single-core processing core internally includes a computing engine, a control unit, a data exchange unit, a parameter RAM, an NI unit, a DMA unit, a queue transfer unit, a weight gradient transfer unit, a data exchange bus, and a control bus. The single-core processing core provides programmable transfer instructions, including memory transfer instructions and network transfer instructions. Memory transfer instructions are pre-programmed and stored in the DMA unit, while network transfer instructions are pre-programmed and stored in the NI unit, adapting to various models and deployment scenarios. The queue transfer instructions and weight gradient transfer instructions are generated in real time based on control signals from the control unit, eliminating the need for pre-programming by software. The control unit activates the computing engine's computational instructions, and upon completion, the computing engine notifies the control unit to accelerate the connection between transmission and computation, enabling rapid switching between processing, transmission, and computation. The queue transmission unit and weight gradient transmission unit inside the single-core processing core are dedicated hardware transmission units for DNN training. The queue transmission unit is used to realize hardware queue expansion and acceleration between forward and backward directions. The weight gradient transmission unit is used to provide transmission acceleration for the frequent reading and writing of DRAM memory for the weight gradient of the deep neural network. The weight gradient is stored in the DRAM memory to accelerate the gradient read and write access of the weight gradient calculation. The dedicated hardware transmission unit for DNN training can offload the control and management overhead of the software and reduce the data transmission delay of DNN training. It should be pointed out that the present invention is oriented towards DNN training acceleration, integration of NOC and shared memory. The multi-core communication architecture of the present invention has higher energy efficiency, reusability and inheritance advantages, and is more suitable for supporting coarse-grained computing instructions. The coarse-grained computing instructions have many types of parameters and large amounts of data, which can give play to the parallel and continuous high-efficiency characteristics of transmission. Therefore, the chip of the present invention can improve the transmission performance and energy efficiency of inter-core communication and reduce the data transmission delay of DNN training.

[0076] As shown in FIG3A , an embodiment of the present invention further provides a control method for a single-core processing core, and the control method is applied to the above-mentioned single-core processing core.

[0077] The control method includes but is not limited to the following steps:

[0078] Step S11, when it is determined that the execution of the previous round of instruction group is completed, the control unit activates the NI receive instruction corresponding to the NI unit and the DMA read instruction corresponding to the DMA unit, and starts the data input transmission to the parameter RAM;

[0079] Step S12: the control unit sends a control signal to the queue transmission unit, so that the queue transmission unit generates a queue transmission instruction according to the control signal, and starts the transmission of the queue data to the parameter RAM input according to the queue transmission instruction;

[0080] Step S13: When the input data is checked, the control unit activates the calculation engine to start calculation;

[0081] Step S14: When it is determined that the calculation engine has completed the calculation, the control unit activates the NI send instruction corresponding to the NI unit and the DMA write instruction corresponding to the DMA unit, wherein the NI send instruction contains the queue data between the forward and backward directions;

[0082] Step S15: When the output of the inspection data is completed, the control unit determines that the execution of this round of instruction group is completed.

[0083] It can be understood that Figure 3B is a control flow of the control unit, which is based on a continuous loop of instruction groups. The execution order within the instruction group ensures data input transmission -> calculation -> data output transmission. Multiple transmission instructions are input concurrently, and multiple transmission instructions are output concurrently. The present invention completes the check of transmission instructions, processes the rapid switching of transmission and calculation, and realizes accelerated transmission and calculation connection to improve transmission performance and energy efficiency. It should be noted that the instructions self-generated by the hardware are not in the process of the control unit, such as the operation of WG. The specific control flow of the control unit is as follows:

[0084] S301: Check the start conditions of the current round of instruction group and require the execution of the previous round of instruction group to be completed.

[0085] S302: Identify calculation-related NI receive and DMA read instructions, generate activation information to the NI unit and DMA unit, and each transmission unit executes the instructions of the activation area to start data input transmission to the parameter RAM.

[0086] S303: Identify the calculation-related queue transfer instructions, trigger the queue_load instruction to the queue transfer unit, and start the queue data input transfer to the parameter RAM.

[0087] S304: Check whether the activated NI receive, DMA read, and queue_load instructions are executed, and wait in a loop.

[0088] S305: Activate calculation-related instructions, and the calculation engine starts calculation.

[0089] S306: Wait for the calculation completion indication.

[0090] S307: Identify computation-related NI send and DMA write instructions, generate activation information for the NI unit and DMA unit, and each transfer unit executes the instructions in the activation area, starting data transfer from the parameter RAM. NI send instructions include forward and backward queue data transmission and are automatically converted into queue_store instructions after being received by the destination core NI.

[0091] S308: Check whether the activated NI send and DMA write instructions are completed, and wait in a loop. If completed, set the execution end flag of this round.

[0092] In one embodiment, as shown in FIG4A , the control method further includes but is not limited to the following steps:

[0093] Step S21, the queue transmission unit receives the queue message from the NI unit;

[0094] Step S22, locating the hardware queue ID according to the data type code in the message header of the queue message;

[0095] Step S23, maintaining the tail pointer and head pointer of the corresponding hardware queue ID;

[0096] Step S24: When the input data associated with the computing instruction requires queue data, the control unit generates a queue_load instruction and distributes it to the queue transmission unit;

[0097] Step S25 : The queue transmission unit reads the data block according to the head pointer and transmits it to the parameter RAM.

[0098] It is understood that as shown in Figure 4B, Figure 4B shows the control flow of the queue transmission unit, which is mainly implemented by hardware. The hardware maintains the queue FIFO (first in first out) between the forward core and the backward core, storing or reading one queue element at a time. The specific control flow of the queue transmission unit is as follows:

[0099] S401: Receive a message from the NI unit, originating from another forward core. The message header specifies the transfer type as forward / forward queue. The hardware queue ID is located based on the data type in the message header. The hardware generates a queue_store instruction. The DMA address of the instruction is calculated by the hardware based on the queue ID, the tail pointer (wptr), and the element size. The data order between forward and backward DNN training is generally first-in, first-out. If there is a need to adjust the order between elements, the hardware supports regular mapping of element DMA addresses in queue_store.

[0100] S402: Maintain the tail pointer wptr of the corresponding queue ID. After the element is stored in the queue buffer, wptr = wptr + 1. The queue is judged to be full according to the queue water level = wptr - rptr. When it is close to full, back pressure is triggered to the NI.

[0101] S403: Implement DMA writing to the queue buffer according to the queue_store instruction.

[0102] S404: The queue can be mapped to SRAM or DRAM, depending on the cache size requirement.

[0103] S405: When the input data associated with the calculation instruction requires queue data, the control unit generates a queue_load instruction and distributes it to S405, and then reads the data block according to the queue head pointer and transfers it to the parameter RAM.

[0104] S406: Maintain the head pointer rptr of the corresponding queue ID. If wptr is not equal to rptr, it means that there is a valid element in the queue and queue_load can be executed. After the element is read, rptr = rptr + 1.

[0105] In one embodiment, as shown in FIG5A , the control method further includes but is not limited to the following steps:

[0106] Step S31, when the calculation engine completes the calculation of a single weight gradient element, it notifies the weight gradient transmission unit to start by setting the lock signal to 1;

[0107] Step S32: After the weight gradient transmission unit completes the transmission of the current element store and the next weight gradient element load, it notifies the calculation engine to start the calculation of the next weight gradient element by clearing the lock signal;

[0108] Step S33, providing the address of the weight gradient in DRAM;

[0109] Step S34, executing a DMA write instruction to transfer the weight gradient element from the SRAM to the DRAM;

[0110] Step S35: execute a DMA read instruction to transfer the weight gradient elements from the DRAM to the SRAM.

[0111] As can be understood, as shown in Figure 5B, which illustrates the control flow of the WG transmission unit, this flow is primarily implemented by hardware. The WG is stored in DRAM, and the WG calculation unit loads one element at a time, writing back the element's DRAM location after the calculation is complete. Because the WG elements are arranged regularly and continuously in DRAM, WG calculation and WG transmission can be combined for multiple iterations, enabling seamless hardware calculation and transmission processing without the need for software involvement. The specific control flow of the WG transmission unit 208 is as follows:

[0112] S501: The calculation engine completes the calculation of a single WG element and notifies the WG transmission 208 to start by setting the lock signal to 1; after the WG transmission completes the store of the current element and the load of the next element, it notifies the calculation engine to start the calculation of the next element by clearing the lock signal.

[0113] S502: Provide the base address of WG in DRAM.

[0114] S503: Execute the DMA write instruction generated in S501, and transfer the element from SRAM to DRAM.

[0115] S504: Execute the DMA read instruction generated in S501, and transfer the element from DRAM to SRAM.

[0116] Figure 6 shows an example of how compute and transfer instructions can be combined to implement different model state machines. Each single processing core can pre-program a task state machine, encompassing both compute and transfer actions. Because the architecture supports a combination of coarse-grained compute and transfer instructions, pre-programming these instructions enables flexible DNN training models.

[0117] For example, the state of core 2 in Figure 6 is decomposed into:

[0118] 1) Receive s0 from core 1 from NI;

[0119] 2) Calculate convolution / matrix multiplication to generate the intermediate result sum2;

[0120] 3) Receive sum3 from core 3 from NI and perform vector calculation sum2^=sum3+sum2;

[0121] 4) Vector calculation generates s2;

[0122] 5)DMA output s2;

[0123] 6) Queue transmission between front and back.

[0124] Based on this, the embodiments of the present invention have at least the following beneficial effects:

[0125] 1) Adapts to various models and deployment scenarios. Each core supports two data transfer methods: memory access and NoC access. Transfer instructions are programmable, and both methods are uniformly addressed within the core using an ID + RAM internal offset address.

[0126] 2) Accelerate the connection between transmission and computing. Perform transmission instruction completion checks and handle rapid switching between transmission and computing.

[0127] 3) Forward and backward transmission capacity expansion and acceleration: A hardware queue mechanism is provided between the forward and backward training directions to accelerate data transmission, and a configurable queue depth is provided in DRAM.

[0128] 4) Accelerated data transfer for WG calculations. Weights W account for the largest portion of storage in neural network models. The gradients involved in WG require batch reduction, occupying the same cache capacity as W and requiring frequent memory read and write accesses. Support for storing gradients in DRAM accelerates gradient read and write accesses for WG calculations.

[0129] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A single-core processing core, characterized in that: The single-core processing core has an on-chip network interface and a memory access interface for external data transmission. The single-core processing core includes a computing engine, a control unit, a data exchange unit, a parameter RAM, an NI unit, a DMA unit, a queue transmission unit, a weight gradient transmission unit, a data exchange bus and a control bus. The control unit is responsible for the activation of computing instructions, memory transmission instructions and network transmission instructions. The computing instructions are pre-arranged and stored in the computing engine, the memory transmission instructions are pre-arranged and stored in the DMA unit, the memory transmission instructions include DMA write instructions and DMA read instructions, the network transmission instructions are pre-arranged and stored in the NI unit, and the network transmission instructions include NI send instructions and NI receive instructions. The control unit is configured to activate the computing instructions, memory transmission instructions and network transmission instructions. The computing instructions are pre-arranged and stored in the computing engine, the memory transmission instructions are pre-arranged and stored in the DMA unit, the memory transmission instructions include DMA write instructions and DMA read instructions, the network transmission instructions are pre-arranged and stored in the NI unit, and the network transmission instructions include NI send instructions and NI receive instructions. The control bus is electrically connected to the computing engine, the NI unit, the DMA unit, and the queue transmission unit, respectively. The data exchange unit is electrically connected to the parameter RAM, the NI unit, the DMA unit, the queue transmission unit, and the weight gradient transmission unit, respectively, through the data exchange bus. The queue transmission unit is used to achieve hardware queue expansion and acceleration between forward and backward directions. The weight gradient transmission unit is used to provide transmission acceleration for frequent reading and writing of DRAM memory for weight gradients of deep neural networks. The weight gradient is stored in the DRAM memory. The control unit triggers the queue transmission unit to generate a queue transmission instruction through a control signal. The computing engine triggers the weight gradient transmission unit to generate a weight gradient transmission instruction through a control signal.

2. The single-core processing core according to claim 1, characterized in that: The NI unit has the on-chip network interface. The NI unit sends the data of the parameter RAM in the source single-core processing core to the destination single-core processing core according to the NI send instruction, and writes the data received from the destination single-core processing core to the parameter RAM in the source single-core processing core according to the NI receive instruction. The paired NI send instructions and the NI receive instructions are associated by the same label number.

3. The single-core processing core according to claim 1, characterized in that: The DMA unit is used to realize data transmission between the parameter RAM and the DRAM memory. The DMA write instruction and the DMA read instruction contain data type coding, source address information and target address information corresponding to the parameter RAM. The source address information of the DMA write instruction and the target address information of the DMA read instruction are used to indicate the internal offset of the parameter RAM. The data exchange unit accesses the parameter RAM according to the data type coding, the source address information and the target address information.

4. The single-core processing core according to claim 1, characterized in that: The queue transmission unit provides a first-in-first-out hardware queue mechanism between the forward and backward directions of training to accelerate data transmission, and provides a configurable queue depth in the DRAM memory.

5. The single-core processing core according to claim 1, characterized in that: The computing engine and the weight gradient transmission unit form an interlocking cooperation.

6. The single-core processing core according to claim 1, characterized in that: The data exchange unit is used to provide data exchange between the parameter RAM and outside the core. The data exchange unit adopts ID addressing in the single-core processing core to unify data scheduling of network and memory access.

7. A many-core communication architecture, characterized in that: It has at least two single-core processing cores as described in any one of claims 1 to 6, the on-chip network interface is used for data transmission between any two of the single-core processing cores, the on-chip network interface is electrically connected to an on-chip router, the on-chip router is used for data forwarding between any two of the single-core processing cores, the single-core processing cores exclusively occupy a dynamic random access memory DRAM space, and the memory logic between any two of the single-core processing cores is isolated.

8. A chip, characterized in that: It includes the many-core communication architecture as claimed in claim 7.

9. A control method for a single-core processing core, characterized in that: Applied to the single-core processing core according to any one of claims 1 to 6, the method comprises: When it is determined that the execution of the previous round of instruction group is completed, the control unit activates the NI receiving instruction corresponding to the NI unit and the DMA reading instruction corresponding to the DMA unit, and starts the data to parameter RAM input transmission; The control unit sends a control signal to the queue transmission unit, so that the queue transmission unit generates a queue transmission instruction according to the control signal, and starts the transmission of queue data to the parameter RAM input according to the queue transmission instruction; When checking the input data is completed, the control unit activates the calculation engine to start calculation; When it is determined that the calculation engine has completed the calculation, the control unit activates the NI send instruction corresponding to the NI unit and the DMA write instruction corresponding to the DMA unit, wherein the NI send instruction contains queue data between the forward and backward directions; When the output of the inspection data is completed, the control unit determines that the execution of this round of instruction group is completed.

10. The control method of a single-core processing core according to claim 9, characterized in that: The method further comprises: The queue transmission unit receives the queue message from the NI unit; Locating the hardware queue ID according to the data type code in the message header of the queue message; Maintain the tail pointer and head pointer of the corresponding hardware queue ID; When the input data associated with the computing instruction needs to use queue data, the control unit generates a queue_load instruction and distributes it to the queue transmission unit; The queue transmission unit reads the data block according to the head pointer and transmits it to the parameter RAM.

11. The control method of a single-core processing core according to claim 9, characterized in that: The method further comprises: When the calculation engine completes the calculation of a single weight gradient element, it notifies the weight gradient transmission unit to start by setting the lock signal to 1; After the weight gradient transmission unit completes the transmission of the current element store and the next weight gradient element load, the calculation engine is notified to start the next weight gradient element calculation by clearing the lock signal; Provide the address of the weight gradient in DRAM; Execute DMA write instructions to transfer weight gradient elements from SRAM to DRAM; Execute a DMA read instruction to transfer the weight gradient elements from DRAM to SRAM.

Images