Data transmission device, processor, chip, board card and data transmission method

Abstract

The invention discloses a data transmission device, a processor, a chip, a board card and a data transmission method. The data transmission device may be included in a computing device, which may be included in a combined processing device, which may also include an interface device and other processing devices. The computing device interacts with other processing devices to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the computing device and the other processing devices and used for storing data of the computing device and the other processing devices. According to the scheme disclosed by the invention, an optimization scheme of convolution operation is provided, the requirement on the pendulum number of software can be saved, and the high-speed computing power of a processor is fully utilized.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computers. More specifically, the present disclosure relates to a data transmission device, a processor, a chip, a board, and a data transmission method. Background Art

[0002] Neural network is one of the most critical technologies in artificial intelligence and deep learning, and convolutional neural network (CNN) is the most important type of network. The most critical calculation in convolutional neural network is the convolution operation of the convolution layer (Conv layer). The function of the convolution layer is to extract features from the input data. Through multiple layers of convolution, complex features can be extracted to ensure that the network has sufficient expression and generalization capabilities. The neural network model contains a large number of various types of convolution operations, and the computational performance of the convolution operation greatly affects the computational performance of the entire neural network model.

[0003] When neural network models are applied to different fields, such as speech recognition, machine translation, image processing, etc., the corresponding input feature maps and weights may have different dimensions. In order to fully utilize the hardware advantages of deep learning processors, it is necessary to optimize convolution operations of different scales and types to improve the computing performance of executing neural network models.

[0004] It is known that there are processing solutions that optimize convolution operations of different scales and types, such as Chinese invention patent application No. 2021111313885. In the above application, it is necessary to arrange the data to be operated into a specified format through software instructions. This kind of software instruction processing takes a lot of time, seriously affects the overall network operation efficiency and greatly increases the programming complexity of programmers.

[0005] In view of this, there is an urgent need to provide an optimization solution for the above data format placement to fully utilize the computing power advantages of deep learning processors. Summary of the invention

[0006] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a data transmission solution and related products in multiple aspects.

[0007] In a first aspect, the present disclosure provides a data transmission device, comprising: a storage circuit for storing data, wherein the data is stored in order according to a first dimension; a plurality of read ports for accessing a plurality of discrete addresses from the storage circuit to read the data; a plurality of data channels for distributing and transmitting the data read via the read ports to a plurality of processing circuit groups; a cache located within the processing circuit group, for receiving and caching the data transmitted via the data channels and to be processed by the processing circuit group; and a block circuit for splitting the data from the cache into data blocks of a fixed size, aligning and adjusting the order, and then transmitting the data to the operation circuit within the processing circuit group, wherein the data blocks are arranged in order according to the second dimension, and the data blocks are arranged in order according to the first dimension.

[0008] In a second aspect, the present disclosure provides a processor, comprising the data transmission device of the first aspect.

[0009] In a third aspect, the present disclosure provides a chip comprising the processor of the aforementioned second aspect.

[0010] In a fourth aspect, the present disclosure provides a board comprising the chip of the third aspect.

[0011] In a fifth aspect, the present disclosure provides a data transmission method, comprising: accessing multiple discrete addresses from a storage circuit via several read ports to read data, wherein the data is stored in the storage circuit in a first dimension order; allocating and transmitting the data read via the read port to multiple processing circuit groups through multiple data channels; receiving and caching the data transmitted via the data channels and to be processed by the processing circuit group at a cache within the processing circuit group; and splitting the data from the cache into data blocks of a fixed size, aligning and adjusting the order, and then transmitting them to the operation circuit within the processing circuit group, wherein the data blocks are arranged in a second dimension order, and the data blocks are arranged in the first dimension order.

[0012] Through the data transmission device, processor, chip, board and data transmission method provided above, the disclosed embodiment provides a solution for adjusting the order of data placement on the data transmission path, which simplifies the requirements for software and facilitates unified management by software programmers without the need for software to perform block storage operations on data. This hardware block operation can be hidden in the operation pipeline, significantly improving processing efficiency and shortening processing time. This solution is particularly suitable for small convolution operation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0014] Figure 1 A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0015] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0016] Figure 3 A schematic diagram showing the internal structure of a processing core when the computing device is a single-core or multi-core device;

[0017] Figure 4a-4b An example of an exemplary convolution operation principle to which the disclosed embodiments can be applied is shown;

[0018] Figure 5 A schematic diagram showing the arrangement format of neuron data according to an embodiment of the present disclosure;

[0019] Figure 6a A schematic structural block diagram of a computing device according to an embodiment of the present disclosure is shown;

[0020] Figure 6b Shown based on Figure 6a Schematic diagram of computing task allocation when a computing device implements a convolution operation in Forward1 mode;

[0021] Figure 7 An exemplary structural block diagram of a data transmission device according to some embodiments of the present disclosure is shown;

[0022] Figure 8a-8c The input neuron data distribution, the first read port allocation and the weight data distribution required for a single round of Forward1 operation according to an embodiment of the present disclosure are respectively shown;

[0023] Figure 9a-9c The input neuron data distribution, the first read port allocation and the weight data distribution required for a single round of Forward1 operation according to another embodiment of the present disclosure are respectively shown;

[0024] Figure 10a-Figure 10c The input neuron data distribution, the first read port allocation and the weight data distribution required for a single round of Forward1 operation according to another embodiment of the present disclosure are respectively shown;

[0025] Fig.11 A schematic structural diagram of a storage circuit implemented by using SRAM according to an embodiment of the present disclosure is shown;

[0026] Fig.12 A schematic structural diagram of a storage circuit implemented by using SRAM according to an embodiment of the present disclosure is shown from another angle;

[0027] Fig.13 A schematic diagram of a hardware block scheme for processing neuron data within a circuit group is shown;

[0028] Fig.14 An exemplary flow chart of a data transmission method of a data transmission device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0030] It should be understood that the terms "first", "second", "third", and "fourth" etc. that may appear in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0031] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0032] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0033] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings. Exemplary Hardware Environment

[0034] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1 As shown, the board 10 includes a chip 101, which is a system-level chip, or a system on chip (SoC), integrated with one or more combined processing devices, which are an artificial intelligence computing unit to support various deep learning and machine learning algorithms, and meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant feature of cloud intelligence applications is the large amount of input data, which has high requirements for the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud intelligence applications, and has huge off-chip storage, on-chip storage and powerful computing power.

[0035] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe (Peripheral Component Interconnect express) interface.

[0036] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a single chip microcomputer, also known as a micro controller unit (MCU).

[0037] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0038] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0039] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into a storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into a control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data from the storage device on the computing device 201 chip and transmit it to the processing device 203.

[0040] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0041] The storage device 204 is used to store data to be processed, which may be DRAM (Dynamic Random Access Memory), which is DDR (Double Data Rate) memory, and is usually 16G or larger in size, and is used to store data of the computing device 201 and / or the processing device 203 .

[0042] Figure 3 Shows Figure 2 The internal structure diagram of the processing core when the computing device 201 in FIG. 2 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three modules: a control module 31 (also called a controller), an operation module 32 (also called an operator), and a storage module 33 (also called a memory).

[0043] The control module 31 is used to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the deep learning task, and includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 is used to obtain instructions from the processing device 203, and the instruction decode unit 312 decodes the obtained instructions and sends the decoding results to the operation module 32 and the storage module 33 as control information.

[0044] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 322 is responsible for the core calculation of the deep learning algorithm, namely matrix multiplication and convolution.

[0045] The storage module 33 is used to store or transfer relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access module (DMA) 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after calculation; WRAM 332 is used to store the convolution kernel of the deep learning network, that is, the weight; DMA 333 is connected to DRAM 204 via bus 34, and is responsible for data transfer between the computing device 301 and DRAM 204. It should be noted that the NRAM and WRAM here can be two storage areas formed by dividing the same memory in the logical storage space, or they can be two independent memories, which are not specifically limited here. Example convolution operation principle

[0046] The convolution layer in the neural network model can perform convolution operations, which extract features by applying convolution kernels (also called filters, weights, etc.) to the input feature map (also called input data, neurons, or input neurons). The convolution layer can contain multiple convolution kernels, and each element of the convolution kernel corresponds to a weight coefficient and a bias.

[0047] The neural network model may include various convolution operation layers, such as a convolution layer that performs forward, conventional 3D convolution operations, and a deconvolution layer that performs depth-wise convolution operations. In reverse training, it may be necessary to perform reverse depth-wise convolution operations or cross-multiplication convolution operations. The disclosed embodiments can be applied to optimization schemes for these different types of convolution operations.

[0048] In conventional 3D convolution operations, assuming that the input feature map tensor shape in the convolution layer is represented by X[N Hi Wi Ci], the tensor shape of the convolution kernel is represented by K[Co Kh Kw Ci], and the output result is Y[NHoWo Co], then the simplified mathematical calculation formula of the convolution operation can be expressed as follows: Y in,jc,jh,jw =∑ 0≤ic≤ci,0≤ih≤kh,0≤iw≤kw X in,ic,jh×sh+ih,jw×sw+iw ×K jc,ic,ih,iw (1)

[0049] In the above formula, X is the input data, Y is the output data, K is the convolution kernel, Kh and Kw are the length and width of K, sh and sw are the strides in the length and width directions, and the formula ignores the bias, padding and dilation, and assumes that the input data X has been padded and the convolution kernel has been dilated. The formula ignores the N dimension and the C dimension. The forward calculation of the neural network model in the N dimension is independent and fully connected in the C dimension. When the convolution kernel is working, it will scan the input features at a certain step length, perform matrix element multiplication on the input features in the convolution window, and superimpose the bias.

[0050] Figure 4a An exemplary conventional 3D convolution operation principle example to which the disclosed embodiments can be applied is shown.

[0051] The figure shows an example of four-dimensional input data X of size [N Hi Wi Ci], which can be represented as N three-dimensional rectangles 410a of size Hi×Wi×Ci. The figure also shows an example of a four-dimensional convolution kernel K of size [Co Kh Kw Ci], which can be represented as Co three-dimensional convolution kernels 420a of size Kh×Kw×Ci. The convolution result of the input data X and the convolution kernel K obtains the output data Y, which is four-dimensional data of size [N Ho Wo Co], which can be represented as N three-dimensional rectangles 430a of size Ho×Wo×Co.

[0052] The figure also specifically shows an example of a convolution operation, in which the input data is an input feature map 440a of size 6×6×3, omitting the N dimension; the convolution kernel is a stereo convolution kernel 450a of size 3×3×3, targeting a single Co; and the output data is an output feature map 460a of size 4×4. The specific operation process is as follows:

[0053] The convolution kernel 450 scans the input feature map 440a at a certain step length, performs matrix element multiplication and summation on the input features in the convolution window 470a, and superimposes the bias. That is, the value at each position in the output feature map 460a is obtained by adding up the corresponding block of each input feature map and the corresponding convolution kernel after performing a two-dimensional convolution operation. For example, the figure shows that the value of the (0,0) position on the output feature map 460a (that is, the convolution output point) is obtained by performing a two-dimensional convolution operation on the convolution window 470a framed by the black cube in the input feature map and the three-dimensional convolution kernel 450a to obtain three values, and then adding them up to obtain the final value.

[0054] In order to obtain outputs at other positions, the position of the convolution kernel 450a can be moved on the input feature map 440a, that is, the convolution window of the convolution output point can be moved. In the example in the figure, the convolution step (Sx, Sy) is (1,1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the value of the position (0,1) or (1,0) on the output feature map 460a can be obtained respectively.

[0055] From the above description, we can see that in a convolutional layer of a neural network, there are N groups of input feature maps, each group contains Hi×Wi×Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci×Co convolution kernels of size Kh×Kw, where Ci is the number of input channels, Co is the number of output feature maps (or the number of output channels), and Kh and Kw are the height and width of the convolution kernel, respectively. The output feature map contains Ho×Wo×Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. In addition, in the convolution operation, the convolution step size (Sx, Sy) is also involved, and the size of the convolution step size will affect the size of the output feature map.

[0056] The difference between depth-wise convolution and conventional 3D convolution is that there is no accumulation in the depth direction, where the depth direction refers to the input channel Ci. In conventional 3D convolution, each convolution kernel needs to be calculated and accumulated with all layers (input channels) of the input feature map, so the number of input channels of each convolution kernel is equal to the number of input channels of the input feature map. In depth convolution, each convolution kernel is single-channel, one convolution kernel is responsible for one channel, and one channel is convolved by only one convolution kernel. Therefore, depth convolution is sometimes also called 2D convolution, which means that it only slides and accumulates in the H and W dimensions. Since the input channels are not accumulated in depth convolution, when it comes to depth convolution, the dimensions of the input feature map, convolution kernel, and output feature map can be simplified to three dimensions: C (channel), H (height), and W (width).

[0057] In the back propagation of neural network model training, the calculation of neuron gradient and weight gradient is involved, as shown below:

[0058] Among them, top_diff and bottom_diff are neuron gradients, W is the weight of this iteration, and ΔW is the weight gradient calculated in this iteration. It is a calculation in back propagation, similar to convolution operation. With respect to the back propagation direction, the bottom_diff of the previous layer is the top_diff of the current layer, and the bottom_diff of the current layer is the top_diff of the next layer, so that the error can be transmitted layer by layer in the reverse direction.

[0059] In the calculation of formula (2), the operation between top_diff and W is similar to the operation process between the input neuron and the weight W, where top_diff is equivalent to the input feature map.

[0060] In the calculation of formula (3), the operation between top_diff and bottom_data is similar to the deep convolution operation, where top_diff is equivalent to the convolution kernel, which slides and accumulates in the XY direction of bottom_data (equivalent to neurons). In this operation scenario, the sizes of top_diff and bottom_data are usually relatively large. Therefore, the disclosed embodiment also provides an optimization solution for the convolution operation in this scenario (referred to as reverse deep convolution).

[0061] In the back propagation, for the convolution layer that performs the conventional 3D convolution operation, the operation in the reverse process can be called a cross-product convolution operation. The disclosed embodiment can also provide an optimization solution for this convolution operation.

[0062] Figure 4b An example of the cross-product convolution operation principle to which the disclosed embodiments can be applied is shown.

[0063] The figure shows a three-dimensional data top_diff of size [Ho Wo Co], which can be represented as a three-dimensional rectangle 410b of size Ho×Wo×Co; the figure also shows a three-dimensional data bottom_data of size [Hi Wi Ci], which can be represented as a three-dimensional rectangle 420b of size Hi×Wi×Ci. After performing a cross-convolution operation on top_diff and bottom_data, output data 430b is obtained, which is four-dimensional data of size [Co Kh Kw Ci], which can be represented as Co three-dimensional rectangles 430b of size Kh×Kw×Ci. Figure 4a By comparison, it can be seen that Figure 4b The cross-product convolution is equivalent to the reverse operation of the conventional 3D convolution, that is, the convolution kernel is calculated by the output feature map (top_diff) and the input feature map (bottom_data). Figure 4b The N dimension is omitted.

[0064] Specifically, for each HoWo face data in top_diff, that is, for each HoWo face of Co value, Ci copies are copied to obtain Ho×Wo×Ci data 440b. The data 440b is subjected to a deep convolution operation with the bottom data, that is, the Ci direction is not accumulated, thereby obtaining an output 460b, which is a three-dimensional data of size Kh×Kw×Ci. For each HoWo face, the copying and deep convolution operation are repeated to obtain Co three-dimensional data of size Kh×Kw×Ci, that is, a four-dimensional convolution kernel 430b of size Co×Kh×Kw×Ci is obtained.

[0065] In this article, input feature map, input data, neuron or input neuron can be used interchangeably; convolution kernel, filter or weight can be used interchangeably. In addition, H (height) and Y dimensions can be used interchangeably, and W (width) and X dimensions can be used interchangeably. Accordingly, the H dimension of the input feature map can be expressed as Hi or Yi, the H dimension of the output feature map can be expressed as Ho or Yo, and the W dimension is similarly expressed. In the disclosed embodiment, each convolution output point has a corresponding convolution window, and the shape of the convolution window is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and the weight within its convolution window. In addition, no matter which type of convolution operation is involved, the data involved can be divided into input feature map (input neuron), convolution kernel and output feature map (output neuron). For example, in the reverse depth convolution operation and the cross convolution operation, top_diff corresponds to the convolution kernel, bottom_data corresponds to the input feature map, and △W corresponds to the output feature map.

[0066] For the sake of simplicity, the scheme of applying the disclosed embodiment to conventional 3D convolution operation is called Forward4, the scheme of applying the disclosed embodiment to deep convolution operation is called Forward1, the splitting scheme of applying the disclosed embodiment to reverse deep convolution operation is called Update1, and the splitting scheme of applying the disclosed embodiment to cross product convolution operation is called Update4. Exemplary Convolution Optimization Scheme

[0067] In the disclosed embodiment, the dimensions of the multidimensional data involved are characterized as (N, H, W, C) or (Co, H, W, Ci), which represents the storage order of the data in the memory. It can be understood that although the multidimensional data has multiple dimensions, because the layout of the memory is always one-dimensional, there is a corresponding relationship between the multidimensional data and the storage order on the memory. Multidimensional data is usually allocated in a continuous storage space, that is, the multidimensional data can be expanded in one dimension and stored in sequence on the memory. For example, in the disclosed embodiment, the initial input feature map can be stored sequentially in a low-dimensional (here C / Ci is the lowest dimension) priority manner; and in order to optimize the convolution operation, the storage order of the input feature map can be adjusted during the operation, as will be described in detail later. Adjacent dimensions refer to dimensions that are adjacent to each other in the dimensional information representation of multidimensional data. For example, W and Ci are adjacent, and adjacent dimensions can also be called continuous dimensions.

[0068] In intelligent processors, due to the need for computing power and considerations of area and power consumption, the main hardware computing unit is the vector multiplication and addition unit. To support various convolution algorithms in hardware design, the essence is to maximize the extraction of multiplication and addition operations in the algorithm and implement them in on-chip RAM (such as Figure 3 Efficiently exchange input and output data of multiplication and addition operations between the NRAM, WRAM, etc. in the processor and the operator.

[0069] Hardware stores data in rows (cache rows) one by one. Read, write, and compute operations are most efficient when the entire row is aligned. Therefore, in order to fully utilize bandwidth and adapt to the memory access requirements of the operator array, it is usually necessary to vectorize and align the data. The design of artificial intelligence chips usually takes the Ci dimension as the lowest dimension, that is, the NHWC arrangement order mentioned above, and the data on the Ci dimension is continuous. Therefore, vectorization alignment requires that the size of the Ci dimension be aligned to a specified value, such as the alignment value M, so that the data is accessed in units of the alignment value M. Based on different hardware designs, M can have different values, such as 32B, 64B, 128B, etc. Usually, the input port size of the operator array is also related to M. For example, when the input data bit width is symmetrical, the input port size of the operator array is usually twice that of M, that is, the input feature map data and weight data of the alignment value M are processed at one time. When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.

[0070] When the Ci dimension of the input feature map is small or when the remainder obtained by dividing Ci by M (that is, the remainder after Ci is split by M) is small, for example, less than the size of a cache line, the Ci dimension needs to be padded to a row of data (for example, 64B), that is, filled with invalid data 0. This padding will cause a large amount of redundant calculations, resulting in resource waste and reduced operation efficiency. Alternatively, when performing a depth-wise convolution operation that does not accumulate in the Ci direction, this alignment requirement in the C direction will cause the operator efficiency to be too low and the performance to be unacceptable.

[0071] For convolution operations with smaller depth-wise and Ci directions, it is known that a small convolution scheme is proposed, in which the data in the W and H directions are folded into the C direction, thereby maximizing the bandwidth of the operator, for example, see Chinese invention patent application No. 2021111313885. However, in this small convolution scheme, many restrictions are imposed on software programmers, which makes it extremely inconvenient for software and hardware to work together. For example, in order to facilitate the hardware to fold the data in the W and H directions into the C direction and improve the effective utilization of the operator, the hardware requires that the data on the memory RAM is not arranged in a conventional natural order (such as HWC), but in a special block format (tiling format). When using this convolution instruction, the naturally arranged original data needs to be tiled through the tiling instruction, and the output data of the convolution instruction needs to be called again to perform the inverse tiling process. This processing method seriously affects the overall network computing efficiency and greatly increases the programming complexity of programmers. For example, the input and output data in different modes have different alignment restrictions, which is not convenient for software to understand and use. For example, in different modes of convolution operations, the storage sources of neuron data and weight data are different, which makes storage management inconvenient. In addition, there are multiple grouping methods in the same convolution operation mode, and the software needs to perceive the data division and overlap caused by the grouping method, which greatly increases the complexity of software use and the complexity of multiple hardware modes; and so on.

[0072] In view of this, in order to further optimize the convolution operation and reduce the software complexity, the disclosed embodiment proposes a small convolution optimization solution, which is optimized from multiple aspects to solve the various limitations mentioned above.

[0073] Specifically, on the one hand, the tiling process and the reverse tiling process originally executed by the software are implemented by the hardware, and the software does not need to call additional tiling instructions, thereby greatly improving the overall network performance and greatly reducing the programming complexity of the programmer. In addition, the hardware ensures that the placement format of the input and output neuron data in all modes is the same layout. Specifically, the input neurons, weights prepared by the programmer on the on-chip memory RAM, and the data placement format of the output neurons output by the hardware to the RAM are all naturally placed, for example, according to the storage order of the HWC, so that it is convenient for the software programmer to manage uniformly without distinguishing the modes. Since the placement format is unified, the alignment restrictions on the input and output data are also unified, which is convenient for software to understand and use. On the other hand, the neuron data and weight data are stored in the same storage circuit, which is convenient for the software programmer to manage the storage space uniformly. On the other hand, a fixed grouping method is implemented in the hardware configuration, and all modes adopt this fixed grouping method to reduce the complexity of software use and the complexity of hardware. The following will be described in detail for each optimization measure.

[0074] In the disclosed embodiment, regardless of the convolution operation mode, the input neuron data and weight data to be operated are stored in the on-chip memory (e.g. Figure 3 The first dimension order is, for example, a natural storage order, such as HWCi, where H is the height dimension, W is the width dimension, Ci is the input channel dimension, and Ci is the lowest dimension. In the data transmission path from the on-chip storage circuit RAM to the computing unit, the data placement format supplied to the computing unit is adjusted by hardware so that the data meets the requirements of the computing unit for the data placement format. For example, the computing unit requires that the input neuron data and weight data be aligned in blocks of fixed size, where the data within the block is arranged in the second dimension order, and the blocks are still arranged in the above-mentioned first dimension order (HWCi). The second dimension order is, for example, CiHW, where W is the lowest dimension.

[0075] Regardless of the convolution operation mode, the hardware will split and arrange the input neuron data and weight data in a unified form of data split blocks. It can be understood that the size of the data split block may be different depending on the processing power of the current and future hardware. In the embodiment of the present disclosure, assuming that the size of a cache line is 64B, the shape of the data split block can be fixed to 4×4×4B (H×W×C) = 64B. In the description of this article, this example is used as an example for description, but those skilled in the art can understand that the same principle can be applied to other hardware environments.

[0076] Figure 5 A schematic diagram of the layout format of neuron data according to an embodiment of the present disclosure is shown.

[0077] As shown in the figure, 510 represents three-dimensional (H, W and C, omitting the N dimension) neuron data to be calculated, which is stored in an off-chip storage circuit (e.g. Figure 2 The original storage order on the DRAM of the chip is HWC, with C being the lowest dimension. It is loaded into the on-chip storage circuit (e.g. Figure 3 After the NRAM is converted into NRAM, its arrangement format on the on-chip storage circuit is still in the original storage order (referred to as the "first dimension order" in this article), that is, each data is stored in the order of C dimension first, then W dimension, and finally H dimension.

[0078] In the disclosed embodiment, on the data transmission path from the on-chip storage circuit, such as (NRAM) to the computing unit (processing circuit), the neuron data is split into fixed data blocks and stored in dimensional order. The shape of each data block is, for example, fixed to 4×4×4B (H×W×C). When storing, the data in each data block is continuously stored as a cache line size (e.g., 64B), so that the subsequent processing circuit can perform calculations in units of data blocks.

[0079] In each data split block, the data is arranged in the order of the second dimension, which is CHW, and W is the lowest dimension. Between each data split block, the data can still be arranged in the order of the first dimension, that is, HWC.

[0080] As shown in the figure, neuron data 510 is split into multiple data split blocks 511-1, 511-2, etc. The neuron data after splitting and re-adjusting the storage order is shown as 520. In each data split block, the data is continuously placed in CHW order; between each data split block, the data is placed in HWC order, for example, in the order of data split blocks 511-1, 511-2, ... That is, the neuron data provided to the computing unit is stored in the order after this splitting and dimension conversion.

[0081] Similarly, after the computing unit performs the convolution operation to generate output neuron data, it stores the output neuron data in the data transmission path of the on-chip memory. No matter what format the output neuron data is in, the output neuron data will still be stored in the on-chip memory in the order of the first dimension, that is, the inverse tiling process is performed by hardware.

[0082] The above hardware execution of tiling and reverse tiling process unifies the format of input and output neuron data in the on-chip storage circuit RAM, that is, arranges them naturally, so as to facilitate unified management by software programmers. Furthermore, when the pipeline method is used, the processing time of the hardware tiling and reverse tiling process can be hidden in the pipeline, thereby improving the overall network computing efficiency.

[0083] In the convolution operation, the weights need to be multiplied and accumulated with the neurons in a corresponding manner. Accordingly, the weights also need to be arranged in a format that is adjusted by hardware in the data transmission path from the on-chip storage circuit RAM to the computing unit so that the weight data meets the requirements of the computing unit for the data arrangement format.

[0084] Furthermore, in order to facilitate the hardware to perform tiling and inverse tiling, a unified alignment standard can be provided for the input and output data of all convolution operation modes, thereby simplifying the complexity of the alignment requirements of each data as much as possible. The determination of the alignment standard will be explained later.

[0085] Table 1 shows the data storage format and alignment requirements in the Forward1 mode according to an embodiment of the present disclosure. Table 1: Data storage format and alignment requirements in Forward1 mode

[0086] In Table 1, stride indicates the stride of the dimension, which refers to the interval between adjacent elements in the current dimension in a higher dimension than the current dimension. For example, ci_stride indicates the stride of the input neuron in the ci dimension, that is, the interval between adjacent elements in the wi dimension in the ci dimension; wi_stride indicates the stride of the input neuron in the wi dimension, that is, the interval between adjacent elements in the hi dimension in the wi dimension; syn_ci_stride indicates the stride of the weight in the ci dimension, that is, the interval between adjacent elements in the kw dimension in the ci dimension; kw_stride indicates the stride of the weight in the kw dimension, that is, the interval between adjacent elements in the kh dimension in the kw dimension; co_stride indicates the stride of the output neuron in the co dimension, that is, the interval between adjacent elements in the wo dimension in the co dimension; wo_stride indicates the stride of the output neuron in the wo dimension, that is, the interval between adjacent elements in the ho dimension in the wo dimension.

[0087] As can be seen from the table above, in simple terms, all data is stored in the on-chip storage circuit in the original storage format, with only step size requirements in the C and W dimensions.

[0088] It can be understood that when the C and / or W dimensions in the original format of the data do not meet the step size requirements, it can be required to be processed according to the above-mentioned dimensional step size. In some embodiments, for the part that is not aligned to the dimensional step size in the C dimension, the software can be required to empty this part of the storage space. In this way, when reading from the on-chip storage circuit, the read address can be conveniently controlled. The hardware will automatically fill zeros in the C direction during the calculation process. Note that the hardware will not modify the space of the unaligned part of the C direction of the input neuron data on the RAM. For the part that is not aligned to the dimensional step size in the W dimension, the software can be required to empty this part of the storage space and fill it with zeros.

[0089] The data storage format and alignment requirements of other convolution modes are similar to those of Forward1, but slightly different depending on the specific algorithm. For example, the convolution mode of Forward4 accumulates the output results in the Ci direction, and the weights have an additional Co dimension, but other aspects are the same as Forward1. As a result, different convolution modes are consistent in data storage format and alignment requirements, which simplifies the processing of software programmers and eliminates the need for mode distinction and perception.

[0090] For ease of understanding, the following Table 2 lists the data storage format and alignment requirements in the Forward4 convolution operation mode. Table 2: Forward4 data format

[0091] Thus, the above describes the data storage format and alignment requirements on the on-chip storage circuit RAM when the tiling process or the reverse tiling process is performed by hardware, and the placement format provided to the processing circuit after the hardware tiling operation. The following further describes the process of hardware tiling in conjunction with a specific hardware circuit. Exemplary Computing Devices and Data Transfer Mechanisms

[0092] With the development of technology, the computing power of processing circuits has greatly increased, and the computing bandwidth has expanded accordingly. For example, the computing bandwidth of the processing circuit mainly responsible for vector operations was 128B / cycle in the past, but now the computing bandwidth of the vector operation processing circuit has become 512B / cycle, which has increased three times; and the computing bandwidth of the processing circuit responsible for matrix operations has increased from 64B / cycle to 128B / cycle. The increase in computing power will also bring some problems. One of the problems is that if the processing circuits are still concentrated in one place, it will become a large plate component. Furthermore, if each component or element is large, there will be many bubbles between the plates during hardware implementation, resulting in low area utilization.

[0093] In view of this, a computing device is provided in the disclosed embodiment, which groups and arranges the processing circuits to improve the area utilization of the device. Furthermore, based on this hardware layout, a data transmission mechanism is provided to improve the data transmission efficiency through multiple data transmission channels, thereby meeting the data demand of the processing circuits with increasing computing power, which is particularly suitable for the implementation of the aforementioned convolution optimization scheme.

[0094] Figure 6a FIG. 6 is a schematic block diagram of a computing device 600 according to an embodiment of the present disclosure. It can be understood that the structure can be regarded as Figure 3The computing device 600 can be configured to perform various types of convolution operations.

[0095] The computing device 600 includes a storage circuit 610 for storing data, such as an input feature map, a convolution kernel, an output feature map, etc. in a convolution operation. By storing all input and output data of the convolution operation on the same storage circuit 610, it is convenient to uniformly manage the storage space on the storage circuit. In this regard, the storage circuit 610 is equivalent to Figure 3 The data stored in the storage circuit 610 is stored according to the on-chip storage format described above.

[0096] The computing device 600 also includes a plurality of parallel processing circuits (PP), which are mainly responsible for the core calculations of deep learning algorithms, such as matrix multiplication and convolution operations. These parallel processing circuits can be organized and divided into groups to form a plurality of parallel processing circuit groups (PPB). Different parallel processing circuit groups PPB are located separately from each other at different positions of the circuit board, so as to facilitate full filling of the space on the circuit board and improve area utilization. Four parallel processing circuit groups PPB0 to PPB3 620 are shown in the figure, and are schematically shown as being located around the storage circuit 610. Those skilled in the art will appreciate that the number of parallel processing circuit groups may be more or less, depending on the specific hardware configuration, and the disclosed embodiments are not limited in this respect.

[0097] The storage circuit 610 and the plurality of processing circuit groups PPB may support a variety of data transmission mechanisms to support different multiplexing methods of input neurons and weights, thereby reducing the amount of data access during operation and improving processing efficiency. The specific data transmission mechanism will be described later.

[0098] The computing device 600 may also include one or more pre-processing circuits and / or post-processing circuits (not shown in the figure). In some implementations, there may be a pre-processing circuit between the storage circuit 610 and the plurality of processing circuit groups PPB to convert data from the storage circuit 610 into the data type required by the processing circuit group PPB, such as converting float32 data into int16, etc. Optionally or additionally, in some implementations, a post-processing circuit may also be included to perform some vector operations to support complex operations such as vector multiplication, addition, and nonlinear transformation, such as performing operations such as addition and subtraction of bias, or activation function operations on the results output by the processing circuit group PPB, and then writing them back to the storage circuit 610.

[0099] Furthermore, each parallel processing circuit group includes multiple parallel processing circuits, and the figure shows that the parallel processing circuit group PPB0 620 includes four parallel processing circuits PP0-PP3 621. Those skilled in the art will appreciate that the number of parallel processing circuits in each parallel processing circuit group may be more or less, depending on the specific hardware configuration, and the disclosed embodiments are not limited in this regard.

[0100] In addition to the parallel processing circuits PP0-PP3 621 contained in the parallel processing circuit group PPB0, a shared cache 622 is included for caching data from the storage circuit 610 for use by the parallel processing circuits PP0-PP3 therein. When applied to convolution operations, the shared cache 622 can be used to store neuron data. For example, in the Forward1 and Forward4 convolution modes, the input neurons are stored; in the Update1 and Update4 convolution modes, the bottom_data is stored.

[0101] Furthermore, each parallel processing circuit PP includes a plurality of operation circuits CU, and the figure shows that the parallel processing circuit PP3 621 includes four operation circuits CU0-CU3 631. Those skilled in the art will appreciate that the number of operation circuits in each parallel processing circuit PP may be more or less, depending on the specific hardware configuration, and the disclosed embodiments are not limited in this respect.

[0102] In addition to the included operation circuits CU0-CU3 631, the parallel processing circuit PP3 also includes a local cache 632 for caching data from the storage circuit 610 for use by the operation circuits CU0-CU3 therein. When applied to convolution operations, the local cache 632 can be used to store weight data. For example, in the Forward1 and Forward4 convolution modes, the weights are stored; in the Update1 and Update4 convolution modes, the top_diff is stored.

[0103] Figure 6a 6 also shows different transmission routes of neuron data and weight data in some embodiments, where the thick lines are neuron data and the thin lines are weight data. As can be seen from the figure, the neuron data read out from the storage circuit 610 is cached in the shared cache 622 and then distributed to the operation circuit CU in each parallel processing circuit PP; while the weight data read out from the storage circuit 610 is directly cached in the local cache 632 in each parallel processing circuit PP and then distributed to each operation circuit CU therein. It can be understood that in these implementations, there is a hard-wired connection path between the local cache 632 and the storage circuit 610 to directly cache the data from the storage circuit 610.

[0104] In the disclosed embodiment, based on the hardware architecture of the computing device, on the data transmission path from the storage circuit 610 to the computing circuit CU, the data is split into blocks in the form of the previous data splitting blocks and provided to the computing circuit, thereby facilitating the distribution of the convolution operation task on the processing circuits of these multiple parallel processing circuit groups to make full use of the computing power of the processing circuits.

[0105] Figure 6b Shown based on Figure 6a Schematic diagram of computing task allocation when the computing device implements the convolution operation in Forward1 mode. In this embodiment, it is assumed that the convolution step size in the H and W directions is 1, and it is assumed that the ci_stride of the input neuron is 16B, and it is assumed that the data types of the input neuron, weights, and output neurons are all fp16, that is, they occupy 2B. As shown in the figure, depending on the hardware conditions of the computing device (for example Figure 6a In one round of operation, 20×20×16B (HiWiC) of input neuron data needs to be sent to the operation unit to perform the convolution operation. A small square in the figure represents a 4×4×4B (HiWiC) = 64B data split block.

[0106] As shown in 650 in the figure, in one round of operation, 5×5×4 (HWC) 64B data blocks need to be sent to the operation unit for processing. In one round of operation, the 5×5×4×64B data is distributed on 4 parallel processing circuit groups PPB to perform operations simultaneously. According to the mapping relationship between output data and input data, these data can be divided into 4 overlapping data blocks (as shown in the following reference). Figure 8b The dotted boxes in 651 in the figure respectively show the data parts that each PPB is responsible for processing. It can be seen that each PPB needs to calculate 3×3×4×64B of data, and a total of 16×16×8 output data are calculated (as shown in 652, each small square represents 4×4×4 data), of which each PPB calculates 8×8×8 output data (as shown in the dotted box 653).

[0107] Figure 660 shows the task allocation of the 3×3×4×64B data within a single PPB. Figure 6a In the computing device, one PPB includes 4 PPs. When allocating tasks, 2×2×2×64B of data is taken out in each cycle (the data taken out in each cycle is shown by the dashed box 661 in the figure) and allocated to the 4 PPs for simultaneous processing. In this way, each PPB calculates a total of 4×4×4 output data in one cycle (as shown by the dashed box 663). In this way, after 8 cycles, the 3×3×4×64B of data can be traversed and processed, and 8×8×8 output data (662) can be output.

[0108] Further, FIG. 670 shows the task allocation of 2×2×2×64B data taken out in one cycle within one PPB. Since the tasks are allocated on a finer granularity basis, FIG. 670 shows the details of each data split block, that is, 2×2×2 64B data blocks are more finely granularly displayed as 8×8×8B data. Each PP takes out 5×5×8B of data from a fixed position of the 8×8×8B data for calculation. The dotted box 671 in the figure shows the data portion taken by each PP. In this way, each PP calculates 2×2×4 output data per cycle (as shown in the dotted box 663), and the four PPs calculate a total of 4×4×4 output data per cycle (as shown in 672).

[0109] Furthermore, FIG. 680 shows the task allocation for the fetched data of size 5×5×8B within a PP. Figure 6a A computing device, one PP includes 4 CUs. When allocating tasks, the data of 5×5×8B size taken out is allocated to 4 CUs for simultaneous processing, and each CU takes out 4×4×8B of data from a fixed position of the 5×5×8B data for calculation. The dotted box 681 in the figure shows the data portion taken by each CU. In this way, each CU calculates 1×1×4 output data per cycle (as shown in the dotted box 683), and the four CUs calculate a total of 2×2×4 output data per cycle (as shown in 682).

[0110] Thus, the above describes the allocation of convolution operation tasks on the computing device in the Forward 1 mode. The task allocation in the Forward 4 mode is similar to that in the Forward 1 mode, and will not be described in detail here.

[0111] As mentioned above, the computing power of the processing circuit has been greatly improved. For example, the computing bandwidth of each computing circuit CU in the parallel processing circuit PP is 128B / cycle, and 4 computing circuits CU require 512B / cycle. However, in order to improve the area utilization, the parallel processing circuits are arranged in groups, and the data transmission bandwidth between each PPB 620 and the storage circuit 610 is correspondingly narrowed, less than 512B / cycle. Therefore, it is necessary to design a data transmission mechanism that can provide each PPB with the required amount of data as quickly as possible to perform operations.

[0112] Figure 7 An exemplary structural block diagram of a data transmission device 700 according to some embodiments of the present disclosure is shown.

[0113] As shown in the figure, the data transmission device 700 includes a storage circuit 710 for storing data, which corresponds to Figure 6aThe data transmission device 700 further includes D processing circuit groups PPB 720 for performing operations on data, wherein D>1, that is, there are multiple processing circuit groups PPB that can perform operations concurrently. Figure 7 4 processing circuit groups PPB0 to PPB3 720 are also exemplarily shown in FIG.

[0114] The data transmission device 700 also includes a first data transmission path. Specifically, the first data transmission path may include: RS first read ports 731, used to read data from the storage circuit 710, where RS is a positive integer; a first distribution circuit 732, used to divide the data read from the RS first read ports into D data blocks, and distribute them to D first data channels for transmission; the D first data channels 733, used to transmit the distributed data blocks to the correspondingly connected D processing circuit groups 720; and a data exchange channel 734 between the D processing circuit groups 720, used to exchange data between the D processing circuit groups.

[0115] In some embodiments, RS>1, that is, there are multiple first read ports, so that the multiple first read ports can read data from the storage circuit 710 concurrently according to the instruction of the read request, thereby expanding the read data bandwidth. Since multiple separate read ports are used, it is convenient to reduce the size of a single component and facilitate compact layout on the circuit board. In addition, different first read ports support reading data with discrete addresses, that is, the addresses read by each first read port can be different, which is convenient for simultaneously retrieving data at non-continuous addresses when needed, such as in small convolution optimization operations. Figure 7 Two first read ports are shown in FIG. 1 , and the read data bandwidth of each read port is 256 B / cycle, and the total bandwidth of the two read ports is 512 B / cycle.

[0116] In some embodiments, the read request can be in the granularity of one-shot operation. The interaction between the storage circuit and the external device is based on the read request, so the granularity of the read request determines the granularity of the data interaction. In the existing read request, a three-dimensional amount of data is usually provided at one time, and the storage circuit needs to execute multiple shots to complete the read request. In the disclosed embodiment, the read request is in the granularity of one-shot operation, which can simplify the processing, such as data dependency judgment. Usually, when making a data dependency judgment, it is judged whether there is a dependency between the data read before and after according to the data range of the read request. It can be understood that the larger the data range, the greater the probability of dependency. Even if the data itself does not actually have a dependency, it may be judged to be dependent due to the overlap caused by the large data range, thereby causing a hardware pipeline interruption and reducing processing efficiency. In the disclosed embodiment, the read request is in the granularity of one-shot operation, which more accurately narrows the data range and reduces the possibility of misjudging data dependency.

[0117] The number RS ​​of first read ports may not match the number D of processing circuit groups, so the first distribution circuit 732 can divide the data read from the RS first read ports into D data blocks and distribute them to D first data channels for transmission to the D processing circuit groups.

[0118] In some implementations, RS first distribution circuits 732 may be configured to connect to RS first read ports 731 , respectively. Figure 7 Two first distribution circuits are shown, each connected to a first read port.

[0119] In order to fully utilize the data transmission bandwidth, the data read from the first read port is evenly split to be distributed to the corresponding first data channels 733. The total bandwidth of the RS first read ports is equal to the total bandwidth of the D first data channels. Figure 7 The 256B data read by each first read port is split into two 128B data, which are provided to the connected processing circuit group through the corresponding first data channels respectively. The bandwidth of each first data channel is 128B / cycle.

[0120] As mentioned above, the processing circuit group needs 512B / cycle data to exert computing power. Only one 128B / cycle of data can be provided through the first data channel, so other data channels are needed to provide the remaining three 128B / cycles. In the disclosed embodiment, these data are realized through the data exchange channel 734 between the processing circuit groups 720, thereby meeting the demand for data supply of the growing computing power.

[0121] When data is exchanged between processing circuit groups, the data transmission direction can be determined according to the information contained in the read request. Specifically, in some embodiments, the read request also includes a mark for the read data, which is used to indicate the processing circuit group that will use the read data.

[0122] Thus, each processing circuit group can process the received data in the following manner: cache the data marked as being used by itself locally (for example, Figure 6a and / or, transmitting data marked as to be used by other processing circuit groups to the marked other processing circuit groups through the data interaction channel 734.

[0123] The above-mentioned mark can be generated by an instruction decoder. For example, the instruction decoder marks the corresponding data according to the specific parameters of the small convolution instruction and the task scheduling allocation of the small convolution operation among these multiple processing circuit groups. The granularity of the mark can be determined according to the operation block of the data and the continuous data block taken out from the storage circuit. It can be understood that in different scenarios, the size of the operation block of the data and the continuous data block taken out may change, so the granularity of the mark can also change accordingly.

[0124] Based on the above tags, the first data transmission path can support a customized distribution mechanism, which can distribute different data blocks to different processing circuit groups 620 according to the indication information. Moreover, through the combination of the first data channel and the data interaction channel, it can be ensured that sufficient data is supplied to the processing circuit group in a timely manner.

[0125] In the convolution operation, in addition to the neuron data, weight data is also included, which is also stored in the storage circuit 710. Therefore, the data transmission device 700 may also have a second data transmission path to support the transmission of other input data in the operation.

[0126] Considering that the data (such as weight data) to be transmitted by the second data transmission path is generally small in size, and the weight data is usually multiplexed between processing circuit groups, the second data transmission path does not need as large a bandwidth as the first data transmission path, and the line design can be simplified. For example, 128B / cycle of weight data can be provided for each processing circuit group PPB.

[0127] In some embodiments, the second data transmission path may include: RL second read ports 741, used to read data from the storage circuit 710; a second distribution circuit 742, used to transmit the data read from the RL second read ports to all or part of the D second data channels 743 according to the instruction; and the D second data channels 743, used to transmit the distributed data to the correspondingly connected D processing circuit groups 720. For example, the second distribution circuit may transmit the data to all the D second data channels 743 by broadcasting, or to part of the D second data channels 743 by multicasting, or to a specified one of the D second data channels by unicasting.

[0128] In the above embodiments, only one second read port can be configured, and RL=1 at this time, and its bandwidth is, for example, 256B / cycle, but only 128B / cycle can be used. In other embodiments, two second read ports can be configured, and RL=2 at this time, and the bandwidth of each second read port can still be, for example, 256B / cycle, but in actual use, each second read port can only use 64B / cycle. Similar to the first read port, multiple separate second read ports are used to facilitate the reduction of the size of a single component and facilitate compact layout on the circuit board. In addition, different second read ports support reading data with discrete addresses, that is, the addresses read by each second read port can be different, thereby facilitating the simultaneous retrieval of data on non-continuous addresses when needed, so as to better support the hardware tiling scheme of the disclosed embodiment.

[0129] Furthermore, considering that the weight data will be reused between the processing circuit groups in the small convolution optimization scheme, the second distribution circuit 742 is configured to realize the broadcast, multicast or unicast of the data. For example, when the weight data needs to be multiplexed between all D processing circuit groups 720, the weight data can be copied D-1 copies, thereby providing a copy of data for each second data channel to be transmitted to the connected processing circuit group. When the weight data only needs to be multiplexed between some processing circuit groups, the corresponding number of copies can be copied and allocated to the corresponding second data channel, thereby realizing the multicast of the data. In the case of unicast, the second distribution circuit directly allocates the data to the corresponding second data channel. In some implementations, RL second distribution circuits 742 can be configured to respectively connect to RL second read ports 741. Figure 7 Two second distribution circuits are shown, each connected to a second read port.

[0130] In some embodiments, when the data transmission device 700 is used to transmit data of convolution-type operations, the first data transmission path can be used to transmit first data, such as neuron data, and the second data transmission path can be used to transmit second data, such as weight data. Exemplary RAM Discrete Memory Access Scheme

[0131] Since the data in the on-chip RAM is arranged in a natural order according to the hardware tiling scheme of the disclosed embodiment, and the data required by the processing circuit is a data split block arranged in a certain order, the data required by the processing circuit in a single round of operation may not be stored continuously in the on-chip RAM. To this end, the disclosed embodiment provides a discrete memory access scheme for the on-chip RAM, for example, each read port can support reading Ad discrete addresses at a time, Ad>1, so as to provide the processing circuit with the dispersed storage data required in a single round of operation.

[0132] Figure 8aThe figure shows the input neuron data distribution required for a single round of Forward1 operation according to an embodiment of the present disclosure. In this embodiment, it is assumed that the convolution step lengths in the H and W directions are both 1, and it is assumed that the ci_stride of the input neuron is 16B. As shown in the figure, depending on the hardware conditions of the computing device (e.g. Figure 6a In one round of operation, 20×20×16B (HiWiC) of input neuron data needs to be sent to the operation unit to perform the convolution operation. A small square in the figure represents 16B of data in the C dimension. It can be understood that since ci_stride = 16B, the adjacent small squares in the Wi direction are stored continuously on the on-chip RAM, while the adjacent small squares in the Hi direction are stored discretely.

[0133] Figure 8a 810 shows the 20×20×16B input neuron data that needs to be taken out from the on-chip RAM. These input neuron data are distributed to four processing circuit groups PPB to perform operations, and each processing circuit group processes 12×12×16B of data 820. There is overlap between the input neuron data processed by different processing circuit groups, as shown in the dotted box.

[0134] In some embodiments, in order to simplify the decoder work, the 20×20×16B data needs 13 cycles to be piped, instead of 12.5 cycles (20×20×16B / 512B / cycle=12.5cycle). In order to better match the processing circuit's requirements for the swing format of the operation data (the data split block is 4×4×4B), based on Figure 7 In the data transmission device, the first data transmission path can be supplied with data through two first read ports, and each first read port reads 4×4×16B (HWC)=256B of data per beat. Furthermore, since the adjacent small squares in the Hi direction are discretely stored, each first read port also supports reading multiple discrete addresses at a time. Specifically, in the disclosed embodiment, each first read port supports reading 4 discrete addresses at a time, that is, Ad=4.

[0135] As shown in the figure, it schematically shows 4×4×16B data 830 and 840 read by two first read ports RS0 and RS1 in the first beat. Furthermore, 4 addresses read by each first read port are also shown, corresponding to the first element address of 4 rows of data. Different grayscales are used to show mutually discontinuous data.

[0136] Figure 8b The first read port allocation for supplying the 20×20×16B neuron data is shown. For the sake of clarity, the 4×4×16B=256B data read by each first read port per beat is represented as a cubic block, which can correspond to Figure 8a 830 or 840. Figure 8b The first read port number for each data block in the WH direction is marked in 850. For example, in the first beat, the data block with WH coordinate (1,1) is read through the RS0 port, and the data block with WH coordinate (1,2) is read through the RS1 port; in the second beat, the data block with WH coordinate (1,3) is read through the RS0 port, and the data block with WH coordinate (1,4) is read through the RS1 port; in the third beat, the data block with WH coordinate (1,5) is read through the RS0 port, and the data block with WH coordinate (2,1) is read through the RS1 port; and so on.

[0137] As mentioned above, the 20×20×16B input neuron data is distributed to four processing circuit groups to perform operations, and each processing circuit group processes 12×12×16B of data. Therefore, according to the customized distribution mechanism described above, it is necessary to mark the processing circuit groups to which these data will be distributed.

[0138] In some embodiments, in order to ensure that the split input neurons can share weights, they can be divided according to the Ho / Wo direction of the output neurons, thereby mapping back to the division of the input neurons. In some embodiments, the input neurons can be divided as follows: according to the size of the corresponding output neurons, the output neurons are divided into D output data blocks in the HW dimension (that is, the Ho / Wo dimension); and according to the input neuron area required to calculate each output data block, the input neurons are divided into D input data blocks in the HW dimension (that is, the Hi / Wi dimension) to be allocated to D processing circuit groups. It can be understood that depending on the weight scale and the convolution step size, the input neurons corresponding to adjacent output points in the output neurons may overlap.

[0139] Figure 8b 860 in the figure shows the splitting of the output neurons among the four processing circuit groups. In this example, the output neuron Wo×Ho=16*16 is divided into four output data blocks of the same shape in a 2×2 manner in the HW direction, and respectively assigned to four processing circuit groups. Each output data block contains an output neuron of size 8×8×16B (HoWoC). The processing circuit group number for calculating each output data block in the WH direction is marked in 860. For example, the output data block in the upper left corner of the figure is assigned to processing circuit group PPB0, the upper right corner is assigned to PPB1, the lower left corner is assigned to PPB2, and the lower right corner is assigned to PPB3.

[0140] Then, based on the division of the output neurons, the division of the input neurons is reversely deduced. As shown in the figure, 850 further marks which processing circuit group each data block in the WH direction needs to be supplied to. In some embodiments, the processing circuit group corresponding to the data block can be indicated by an enable bit. For example, for 4 processing circuit groups, a 4-bit enable signal can be used, and each bit indicates whether the data needs to be transmitted to a corresponding processing circuit group. For example, "1" indicates that it needs to be transmitted to the corresponding PPB, and "0" indicates that it does not need to be transmitted to the corresponding PPB. For example, it can be seen from 850 that the data in the third row and the third column needs to be transmitted to at least 2 processing circuit groups, while the data in the center needs to be transmitted to 4 processing circuit groups. Through Figure 7 The data transmission device described can quickly transmit data to the corresponding processing circuit group according to these marks, so as to realize efficient data supply. It should be understood that the marks in 850 are only schematic, and the granularity of the marks can be determined according to the operation block of the data and the continuous data block taken from the storage circuit. For example, in this embodiment, the operation block of the data is 4×4×16B (HWC), but 4 discrete addresses are read per beat, and the size of the continuous data block taken is 1×4×16B (HWC), so it can be marked according to the granularity of 1×4×16B. For the sake of simplicity and clarity of the illustration, finer-grained marks are not further shown in 850.

[0141] Figure 8c The figure shows the distribution of weight data required for a single round of Forward1 operation according to some embodiments of the present disclosure. In these embodiments, the weight data is shared among the four processing circuit groups, that is, the weights are reused among the processing circuit groups. Figure 8a The conditions are similar, assuming that the weight syn_ci_stride = 16B. In one round of operation, 4×4×16B (KhKwC) weight data needs to be sent to the processing circuit to perform the convolution operation. A small square in the figure represents 16B of data in the C dimension. It can be understood that since syn_ci_stride = 16B, the adjacent small squares in the Kw direction are stored continuously on the on-chip RAM, while the adjacent small squares in the Kw direction are stored discretely.

[0142] In some embodiments, the second data transmission path of the data transmission device can be used to supply weight data to the four processing circuit groups. In some implementations, the second data transmission path includes two second read ports. Therefore, data can be supplied to the four processing circuit groups in two steps, and each step is read by two second read ports to read 1×4×16B of weight data. It can be understood that since two independent second read ports are used, the two second read ports can support independent memory access addresses. Figure 8cAs shown, it schematically shows 1×4×16B data 870 and 880 read by two second read ports RL0 and RL1 in the first beat respectively. The data read by the two read ports are not continuous on the RAM. Different grayscales are used to show mutually discontinuous data.

[0143] Furthermore, considering that the weight data is multiplexed between the processing circuit groups, the read weight data may be copied to be transmitted to all the processing circuit groups. Figure 8c 890 shows that the weight data read by each second read port is copied three times, thereby providing one copy of weight data for each processing circuit group.

[0144] Therefore, using Figure 7 The described data transmission device can quickly transmit neuron data and weight data to multiple processing circuit groups to meet the data requirements of parallel processing.

[0145] When the step size of the ci dimension takes different values, the above principle is still applicable. Considering the computational limitations of the processing circuit, the number of data in the H and W directions needs to be adjusted.

[0146] Figure 9a The distribution of input neuron data required for a single round of operation of Forward1 according to another embodiment of the present disclosure is shown, where ci_stride=32B, syn_ci_stride=32B. In this embodiment, it is also assumed that the convolution step sizes in the H and W directions are both 1. When the step size of the Ci dimension is 32B, in one round of operation, 12×20×32B (HiWiC) of input neuron data needs to be sent to the processing circuit to perform the convolution operation. A small square in the figure represents 32B of data in the C dimension. It can be understood that since ci_stride=32B, the adjacent small squares in the Wi direction are stored continuously on the on-chip RAM, while the adjacent small squares in the Hi direction are stored discretely. Since the 32B data in the Ci dimension is now continuous, if the data is still taken in the manner of continuous data acquisition every 16B in the Ci dimension in the previous example, there will be a large number of discrete memory access addresses, which increases the memory access overhead. Therefore, in this scenario, the amount of data in each dimension read each time can be adjusted to make the memory access mode consistent with the previous one.

[0147] Figure 910 shows the 12×20×32B input neuron data that needs to be taken out from the on-chip RAM. Figure 8aSimilar to the embodiment of , neuron data is provided through the first data transmission path. Specifically. Data is provided through two first read ports, each first read port reads 4×2×32B (HWC) = 256B of data per beat, and each first read port supports reading 4 discrete addresses at a time. 910 schematically shows 4×2×32B of data 930 and 940 read by the two first read ports RS0 and RS1 in the first beat, respectively. Furthermore, the 4 addresses read by each first read port are also shown, corresponding to the first element addresses of 4 rows of data, respectively. The figure shows mutually discontinuous data with different grayscales.

[0148] In some embodiments, the processing circuit is limited to a single hardware filling that can only process 16B of operations on the Ci dimension, thereby facilitating the unification of the computational behavior of the processing circuit. In the case of ci_stride=32B above, it is necessary to readjust the division of computational tasks of the processing circuit group in a single round of operations, that is, the division of neuron data between the processing circuit groups. In one implementation, the first 16B of data for ci=0-15 can be allocated to PPB0 and PPB1, and the last 16B of data for ci=16-31 can be allocated to PPB2 and PPB3.

[0149] Figure 9b The figure shows the distribution of the first read ports supplying the 12×20×32B input neuron data and the distribution of the neuron data between the processing circuit groups according to another embodiment of the present disclosure. For the sake of clarity, the 4×2×32B=256B data read by each first read port per beat is split into two cubic blocks according to Ci, one cubic block represents ci=0-15, and the other cubic block represents ci=16-31. Figure 9b The first read port number for each data block in the WH direction is marked in 950, which will not be described in detail here. Furthermore, 950 also marks which processing circuit group each data block in the WH direction needs to be supplied to. In this embodiment, the data operation segmentation block is still 4×4×16B (HWC), but 4 discrete addresses are read per beat, and the size of the continuous data block for fetching is 1×2×32B (HWC), so it can be marked according to the granularity of 1×2×16B, that is, the smaller value of each dimension is taken as the granularity. For the sake of simplicity and clarity of the diagram, finer-grained markings are not further shown in 950. It can be seen from 950 that the first 16B data of ci=0~15 are allocated to PPB0 and PPB1, and the last 16B data of ci=16~31 are allocated to PPB2 and PPB3. From the perspective of the processing circuit, the shape of the allocated neuron data is 12×12×16B (HiWiC), so the operational behavior of the processing circuit can be kept consistent.

[0150] Fig.9cThe distribution of weight data required for a single round of operation of Forward1 according to another embodiment of the present disclosure is shown. In one round of operation, 4×4×32B (KhKwC) of weight data needs to be sent to the processing circuit to perform the convolution operation. A small square in the figure represents 32B of data in the C dimension. In the transmission through the second data transmission path, data can be supplied to 4 processing circuit groups through four beats, and 1×2×32B of weight data are read by 2 second read ports in each beat. 960 schematically shows 1×2×32B of data 970 and 980 read by the two second read ports RL0 and RL1 in the first beat. The data read by these two read ports are not continuous on the RAM. The mutually discontinuous data are shown in different grayscales in the figure. Note that although 960 shows the first beat reading data 970 and 980, it can also be read in other orders, and it is only necessary to take these data in four beats, for example, the data can be taken row by row in 960. Furthermore, in order to reuse the weights, FIG. 990 shows that the weight data read by each second read port is copied three times, thereby providing each processing circuit group with a weight data. Note that although in this embodiment, according to the division of the computational tasks of the processing circuit group in a single round of computation, the weight data of ci=0 to 15 is only reused between the processing circuit groups PPB0 and PPB1, and the weight data of ci=16 to 31 is only reused between PPB2 and PPB3, in order to maintain consistent hardware behavior, three copies can still be made, and after being transmitted to the processing circuit group, only the weight data to be processed by itself is selected for storage as needed.

[0151] Fig.10a The distribution of input neuron data required for a single round of operation of Forward1 according to another embodiment of the present disclosure is shown, where ci_stride=64B, syn_ci_stride=64B. In this embodiment, it is also assumed that the convolution step sizes in the H and W directions are both 1. When the step size of the Ci dimension is 64B, in one round of operation, 12×12×64B (HiWiC) of input neuron data needs to be sent to the processing circuit to perform the convolution operation. A small square in the figure represents 64B of data in the C dimension. It can be understood that since ci_stride=64B, the adjacent small squares in the Wi direction are stored continuously on the on-chip RAM, while the adjacent small squares in the Hi direction are stored discretely. Similarly, the amount of data in each dimension read each time can be adjusted to keep the memory access mode consistent with the previous one.

[0152] Figure 1010 shows 12×12×64B of input neuron data that needs to be taken out from the on-chip RAM. Similarly, neuron data is provided through the first data transmission path. Specifically. Data is provided through two first read ports, each first read port reads 4×1×64B (HWC) = 256B of data per beat, and each first read port supports reading 4 discrete addresses at a time. 1010 schematically shows 4×1×64B of data 1030 and 1040 read by the two first read ports RS0 and RS1 in the first beat, respectively. Furthermore, the four addresses read by each first read port are also shown, corresponding to the first element addresses of the four rows of data. The figure shows mutually discontinuous data with different grayscales.

[0153] Similarly, in some embodiments, the processing circuit is limited to a single hardware filling that can only process 16B operations on the Ci dimension, thereby facilitating the unification of the computational behavior of the processing circuit. In the above case of ci_stride=64B, it is necessary to readjust the division of computational tasks of the processing circuit group in a single round of operations, that is, the division of neuron data between processing circuit groups. In one implementation, the first 16B data of ci=0-15 can be assigned to PPB0, the second 16B data of ci=15-31 can be assigned to PPB1, the third 16B data of ci=32-47 can be assigned to PPB2, and the fourth 16B data of ci=48-63 can be assigned to PPB3.

[0154] Fig.10b The distribution of the first read port supplying the 12×12×64B input neuron data and the distribution of the neuron data between the processing circuit groups according to another embodiment of the present disclosure is shown. For the sake of clarity, the 4×1×64B=256B data read by each first read port per beat is split into four cubic blocks according to Ci, and each cubic block represents 16B of data on ci. Fig.10bThe first read port number for each data block in the WH direction is marked in 1050, which will not be described in detail here. Furthermore, 1050 also marks which processing circuit group each data block in the WH direction needs to be supplied to. In this embodiment, the data operation segmentation block is still 4×4×16B (HWC), but 4 discrete addresses are read per beat, and the size of the continuous data block for fetching is 1×1×64B (HWC), so it can be marked according to the granularity of 1×1×16B, that is, the smaller value of each dimension is taken as the marking granularity. For the sake of simplicity and clarity of the diagram, finer-grained markings are not further shown in 1050. It can be seen from 1050 that the first 16B data of ci=0~15 is assigned to PPB0, the second 16B data of ci=15~31 is assigned to PPB1, the third 16B data of ci=32~47 is assigned to PPB2, and the fourth 16B data of ci=48~63 is assigned to PPB3. From the perspective of the processing circuit, the shape of the allocated neuron data is still 12×12×16B (HiWiC), so the computational behavior of the processing circuit can be kept consistent.

[0155] Fig.10cThe distribution of weight data required for a single round of operation of Forward1 according to another embodiment of the present disclosure is shown. In one round of operation, 4×4×64B (KhKwC) of weight data needs to be sent to the processing circuit to perform the convolution operation. A small square in the figure represents 64B of data in the C dimension. In the transmission through the second data transmission path, data can be supplied to four processing circuit groups through eight beats, and 1×1×64B of weight data are read by two second read ports in each beat. 1060 schematically shows 1070 and 1080 of 1×1×64B of data read by two second read ports RL0 and RL1 in the first beat. The data read by these two read ports are not continuous on the RAM. The figure shows mutually discontinuous data with different grayscales. Note that although 1060 shows the first beat reading data 1070 and 1080, the reading order can also be in other orders, and the data only needs to be taken in eight beats, for example, the data can be taken row by row in 1060. Furthermore, in order to reuse the weights, the figure 1090 shows that the weight data read by each second read port is copied 3 times, thereby providing each processing circuit group with a weight data. Note that although in this embodiment, according to the division of the operation tasks of the processing circuit group in a single round of operation, the weight data of ci=0-15 is only used in the processing circuit group PPB0, the weight data of ci=16-31 is only used in PPB1, the weight data of ci=32-47 is only used in the processing circuit group PPB2, and the weight data of ci=48-63 is only used in PPB3, that is, the weight data is not reused between PPBs, but in order to keep the hardware behavior consistent and the subsequent connection simple, it can still be copied 3 times, and after being transmitted to the processing circuit group, only the weight data to be processed by itself is selected for storage as needed.

[0156] The above provides an exemplary implementation scheme when ci_stride takes different values. It can be understood that the actual data size ci_real on the Ci dimension does not exceed ci_stride. Therefore, for the scene of ci_stride = 16B, it can handle the situation of ci_real = 1B to 16B; for the scene of ci_stride = 32B, it can handle the situation of ci_real = 1B to 32B; for the scene of ci_stride = 64B, it can handle the situation of ci_real = 1B to 64B; and so on. Example RAM structure supporting discrete memory access

[0157] The following is an exemplary scheme for simultaneously executing multiple discrete address memory accesses from a storage circuit through a single port provided by the embodiments of the present disclosure. SRAM (static random access memory) is usually used to store relatively large amounts of data. SRAM can use banks (memory bodies) to organize storage units to improve access speed while achieving large-capacity storage. Different banks can be accessed independently in the same cycle or in successive cycles. SRAMs of the same capacity (e.g., 1MB) can also be constructed in many ways, such as the number of banks N. bank , the bank width Wid bank , the depth of the bank Dep bank etc. can be different. In implementation, these parameters are preferably powers of 2.

[0158] Fig.11 A schematic structural diagram of a storage circuit implemented using SRAM according to an embodiment of the present disclosure is shown. The figure shows an implementation of a 1MB SRAM, which includes 128 banks. The bit width of each bank is 16B and the depth is 512, so 1MB = 128 * 16B * 512 = 2KB * 512. The maximum access bandwidth of the data transmission path is 2048B.

[0159] As shown in the figure, a small square represents a bank with a depth of 512, which are numbered from 0 to 127, and the depth direction is not shown. Banks with adjacent numbers are continuous in address. These 128 banks can be divided into different groups, and each group can be further divided into different subgroups. The figure shows a bank partitioning scheme. In this partitioning scheme, you can first allocate a bank to each group in sequence, and allocate it in a loop until all 128 banks are allocated. Then, within each group, further allocate a bank to each subgroup in sequence, and allocate it in a loop until all banks in the group are allocated. The number of banks in the example in the figure is N group =4, the number of sub-groups of each bank in N subgroup / group =4, the total number of subgroups N subgroup =N group ×N subgroup / group = 16. By dividing and arranging the subgroups in this way in a disrupted bank order, each read / write port can access one bank in each subgroup at a time when performing memory access, thereby supporting one-time memory access to multiple discrete addresses. subgroup≥Ad, a single read / write port can access Ad discrete addresses at a time. It should be understood that the above bank partitioning scheme is only an example, and there can be other partitioning schemes that can support accessing multiple discrete addresses at a time. For example, according to different application scenarios, such as convolution operations or vector calculations, the data storage order and the calculation requirements for the data storage order are different, and different bank partitioning schemes can be used to adapt to as many application scenarios as possible.

[0160] Furthermore, in order to reduce the number of connections, it is possible to restrict the access to only one address in the depth direction of a bank. For example, in the example in the figure, there are 16 subgroups in total, each subgroup includes 8 banks, and the depth of each bank is 512. In terms of functionality, it is possible to support simultaneous access to 16 subgroups each time, each subgroup selects 1 bank from 8 banks, and each bank selects an address from the depth of 512.

[0161] With this structure, SRAM can support simultaneous access to 16 banks in 16 subgroups through a single read / write port. When there are multiple ports, for example, 2 ports, one shot can access 2 banks in each subgroup, that is, 8 out of 2, but multiple ports can still only access 512 out of 1 in depth, so repeated selection is not supported in 8 out of 2.

[0162] In the 16 16B data accessed by a single port at a time, each 16B of data is continuous, but these 16 16B of data can be continuous or discontinuous. On this basis, when some or all of these 16 16B of data are continuous, SRAM can support simultaneous access to 8 32B of data, 4 64B of data, 2 128B of data, or 1 256B of data at a time. It can be seen that each read / write port can access 256B of data in one shot.

[0163] When applied to the convolution optimization operation mentioned in this disclosure, it can be seen from the above RAM discrete memory access scheme that the RAM needs to support each read / write port to access 4 64B data in one beat, and these 64B data are continuous. That is, each read / write port supports 4 discrete memory access addresses per beat.

[0164] Fig.12From another perspective, a schematic structural diagram of a storage circuit implemented using SRAM according to an embodiment of the present disclosure is shown. In this figure, 128 banks are allocated to 16 subgroups in sequence according to the order of numbers, and the allocation is cyclic. Adjacent numbers indicate that the addresses are continuous. In some embodiments of the present disclosure, in order to support one-shot access to 4 64B data, it is necessary to limit the address relationship of these 4 64B data on the RAM to meet the aforementioned simultaneous access conditions, that is, the data accessed simultaneously each time is located in 16 banks in the 16 subgroups.

[0165] In some embodiments, when the size of the read continuous data blocks is size=64B, the memory access address step length stride between these data blocks may be required to meet the following condition: stride=128B*N+64B. Accordingly, the first addresses of the four continuous data blocks are Addr0, Addr1, Addr2 and Addr3, where Addr1=Addr0+128*N+64, Addr2=Addr0+256*N+128, and Addr3=Addr0+384*N+192.

[0166] When N is an odd number, the four memory access addresses are:

[0167] Addr0

[0168] Addr1=Addr0+128*N+64=Addr0+256*P+192

[0169] Addr2=Addr0+256*N+128=Addr0+256*Q+128

[0170] Addr3=Addr0+384*N+192=Addr0+256*R+64

[0171] P, Q, and R are natural numbers. Simply put, Addr0, Addr1, Addr2, and Addr3 are summed up to extract the offset multiple of 256B (16 banks are exactly 16B*16=256B). The remaining offsets are exactly 0, 192, 128, and 64, that is, they are exactly distributed on the 16 banks and can be read out simultaneously in one shot.

[0172] Similarly, when N is an even number, the four memory access addresses are:

[0173] Addr0

[0174] Addr1=Addr0+128*N+64=Addr0+256*P+64

[0175] Addr2=Addr0+256*N+128=Addr0+256*(2P)+128

[0176] Addr3=Addr0+384*N+192=Addr0+256*(3P)+192

[0177] P is a natural number. Similarly, Addr0, Addr1, Addr2 and Addr3 are processed as an address summary, and the part with an offset that is a multiple of 256B (16 banks is exactly 16B*16=256B) is extracted. The remaining offsets are exactly 0, 64, 128, and 192, that is, they are exactly distributed on the 16 banks and can be read out simultaneously in one beat.

[0178] Fig.12 Two exemplary memory access scenarios are shown in the figure, where light gray represents the case where N=1 (odd number), and dark gray represents the case where N=2 (even number). As shown in the figure, for the case where N=1, assuming that the first address of the four addresses accessed simultaneously, Addr10=0, then according to the above restrictions, it can be deduced that Addr11=128+64, Addr12=256+128, and Addr13=384+192. For the case where N=2, assuming that the first address of the four addresses accessed simultaneously, Addr20=1024, then according to the above restrictions, it can be deduced that Addr21=1024+256+64, Addr22=1024+512+128, and Addr23=1024+768+192. From Fig.12 It can be seen that the data accessed in each beat are located in 16 banks of 16 subgroups, which meets the conditions for simultaneous memory access.

[0179] Based on the above situation, when there is N subgroup bank subgroups, each bank has a bit width of Wid bank It can support up to N simultaneous accesses at a time. subgroup Wid bank If a single read / write port is expected to read Ad discrete addresses at a time, and the amount of continuous data read from each discrete address is size, then the following conditions exist: N subgroup ×Wid bank ≥Ad×size, and each discrete address can be split into N subgroup ×Wid bank The integer part and remainder part of the number, where the remainder parts of each discrete address are at least size apart and distributed in the N subgroup On a subgroup.

[0180] In some cases, the storage circuit can be designed as Nsubgroup ×Wid bank =Ad×size, at this time, the step length stride between the Ad discrete addresses accessed at the same time satisfies the following conditions: stride=(Nsubgroup×Widbank)*2 / Ad*N+size=size*2N+size, where N is a natural number.

[0181] As described above, the illustrated SRAM can also support simultaneous access to 8 32B data, 4 64B data, 2 128B data, or 1 256B data. Based on different access modes and according to the above principles, constraints on the memory access address can be given accordingly.

[0182] For example, for each access to 1 256B of data, it is obvious that only the 256B of data needs to be stored continuously in the RAM. At this time, stride=0, and there is only one Addr0.

[0183] For each access to 2 128B data, stride=256B*N+128B is required, where N is a natural number.

[0184] For each access to 4 64B of data, stride=128B*N+64B is required, where N is a natural number.

[0185] For each access to 8 32B data, stride=64B*N+32B is required, where N is a natural number.

[0186] For each access to 16 16B data, stride=32B*N+16B is required, where N is a natural number.

[0187] From the memory access process described above in conjunction with Figures 8 to 10, it can be seen that the limitation of this address relationship will be reflected in the storage stride of the C dimension and W dimension of the data. Specifically, when the Ad discrete addresses accessed at one time point to the first address of the Ad data with the same index in the W dimension and adjacent index in the H dimension in the data storage circuit, the Ci dimension stride ci_stride and the W dimension stride wi_stride meet the following conditions: ci_stride×wi_stride=stride.

[0188] Table 3 below lists the relevant constraints on the dimensional strides of neurons and weights in the C dimension and W dimension on the storage circuit RAM in Forward1 mode. It is assumed here that the bit widths of neurons and weights on RAM are the same. It should be noted that although the constraint on the dimensional stride kw_stride on the W dimension of the weight is given here, in fact, since two read ports are used to read the weight data, each read port only reads one address, so the constraint on kw_stride can actually be canceled. It is listed here only to unify the requirements for software. Table 3 Storage dimension step size constraints of neurons and weight data in Forward1 mode

[0189] As described above, the output neuron data also satisfies storage constraints similar to those of the input neuron data, so that it can be used as the input neuron data of the next convolutional layer. Considering that the data types of input and output on the storage circuit RAM may be different, the co_stride of the output neuron is determined by the ci_stride of the input neuron and the bit width ratio of the data type of the input and output neurons on the RAM. Table 4 shows the co_stride constraints of the output neuron in Forward1 mode. Table 4 Co_stride constraints of output neurons in Forward1 mode

[0190] The constraint of wo_stride of the output neuron is determined by co_stride. Table 5 shows the wo_stride constraint of the output neuron in Forward1 mode. co_stride size wo_stride size co_stride = 16B 8Q+4(Q≥0) co_stride = 32B 4Q+2(Q≥0) co_stride = 128B*M+64B(M≥0) 2Q+1(Q≥0) Table 5 wo_stride constraints of output neurons in Forward1 mode

[0191] In Forward4 mode, since the convolution operation will be accumulated in the ci direction, in order to avoid the interaction between the processing circuit groups and increase the complexity of the circuit, ci_stride can be limited to 16B, that is, only one case is processed. At this time, the constraints of the step size of each dimension are as follows: ci_stride = syn_ci_stride = 16B, wi_stride = 8N+4 (N≥0), kw_stride = 8M+4 (M≥0). There is no step size constraint on the kh and co dimensions of the weights, so they can be calculated by default as follows: syn_kh_stride=ram_input_synapse_data_width*syn_ci_stride*kw_stride syn_co_stride=ram_input_synapse_data_width*syn_ci_stride*kw_stride*kh The co_stride and wo_stride constraints of the output neurons are the same as in Table 5.

[0192] Thus, the above describes the process of how to retrieve discretely stored data from the storage circuit RAM according to certain rules. The retrieved data is transmitted to the connected processing circuit group through the corresponding data transmission path. At the processing circuit group, the data to be processed can be obtained and cached in the corresponding buffer in the desired form for the processing circuit to retrieve and calculate. Exemplary Hardware Partitioning Scheme

[0193] Fig.13 The schematic diagram of the hardware block scheme for processing neuron data in the processing circuit group is shown. The figure shows the first data transmission ports 1310 of the four processing circuit groups PPB, namely PPB0_port, PPB1_port, PPB2_port and PPB3_port. The shared cache in each processing circuit group (e.g., reference Figure 6a The shared buffer 622 in the embodiment of the present invention is divided into two buffer areas, which are respectively called the first ping buffer area (ping buffer) 1320 and the first pong buffer area (pong buffer) 1330. Generally, the storage space size of the ping buffer area is smaller than the storage space size of the pong buffer area. For example, the size of the ping buffer area is 512B, and the size of the pong buffer area is 12×12×16B=2304B as shown in the figure. The ping buffer area and the pong buffer area can be separate buffers or different areas of the same buffer. For simplicity, the pong buffer area only shows PPB0 pongbuffer.

[0194] According to the data transmission mechanism described above, neuron data is transmitted through the first data transmission path. In one embodiment, 512B of data is read through two first read ports in one cycle (one beat), where each first read port reads 256B of data. The 512B of data is divided into 4 data blocks and allocated to 4 first data channels to be transmitted to the corresponding 4 processing circuit groups. Each first data channel transmits 128B of data. The allocation of data read by the read port to the first data channel can be in various forms. Preferably, continuous data blocks are allocated to the same data channel as much as possible. For example, refer to Figure 8aIn the embodiment, the first row of data read through the two first read ports RS0 and RS1 (that is, two 1×4×16B data blocks are spliced ​​into a 1×8×16B data block) can be assigned to the data channel corresponding to PPB0, and the second row of data read through the two first read ports RS0 and RS1 can be assigned to the data channel corresponding to PPB1, and so on. Thus, each port of PPB receives a 1×8×16B data block, corresponding to hi=0 to 3 respectively.

[0195] The data blocks received by each PPB may not be what it needs, and it is necessary to exchange data through the data exchange channel between PPBs. In some embodiments, there is a full connection between the first data transmission port of each PPB and the first ping-pong buffer area of ​​each PPB, that is, the first data transmission port of each PPB is connected to the first ping-pong buffer area of ​​each PPB, and the first ping-pong buffer area of ​​each PPB can receive data from the first data transmission port of each PPB. This fully connected path between the first data transmission port of the PPB and the first ping-pong buffer area of ​​the PPB constitutes a data exchange channel between PPBs. In some embodiments, the connection between the first data transmission port and the first ping-pong buffer area can be one connection per 16B. As previously referenced Figure 6a As described, the four PPBs are located separately from each other at different positions on the circuit board, for example, around the storage circuit, so as to fully fill the space on the circuit board and improve the area utilization. Therefore, the first data transmission ports of the four PPBs are also located on the outer layer of the chip. It is easier to route the wires to achieve the above-mentioned full connection on the outer layer of the chip than to achieve it inside the chip.

[0196] For example, reference Figure 8b In the embodiment, it can be seen from the mark in 850 that the two 256B data read in the first beat are both to be supplied to PPB0. Therefore, the 1×8×16B data blocks of this beat received by the first data transmission ports of the four PPBs respectively need to be transmitted to the first ping buffer of PPB0. As shown by the arrows in the figure, the first data transmission ports of the four PPBs all transmit the current data to the first ping buffer of PPB0. Similarly, in the subsequent processing cycles, after the data read in each beat arrives at the first data transmission ports of the four PPBs, it is transmitted to the first ping buffer of the corresponding PPB according to the mark. It can be seen from the above process that for data that will be reused between PPBs, for example Figure 8bThe data blocks marked as 1100, 1111, 0011, 1010, and 0101 in 850 only need to be read from the storage circuit once and can be transmitted to multiple PPBs, thereby saving the number of memory accesses. The data in the first ping buffer area can be stored in sequence according to the order of reception. For example, the delays of each first data transmission port to the ping buffer area are different, so the data of each port will be received in sequence.

[0197] Next, the data cached in the first ping buffer can be transmitted to the first pong buffer of the corresponding PPB. The figure shows the transmission from PPB0 ping buffer to PPB0 pong buffer. Generally speaking, the size of the first pong buffer of a PPB needs to be able to accommodate the input neuron data required by all processing circuits in the PPB in a single round of calculation. With reference to the embodiments described in conjunction with Figures 8-10 above, it can be seen that in various embodiments, in order to maintain consistent computational behavior of the processing circuits, the shape of the neuron data allocated to each PPB is 12×12×16B (HiWiC), so the size of the first pong buffer can be set to 12×12×16B.

[0198] In the process of transferring data from the first ping-pong buffer to the first pong buffer, the data can still be placed in the order of storage on the RAM, that is, the 12×12×16B data required by each PPB is still placed in the order of HWC, C is the lowest dimension, and the size is 16B. In this way, the 12×12×16B data required by each PPB can be transferred to the first pong buffer of the shared cache of each PPB.

[0199] Since the rules for fetching data from the storage circuit RAM (including fetching order, fetching size, etc.) are determined in various modes (different convolution modes, or different ci_stride in the same convolution mode), the transmission delay from each port to the first ping-pong buffer area is also determined, so the storage position of the data received in the first ping-pong buffer area in the first pong buffer area can also be determined. Based on this position correspondence, the data cached in the first ping-pong buffer area can be moved to the corresponding position of the first pong buffer area to facilitate subsequent further block operations. Those skilled in the art can derive the data and corresponding positions of the first ping-pong buffer area and the first pong buffer area in each processing cycle according to the above rules, which will not be described in detail here.

[0200] like Fig.13 As shown, the data in the first pong buffer area is further divided into blocks by the block circuit 1340 and then transmitted to each processing circuit in the processing circuit group. According to the periodic processing of the convolution operation task in a single processing circuit group and the task allocation among multiple processing circuits in each cycle (for example, refer to the previous Figure 6b), for the data output from the first pong buffer area in each cycle, the blocking circuit 1340 arranges the data according to the data blocking rule, adjusts the order of the data, and then transmits it to the operation circuit.

[0201] In some implementations, the blocking circuit 1340 may include a plurality of first read control ports 1341, each of which is used to filter data to be transmitted to a corresponding processing circuit. Figure 6b 670, which shows the distribution of data 8×8×8B (HWC) output from the first pong buffer area among four processing circuits PP in one cycle, each PP can fetch 5×5×8B neurons from a fixed position for calculation through the corresponding first read control port. Figure 6a As can be seen from the internal structure diagram of the computing device, the data output from the shared cache enters the PP and is directly sent to the corresponding computing circuit CU. Therefore, the first read control port can further filter out the data distributed to each CU, and the retrieved data can be adjusted in order according to the data block placement rules and then transmitted to the CU.

[0202] In some implementations, the blocking circuit 1340 may further include a plurality of first mapping circuits 1342 for reordering the data filtered out from the corresponding first read control port according to the data block placement rule. Specifically, the first mapping circuit may adjust the order of the data according to the mapping rule between the index of each data point in the received data (e.g., bit address) and the index of the data point after the block arrangement according to the data block requirement according to the data bit width information, and then supply the data to the subsequent operation circuit 1350. For example, assuming that the data type is FP32, the data bit width is 32 bits, and one data point contains 32 bits, that is, 4B, and the data to be supplied to the operation circuit from the first pong buffer area is 4×4×2 (HWC) data points. According to the mapping rule, the first data point is still the first data point after the block arrangement, so the first 32 bits of data received are the first data point after the block arrangement. Since the block arrangement requires the data block size to be 4×4×4B (HWC), and the blocks are arranged in the order of CHW (W is the low dimension), the second data point after the block arrangement should be the second point on the W dimension. Therefore, according to the data bit width, the second point on the W dimension can be calculated to correspond to the 64th to 95th bits of the received data, that is, one data point will be skipped. Similarly, the order of these 32 data points can be rearranged according to the data block arrangement rules.

[0203] Therefore, the data to be calculated received by the calculation circuit is in units of data blocks, and calculations can be performed based on this to fully utilize the calculation power.

[0204] The hardware blocking process of weight data is similar to that of neurons. The difference is that since the amount of weight data is usually not large, the weight data is directly copied when it is transmitted through the second data transmission path. In this way, each processing circuit group can receive all the weight data, avoiding the need for data interaction between processing circuit groups, thereby saving hardware connection design and on-chip wiring space.

[0205] review Figure 6a , the transmission route of the weight data is to read it from the storage circuit 610 and directly cache it in the local cache 632 in each parallel processing circuit PP, and then distribute it to each computing circuit CU therein. Similar to the hardware blocking scheme of neuron data, the local cache 632 used to cache weight data is also divided into two cache areas: a second ping-pong cache area and a second pong cache area. The second ping-pong cache area is connected to the second data transmission port of the processing circuit group to which it belongs. The second ping-pong cache area is used to select and cache the weight data part assigned to its own parallel processing circuit PP from the weight data transmitted from the second data transmission port of the processing circuit group. For example, for the mode of ci_stride=16B, the weight data required by each PPB and the PP therein is the same, which is the entire weight data, so it is sufficient to receive all the weight data. For another example, for the mode of ci_stride=32B, PPB0 and PPB1 (and the PPs within them) only need to take the lower 16B of data for every 32B of data on the Ci dimension, while PPB2 and PPB3 (and the PPs within them) only need to take the upper 16B of data for every 32B of data on the Ci dimension, so only half of the weight data needs to be received. For another example, for the mode of ci_stride=64B, each PPB only needs to take its own 16B of data for every 64B of data on the Ci dimension, that is, only one quarter of the weight data needs to be received. The subsequent processing of the weight data is similar to that of the neuron data. For example, the second pong buffer receives the weight data transmitted from the second ping buffer, and caches the weight data in the order of the first dimension.

[0206] Since the weight data is multiplexed in each processing circuit group, the block processing of the weight data only needs to be performed once in each processing circuit group. For example, for the weight data, the above-mentioned block circuit may include a second read control port and a second mapping circuit. The second read control port is used to filter out the weight data to be transmitted to each operation circuit in the processing circuit from the second pong buffer area. The second mapping circuit is used to split the weight data filtered out from the second read control port into data blocks of fixed size, align them, adjust the order, and then transmit them to each operation circuit. The specific process is similar to the previous processing of neuron data, and will not be expanded in detail here.

[0207] Therefore, on the data transmission path, it is achieved through hardware that the neuron data and weight data are adjusted and arranged in order according to the pendulum number requirements of the computing components. This pendulum number process can be hidden in the entire calculation pipeline, thereby greatly improving the computing efficiency.

[0208] The inverse tiling process of the output data is similar to the tiling process, but it is done in reverse. Figure 6b The described computing task allocation and generation of output data are also generated one by one in blocks. Therefore, when storing the output data back into the storage circuit, the data order needs to be adjusted accordingly to facilitate writing back to the storage circuit. The write-back process is similar to the read-out process, and a single write port can support writing to multiple discrete addresses at the same time. For example, in the disclosed embodiment, based on the storage circuit structure described above, a single write port supports writing to at least four discrete addresses at the same time.

[0209] Fig.14 An exemplary flow chart of a data transmission method of a data transmission device according to an embodiment of the present disclosure is shown.

[0210] As shown in the figure, in step 1410, a plurality of discrete addresses are accessed from a storage circuit via a plurality of read ports to read data, wherein the data is sequentially stored in the storage circuit according to a first dimension.

[0211] In step 1420 , the data read via the read port is distributed and transmitted to multiple processing circuit groups through multiple data channels.

[0212] In step 1430 , data transmitted via a data channel and to be processed by the processing circuit group is received and buffered at a buffer within the processing circuit group.

[0213] Finally, in step 1440, the data from the cache is split into data blocks of fixed size, aligned and sequenced, and then transmitted to the operation circuit in the processing circuit group, wherein the data blocks are arranged in sequence according to the second dimension, and the data blocks are stored in sequence according to the first dimension.

[0214] In some embodiments, the data includes first data, such as neuron data in a convolution operation. The read port includes a plurality of first read ports, and the data transmission method includes: reading the first data via the first read port, wherein each first read port reads Ad discrete addresses at a time, Ad>1.

[0215] In some implementations, the order of the first dimension is HWCi, where H is the height dimension, W is the width dimension, Ci is the input channel dimension, Ci is the lowest dimension, and the Ci dimension has a dimension stride ci_stride, and the W dimension has a dimension stride wi_stride.

[0216] In some embodiments, the above-mentioned Ad discrete addresses point to the first addresses of Ad data with the same index in the W dimension and adjacent index in the H dimension of the first data, and the Ci dimension step ci_stride and the W dimension step wi_stride satisfy the following conditions: ci_stride×wi_stride=size*2N+size, N is a natural number, and size is the amount of continuous data read by each discrete address.

[0217] In some implementations, Ad=4, size=64B, B represents bytes, ci_stride is selected from 16B, 32B, and 128B*M+64B, and M is a natural number. The number of the first read ports is 2.

[0218] In some embodiments, the data includes second data, such as weight data in a convolution operation. The read port includes a plurality of second read ports, and the data transmission method includes: reading the second data via the second read ports, wherein each second read port reads one address at a time.

[0219] In some embodiments, the method further comprises: duplicating a plurality of copies of the second data read via the second read port, so as to transmit a copy of the second data to each processing circuit group.

[0220] The specific implementation of steps 1430 and 1440 can refer to the above combined Fig.13 The description is not repeated here.

[0221] Those skilled in the art will appreciate that the various features of the computing device and the data transmission device described above in conjunction with the accompanying drawings can be similarly applied to Fig.14 The data transmission method is described in detail above and therefore will not be repeated here.

[0222] Those skilled in the art can understand that although the above solution is proposed based on the small convolution optimization operation problem, the above solution is not limited thereto, but can be applied to any other scenarios with similar data transmission requirements, such as matrix operations, etc. In addition, although the above solution is a solution proposed for artificial intelligence processors, it can also be applied to general or special processors such as CPUs and GPUs, and the embodiments of the present disclosure are not limited in this regard.

[0223] The present disclosure also provides a processor, including the aforementioned data transmission device. The present disclosure also provides a chip, which may include the processor of any embodiment described above in conjunction with the accompanying drawings. Further, the present disclosure also provides a board, which may include the aforementioned chip.

[0224] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0225] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0226] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them on the basis of considering the logical functions, and there may be other splitting methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0227] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0228] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0229] The embodiments of the present disclosure are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A data transmission device, comprising: A storage circuit, used for storing data, wherein the data is stored sequentially according to a first dimension; a plurality of read ports for accessing a plurality of discrete addresses from the storage circuit to read the data; a plurality of data channels for distributing and transmitting data read via the read port to a plurality of processing circuit groups; a cache located in the processing circuit group, for receiving and caching data transmitted via the data channel and to be processed by the processing circuit group; as well as The block division circuit is used to divide the data from the cache into data blocks of fixed size, align and adjust the order, and then transmit them to the operation circuit in the processing circuit group, wherein the data blocks are arranged in order according to the second dimension, and the data blocks are arranged in order according to the first dimension.

2. The data transmission device according to claim 1, wherein the data comprises first data, and the read port comprises a plurality of first read ports for reading the first data, wherein each first read port supports reading Ad discrete addresses at a time, Ad>1.

3. The data transmission device according to claim 2, wherein the storage circuit comprises a plurality of memory banks, the plurality of banks being divided into N subgroup subgroups, and only one bank in each subgroup can be accessed at a time, where N subgroup ≥Ad, to support a single first read port to read Ad discrete addresses at a time.

4. The data transmission device according to claim 3, wherein the bit width of each bank is Wid bank , to support up to N simultaneous accesses subgroup Wid bank When a single first read port reads Ad discrete addresses at a time, and the amount of continuous data read from each discrete address is size, N subgroup ×Wid bank ≥Ad×size, the Ad discrete addresses satisfy the following conditions: Each discrete address is divided into N subgroup ×Wid bank The integer part and remainder part of the discrete address are at least size apart and distributed in the N subgroup On a subgroup.

5. The data transmission device according to claim 4, wherein N subgroup ×Wid bank =Ad×size, the step length stride between the Ad discrete addresses satisfies the following conditions: stride=(N subgroup ×Wid bank )*2 / Ad*N+size, where N is a natural number.

6. The data transmission device according to claim 5, wherein the first dimension order is HWCi, H is the height dimension, W is the width dimension, Ci is the input channel dimension, Ci is the lowest dimension, and there is a dimension step ci_stride on the Ci dimension, The W dimension has a dimension stride of wi_stride.

7. The data transmission device according to claim 6, wherein the Ad discrete addresses point to the first addresses of Ad data with the same index in the W dimension and adjacent index in the H dimension of the first data, and the Ci dimension step ci_stride and the W dimension step wi_stride satisfy the following conditions: ci_stride×wi_stride=stride.

8. The data transmission device according to claim 7, wherein Ad=4, size=64B, B represents bytes, ci_stride is selected from 16B, 32B and 128B*M+64B, and M is a natural number.

9. The data transmission device according to any one of claims 2 to 8, wherein the number of the first read ports is 2.

10. The data transmission device according to any one of claims 2 to 9, wherein the cache comprises a shared cache located in each processing circuit group, for receiving and caching first data read via the first read port and transmitted via the data channel and to be used by all processing circuits in the processing circuit group.

11. The data transmission device according to claim 10, wherein the shared cache comprises a first ping-pong buffer area, the first ping-pong buffer area being connected to the first data transmission port of each of the processing circuit groups to receive therefrom the first data which is transmitted to the first data transmission port of each processing circuit group via the multiple data channels and is to be used by all processing circuits within the processing circuit group. 12 . The data transmission device according to claim 11 , wherein the shared cache further comprises a first pong buffer area, the first pong buffer area being used to receive the first data transmitted from the first ping-pong buffer area and cache the first data in order according to the first dimension.

13. The data transmission device according to claim 12, wherein the blocking circuit comprises a plurality of first read control ports and a plurality of first mapping circuits, wherein: Each of the first read control ports is used to filter out first data to be distributed to each computing circuit in each processing circuit from the first pong buffer area; Each first mapping circuit is used for splitting the first data screened out from the corresponding first read control port into data blocks according to the fixed size, aligning the data blocks, adjusting the order, and transmitting the data blocks to the corresponding operation circuit.

14. The data transmission device according to claim 13, wherein the first mapping circuit is configured to adjust the order of each data point in the first data according to data bit width information and a mapping rule between the index of each data point in the filtered first data and the index of the data point after being aligned and adjusted in order by splitting the data blocks according to the fixed size.

15. The data transmission device according to any one of claims 1-14, wherein the second dimensional order is CiHW, Ci is the input channel dimension, H is the height dimension, W is the width dimension, and W is the lowest dimension.

16. The data transmission device according to any one of claims 1-15, wherein the fixed size of the data block is H×W×Ci=4×4×4B, where B represents bytes.

17. The data transmission device according to any one of claims 1-16, wherein the data comprises second data, and the read port comprises a plurality of second read ports for reading the second data, wherein each second read port reads one address at a time.

18. The data transmission device according to claim 17, wherein the plurality of data channels are further used to: replicate the second data read via the second read port into a plurality of copies, so as to transmit a copy of the second data to each of the processing circuit groups. The data transmission device according to claim 18 , wherein the number of the second read ports is 2.

20. The data transmission device according to any one of claims 17 to 19, wherein the cache comprises a local cache located in each processing circuit, for receiving and caching second data read via the second read port and transmitted via the data channel to be used by the respective processing circuit.

21. The data transmission device according to claim 20, wherein the local cache comprises a second ping-pong cache area, and the second ping-pong cache area is connected to the second data transmission port of the processing circuit group where it is located, so as to receive second data from it, which is transmitted to the second data transmission port of each processing circuit group via the multiple data channels and is to be used by the processing circuit where it is located.

22. The data transmission device according to claim 21, wherein the local cache further comprises a second pong buffer area, the second pong buffer area is used to receive the second data transmitted from the second ping-pong buffer area, and cache the second data in order according to the first dimension.

23. The data transmission device according to claim 22, wherein the blocking circuit comprises a second read control port and a second mapping circuit, wherein: The second read control port is used to filter out second data to be transmitted to each operation circuit in the processing circuit from the second pong buffer area; The second mapping circuit is used for splitting the second data screened from the second read control port into data blocks of the fixed size, aligning the data blocks, adjusting the order, and transmitting the data blocks to each operation circuit.

24. A processor, comprising the data transmission device according to any one of claims 1-23.

25. A chip comprising the processor according to claim 24.

26. A board comprising the chip according to claim 25.

27. A data transmission method, comprising: Accessing a plurality of discrete addresses from a storage circuit via a plurality of read ports to read data, wherein the data is sequentially stored in the storage circuit according to a first dimension; Allocating and transmitting the data read via the read port to multiple processing circuit groups through multiple data channels; receiving and buffering data transmitted via the data channel and to be processed by the processing circuit group at a buffer within the processing circuit group; as well as The data from the cache is split into data blocks of fixed size, aligned and sequenced, and then transmitted to the operation circuit in the processing circuit group, wherein the data blocks are arranged in sequence according to the second dimension, and the data blocks are arranged in sequence according to the first dimension.

28. The data transmission method according to claim 27, wherein the data comprises first data, the read port comprises a plurality of first read ports, and the method comprises: The first data is read via the first read port, wherein each first read port reads Ad discrete addresses at a time, Ad>1.

29. The data transmission method according to claim 28, wherein the first dimensional order is HWCi, H is the height dimension, W is the width dimension, Ci is the input channel dimension, Ci is the lowest dimension, and there is a dimensional step ci_stride on the Ci dimension, The W dimension has a dimension stride of wi_stride.

30. The data transmission method according to claim 29, wherein the Ad discrete addresses point to the first addresses of Ad data with the same index in the W dimension and adjacent index in the H dimension of the first data, and the Ci dimension step ci_stride and the W dimension step wi_stride satisfy the following conditions: ci_stride×wi_stride=size*2N+size, where N is a natural number and size is the amount of continuous data read from each discrete address.

31. The data transmission method according to claim 30, wherein Ad=4, size=64B, B represents bytes, ci_stride is selected from 16B, 32B and 128B*M+64B, and M is a natural number.

32. The data transmission method according to any one of claims 28 to 31, wherein the number of the first read ports is 2.

33. The data transmission method according to any one of claims 28 to 32, wherein the cache comprises a shared cache located in each processing circuit group, and the method comprises: First data read via the first read port and transmitted via the data channel to be used by all processing circuits within the processing circuit group is received and cached at the shared cache.

34. The data transmission method according to claim 33, wherein the shared cache comprises a first ping-pong cache area, the first ping-pong cache area is connected to a first data transmission port of each of the processing circuit groups, the method comprising: First data to be used by all processing circuits in the processing circuit group, which is transmitted to the first data transmission port of each processing circuit group via the plurality of data channels, is received at the first ping buffer area.

35. The data transmission method according to claim 34, wherein the shared cache further comprises a first pong cache area, the method comprising: First data transmitted from the first ping buffer area is received at the first pong buffer area, and the first data is sequentially buffered according to the first dimension.

36. The data transmission method according to claim 35, further comprising: Filtering out first data to be distributed to each computing circuit in each processing circuit from the first pong buffer area; as well as The filtered first data is split into data blocks according to the fixed size, aligned and adjusted in order, and then transmitted to the corresponding operation circuit.

37. The data transmission method according to claim 36, further comprising: According to the data bit width information, the order of each data point in the first data is adjusted according to the mapping rule between the index of each data point in the filtered first data and the index of the data point after being split into data blocks according to the fixed size and aligned and adjusted in order.

38. The data transmission method according to any one of claims 27-37, wherein the second dimensional order is CiHW, Ci is the input channel dimension, H is the height dimension, W is the width dimension, and W is the lowest dimension.

39. The data transmission method according to any one of claims 27-38, wherein the fixed size of the data block is H×W×Ci=4×4×4B, where B represents bytes.

40. The data transmission method according to any one of claims 27 to 39, wherein the data comprises second data, the read port comprises a plurality of second read ports, and the method comprises: The second data is read via the second read ports, wherein each second read port reads one address at a time.

41. The data transmission method according to claim 40, further comprising: The second data read through the second read port is copied into a plurality of copies, so as to transmit one copy of the second data to each of the processing circuit groups.

42. The data transmission method according to claim 41, wherein the number of the second read ports is 2.

43. The data transmission method according to any one of claims 40 to 42, wherein the cache comprises a local cache located in each processing circuit, and the method comprises: Second data to be used by respective processing circuits, which is read via the second read port and transmitted via the data channel, is received and buffered at the local buffer.

44. The data transmission method according to claim 43, wherein the local cache comprises a second ping-pong cache area, the second ping-pong cache area is connected to the second data transmission port of the processing circuit group where the second ping-pong cache area is located, the method comprising: The second ping-pong buffer area receives second data to be used by the processing circuits therein, which are transmitted to the second data transmission ports of each processing circuit group via the plurality of data channels.

45. The data transmission method according to claim 44, wherein the local cache further comprises a second pong cache area, the method comprising: Second data transmitted from the second ping buffer area is received at the second pong buffer area, and the second data is cached in sequence according to the first dimension.

46. ​​The data transmission method according to claim 45, further comprising: Filtering out second data to be transmitted to each operation circuit in the processing circuit from the second pong buffer area; as well as The filtered second data is split into data blocks according to the fixed size, aligned and adjusted in order, and then transmitted to each operation circuit.

Images