Apparatus and method for accelerating data migration

Abstract

The present disclosure relates to an artificial intelligence accelerator for accelerating data movement, an integrated circuit device, a board, a method, a readable storage medium, a computer program product, and a computer device, wherein the computer device of the present disclosure is included in the integrated circuit device, which includes a general-purpose interconnection interface and other processing devices. The computer device and the other processing devices interact together to complete a user-specified computing operation. The integrated circuit device can also include a storage device connected to the computer device and the other processing devices, respectively, for data storage of the computer device and the other processing devices.

Description

Technical Field

[0001] This disclosure generally relates to the computer field. More specifically, this disclosure relates to artificial intelligence accelerators, integrated circuit devices, boards, methods, readable storage media, computer program products, and computer devices that accelerate the movement of data. Background Technology

[0002] Convolution is a general and fundamental operation in neural networks, playing an irreplaceable and crucial role in current convolutional neural networks. However, convolution operations are computationally intensive, placing high demands on the hardware performance of artificial intelligence accelerators.

[0003] Generally, AI accelerators are designed with specific hardware dedicated to convolution tasks. During convolution operations, the data to be processed needs to be computed, resulting in a large amount of data being repeatedly moved, especially when the stride is greater than 1. This is because the data is fragmented due to skipping steps, and the small amount of data moved leads to a significant drop in bandwidth. Therefore, improving data movement bandwidth is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In order to address at least one or more of the technical problems mentioned above, this disclosure provides, in several aspects, artificial intelligence accelerators, integrated circuit devices, boards, methods, readable storage media, computer program products, and computer devices for accelerating data transfer.

[0005] In a first aspect, this disclosure provides an artificial intelligence accelerator for reassembling input data to accelerate data transfer, comprising: a processing module, a temporary storage module, a storage module, and a computing module. The processing module is used to sample multiple data blocks from the input data at intervals decreasing by one step size when the step size is greater than 1 and larger than the original data transfer size, and to reassemble the input data, wherein the multiple data blocks are arranged adjacently to form a first data group; the temporary storage module is used to store the reassembled input data; the storage module prioritizes transferring and storing the first data group from the temporary storage module according to the transfer size of the reassembled input data; the computing module obtains the first data group from the storage module and performs calculations.

[0006] In a second aspect, this disclosure provides an artificial intelligence accelerator, comprising: a processing module, a storage module, and a computation module. The processing module is used to: sample multiple data blocks from the input data at intervals decreasing by one step size when the step size is greater than 1 and larger than the original data size being transported; reassemble the input data, wherein the multiple data blocks are arranged adjacently to form a data cluster; and perform im2col expansion on the reassembled input data. The storage module moves and stores the data cluster according to the order after im2col expansion, based on the transported data size of the reassembled input data. The computation module obtains the data cluster from the storage module and performs matrix multiplication operations.

[0007] In a third aspect, this disclosure provides an integrated circuit device including the aforementioned artificial intelligence accelerator, and a board including the aforementioned integrated circuit device.

[0008] In the fourth aspect, this disclosure provides a method for reassembling input data to accelerate data transfer, comprising: sampling multiple data blocks in the input data at intervals with a step size greater than 1 and larger than the original data transfer size; reassembling the input data, wherein the multiple data blocks are arranged adjacently to form a first data group; prioritizing the transfer of the first data group according to the transfer size of the reassembly input data; and performing calculations on the transferred first data group.

[0009] In a fifth aspect, this disclosure provides a computer-readable storage medium having stored thereon computer program code for processing input data in an artificial intelligence accelerator, wherein the above-described method is executed when the computer program code is run by a processing device.

[0010] In a sixth aspect, this disclosure provides a computer program product including a computer program that processes input data in an artificial intelligence accelerator, the computer program implementing the steps of the above-described method when executed by a processor.

[0011] In a seventh aspect, this disclosure provides a computer apparatus including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0012] Since the size of the reconstructed input data is larger than the original size, the data is moved based on the size of the reconstructed input data, thereby increasing the data transfer bandwidth and achieving the technical effect of accelerating the transfer and effectively utilizing the input and output bandwidth. Attached Figure Description

[0013] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0014] Figure 1 This is a structural diagram of the board card shown in the embodiment of this disclosure;

[0015] Figure 2 This is a structural diagram illustrating an integrated circuit device according to an embodiment of this disclosure;

[0016] Figure 3 This is a schematic diagram illustrating the internal structure of a computing device according to an embodiment of the present disclosure;

[0017] Figure 4This is a schematic diagram illustrating the internal structure of the processor core in an embodiment of this disclosure;

[0018] Figure 5 This is a diagram illustrating when one processor core wants to write data to another processor core in a different cluster;

[0019] Figure 6 and Figure 7 This shows a schematic diagram of the im2col operation;

[0020] Figure 8 This is a schematic diagram illustrating the execution of im2col on input data of dimension NCHW;

[0021] Figure 9 This is a schematic diagram illustrating the input data of an embodiment of this disclosure;

[0022] Figure 10 This is a schematic diagram illustrating the im2col expansion of the recombined input data in the embodiments disclosed herein;

[0023] Figure 11 This is a flowchart illustrating another embodiment of the present disclosure of implementing matrix multiplication;

[0024] Figure 12 This is a schematic diagram illustrating the reorganization of neuronal data according to another embodiment of this disclosure. Detailed Implementation

[0025] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] It should be understood that the terms “comprising” and “including” used in this disclosure and claims indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0028] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0029] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0030] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0031] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0032] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0033] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown...) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0034] The computing device 201 is an artificial intelligence accelerator configured to perform user-specified operations. It is mainly implemented as a single-core or multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0035] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0036] Processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0037] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0038] Figure 3A schematic diagram of the internal structure of computing device 201 is shown. Computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. As a system-on-a-chip, computing device 201 includes multiple computing clusters (computing clusters 305), and each computing cluster includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchical structure of system-on-a-chip, computing clusters, and processor cores.

[0039] From the perspective of system-on-a-chip hierarchy, such as Figure 3 As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple computing clusters 305.

[0040] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to respond to access requests from the processor core to access external storage devices, such as… Figure 2 The DRAM 204 in the chip allows the computing device 201 to read data from external sources or write data to external sources. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external memory controller 301, the peripheral communication module 302, and multiple computing clusters 305 to transmit data and control signals between modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each computing cluster and ensure information synchronization. The multiple computing clusters 305 are the computing core of the computing device 201. Four are shown exemplary in the figure; however, with hardware development, the computing device 201 disclosed herein may also include 8, 16, 64, or even more computing clusters 305. The computing clusters 305 are used to efficiently execute deep learning algorithms.

[0041] From the perspective of the hierarchy of computational clusters, such as Figure 3 As shown, each computing cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0042] Four processor cores 306 are shown in the figure as an example; this disclosure does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0043] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0044] The computation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0045] Storage module 43 is used to store or move related data, including neuron RAM (NRAM) 431, weight RAM (WRAM) 432, input / output direct memory access (IODMA) 433, and move direct memory access (MVDMA) 434. NRAM 431 stores input, output data, and intermediate results for computation by processor core 306; WRAM 432 stores the weights of the deep learning network; IODMA 433 controls memory access between NRAM 431 / WRAM 432 and DRAM 204 via broadcast bus 309; MVDMA 434 controls memory access between NRAM 431 / WRAM 432 and SRAM 308. It should be noted that NRAM and WRAM here can be two storage regions formed by dividing the same memory in logical storage space, or they can be two independent memories; no specific limitation is made here.

[0046] Back Figure 3 The storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between computing clusters 305 and DRAM 204, communication between computing clusters 305, and communication between processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0047] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a compute cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same compute cluster 305 does not need to be obtained from DRAM 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip input / output access.

[0048] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between computing clusters 305, and data transfer between computing clusters 305 and DRAM 204, respectively. These will be explained separately below.

[0049] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the computing cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0050] CDMA 310 is used to control access to SRAM 308 between different computing clusters 305 within the same computing device 201. Figure 5 This diagram illustrates the operation of CDMA 310 when one processor core attempts to write data to another processor core in a different computing cluster. In this application scenario, the same computing device includes multiple computing clusters. For simplicity, only computing cluster 0 and computing cluster 1 are shown in the diagram. Both clusters include multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown for computing cluster 0, and only processor core 1 is shown for computing cluster 1. Processor core 0 intends to write data to processor core 1.

[0051] First, processor core 0 sends a unicast write request to write data into local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transfer the data to SRAM 1 of computing cluster 1. Then, the slave sends a write response B as a response. Finally, processor core 1 of computing cluster 1 sends a unicast read request to read the data from SRAM 1.

[0052] Back Figure 3 The GDMA 311, in conjunction with the external memory controller 301, controls memory access from the SRAM 308 of the computing cluster 305 to the DRAM 204, or reads data from the DRAM 204 into the SRAM 308. As described above, communication between the DRAM 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between the DRAM 204 and the NRAM 431 or WRAM 432 via the IODAM 433; the second channel involves first transmitting data between the DRAM 204 and SRAM 308 via the GDMA 311, and then transmitting data between the SRAM 308 and the NRAM 431 or WRAM 432 via the MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between the DRAM 204 and the NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments disclosed herein allow for the selection of data transmission channels based on the hardware capabilities available.

[0053] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, this disclosure treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure.

[0054] The computing device 201 is often the computational bottleneck of the entire neural network operation when processing convolution operations, especially large-scale convolution operations. Therefore, it is crucial to make full use of the hardware's computing performance to accelerate convolution operations.

[0055] When performing convolution operations using matrix multiplication, the computing device 201 frequently uses `im2col` to expand the input data (neuronal data) from computer vision, speech, natural language processing, and data mining, transforming high-dimensional data into two-dimensional data for computation. For example, this is done when updating the convolution kernel during backpropagation in neural network training. Backpropagation for kernel updates (convolution backward filter) is generally represented as `convbpfilter`, where the output is the gradient of the convolutional layer's weights. When performing `convbpfilter`, the computing device 201 typically uses the `im2col` operation to expand the gradients of the input and output neurons into two matrices, then uses matrix multiplication to accelerate computation.

[0056] The purpose of im2col is to optimize convolution operations. For each small window that the convolution kernel needs to process, it is expanded into a row (column) of a new matrix. The number of columns (rows) of the new matrix is ​​the number of times the convolution operation is performed on the input data, which is the number of times the convolution kernel slides.

[0057] Figure 6 and Figure 7 The diagram illustrates the operation of im2col, using a 3×3 kernel, 4×4 input data (A1 to A16), and a stride of 1 as an example. Figure 6 As shown, since the convolution kernel size is 3×3, during im2col, the 3×3 matrix in the upper left corner of the input data is first expanded to form the first column 601, including A1, A2, A3, A5, A6, A7, A9, A10, and A11; then the 3×3 matrix in the upper right corner of the input data is expanded to form the second column 602, including A2, A3, A4, A6, A7, A8, A10, A11, and A12. Figure 7 As shown, the 3×3 matrix in the lower left corner of the input data is then expanded to form a third column 701 containing A5, A6, A7, A9, A10, A11, A13, A14, and A15; finally, the 3×3 matrix in the lower right corner of the input data is expanded to form a fourth column 702 containing A6, A7, A8, A10, A11, A12, A14, A15, and A16. Since the convolution kernel slides a total of 4 times, the im2col operation results in a 9×4 matrix 703, which the computing device 201 then multiplies based on matrix 703 to obtain the operation result.

[0058] In this embodiment, the storage core 307 determines whether the input data involves the im2col instruction operation. When the input data involves the im2col instruction operation, the storage core 307 initiates the reconstruction of the input data. Figure 8An exemplary diagram illustrates the execution of im2col on input data of dimension NCHW, where N represents the batch size, C represents the number of feature map channels, H represents the height of the feature map, and W represents the width of the feature map. Taking image data as an example, N represents the number of images in the batch, H represents the number of pixels in the vertical direction of the image, W represents the number of pixels in the horizontal direction, and C represents the number of channels (e.g., C is 1 for a black and white image and C is 3 for an RGB color image).

[0059] For ease of explanation, for a single image (N=1), the input data can be considered as a two-dimensional matrix 801 of size Hi×Wi with input channels (Ci), where the subscript i represents the input. If the stride is 1, the processing module (storage kernel 307) selects Wo elements along the Wi dimension each time, and moves them from the temporary storage module (SRAM 308) to the NRAM 431 in the storage module 43 based on the move instruction (move) (the gradient of the output neuron is moved to WRAM 432). Assuming the dimension of the convolution kernel is Kw×Kh, the operation is repeated Kw times in the Wi dimension and Kh times in the Hi dimension to obtain the submatrix 802 after im2col expansion, which has the shape [Kh×Kw, Ho×Wo]. Since the input data has Ci channels, the storage kernel 307 repeats the above operation Ci times to obtain the matrix after im2col expansion, which has the shape [Ci×Kh×Kw, Ho×Wo], where the subscript o represents the output. The gradient of the output neuron is also expanded using im2col in the same way.

[0060] If the step size is greater than 1, the situation becomes much more complicated. Figure 9 Another schematic diagram of the input data is shown. As shown, the size of a single input data sheet 901 is Hi×Wi×Ci. Assuming the bandwidth of MVDMA 434 can move 2 data blocks per frame, then in the first frame, data blocks B1 and B2 are moved from SRAM 308 to NRAM 431; in the second frame, data blocks B3 and B4 are moved; in the third frame, data blocks B5 and B6 are moved, and so on. Further assuming a step size of 2, the data blocks actually required by the computation module 42 are no longer consecutive, but sequentially B1, B3, B5, etc. In other words, B2 moved in the first frame is redundant data not currently needed, B4 moved in the second frame is redundant data not currently needed, and B6 moved in the third frame is redundant data not currently needed, resulting in a decrease in effective bandwidth. In other words, when the step size is greater than 1, the data size of the input data is only 1 data block. In this embodiment, the data size refers to the amount of data moved per frame, that is, the maximum data length required for computation that can be moved per frame.

[0061] To avoid a decrease in effective bandwidth, this embodiment reorganizes the input data based on the step size and the size of the transported data to generate reorganized input data 902. In this way, each data block transported in each cycle contains mostly the data needed at that moment, reducing redundant data. Furthermore, as mentioned earlier, when the step size is 2 and the transported data size is 1 (i.e., when the step size is greater than the transported data size), this embodiment can be used to reorganize the input data to increase the transported data size. To clearly distinguish the transported data size of the input data before and after reorganization, in this embodiment, the transported data size before reorganization is referred to as the original transported data size.

[0062] Another embodiment of this disclosure is a method for reconstructing input data. This method is applied to an artificial intelligence accelerator, which includes a processing module, a temporary storage module, a storage module, and a computing module. The method for reconstructing input data includes:

[0063] Step S1: When the step size is greater than 1 and greater than the original data size, the processing module samples multiple data blocks in the input data at an interval of one less than the step size;

[0064] The input data can be image data or voice data for image processing or voice processing, and the input data can be two-dimensional or multi-dimensional data, including high-dimensional and low-dimensional data. This application does not impose any restrictions on this.

[0065] Optionally, the input data is transposed data, that is, the input data is transposed before sampling the data blocks in the input data, and the high dimension of the input data is transposed to the low dimension.

[0066] This embodiment only reassembles the input data when the higher-dimensional size of the input data is larger than the original size of the transported data. Taking two-dimensional input data as an example, the first dimension is the lower dimension, and the second dimension is the higher dimension. Data transmission occurs along the first dimension. When the second-dimensional size of the input data is larger than the original size of the transported data, the size of the transported data during data reassembly is equal to the size of the higher-dimensional size. Therefore, the higher-dimensional size must be larger than the original size of the transported data to have a bandwidth-enhancing effect, and this embodiment will only reassemble the input data.

[0067] The processing module samples data blocks of the transposed input data based on the step size and the size of the transposed input data.

[0068] Step S2: The processing module reassembles the input data, wherein the plurality of data blocks are arranged adjacently to form a first data group; the processing module arranges the sampled plurality of data blocks adjacently to form the first data group. The processing module arranges the plurality of data blocks adjacently based on their order in the input data. The processing module arranges the sampled plurality of data blocks adjacently to form the first data group.

[0069] When reorganizing the input data, the processing module arranges the remaining data blocks that are not in the first data group adjacently to form a second data group.

[0070] When the processing module reassembles the input data, it arranges the first data group before the second data group.

[0071] Step S3: The temporary storage module stores the reconstructed input data.

[0072] Step S4: The storage module prioritizes transferring and storing the first data group according to the size of the reorganized input data from the temporary storage module;

[0073] Optionally, the method further includes transposing the reconstructed input data and moving and storing it according to the transposed data at the scale of the reconstructed transport data.

[0074] The size of the reorganized transported data is equal to the output width Wo of the input data.

[0075] Step S5: The operation module obtains the first data group from the storage module and performs the operation; the operation can be matrix operation, convolution operation, etc. when processing the input data, and this application does not limit it.

[0076] The following demonstration uses a convbpfilter operation with a step size greater than 1 and larger than the original data size to illustrate the process of reconstructing the input data in this embodiment, with the input data arranged in the order of NHWC as an example. When performing the convbpfilter operation, the input data (neuron data) consists of the input neurons during backpropagation in the neural network training, and the weight data consists of the gradients of the output neurons during backpropagation in the neural network training.

[0077] For a single image, input data 901 has a three-dimensional structure (Hi×Wi×Ci). Without considering the Ci dimension, input data 901 has a two-dimensional structure (Hi×Wi) as shown in the figure. When the step size is greater than 1, storage kernel 307 samples data blocks in input data 901 at intervals one less than the step size. Taking a step size of 2 as an example, storage kernel 307 samples input data 901 at intervals of one data block, that is, sampling 15 data blocks such as B1, B3, B5, ..., B29. Therefore, the original transported data size is one data block. Further considering the Ci dimension, the actual three-dimensional step size is step size × Ci. Storage kernel 307 repeats sampling multiple times based on the three-dimensional step size. The number of samplings depends on the smaller of the step size and Kw, with the starting position shifted to the right by Ci each time. Repeating the above procedure Hi times yields all sampled data blocks for a single image.

[0078] Storage core 307 arranges the sampled data blocks adjacently to form a structure such as Figure 9The first data group 903 shown is formed by arranging the remaining data blocks (excluding those in the first data group 903) adjacently in the input data to create a second data group 904. Again, using a two-dimensional structure (Hi×Wi) without considering the Ci dimension, the storage core 307 arranges the data blocks adjacently based on their order in the input data 901. Taking B1 to B6 as an example, B1, B3, and B5 are data blocks in the first data group 903, and B2, B4, and B6 are data blocks in the second data group 904. During reassembly, the storage core 307 arranges B1, B3, and B5 adjacently in the first data group 903 according to their order in the input data 901, and arranges B2, B4, and B6 adjacently in the second data group 904 according to their order in the input data 901. Furthermore, during the reassembly of the input data, the storage core 307 places the first data group 903 before the second data group 904, ultimately forming the reassembled input data 902. Those skilled in the art can easily deduce the recombined three-dimensional or higher-dimensional structure, so it will not be elaborated upon.

[0079] Storage core 307 stores the reconstructed input data 902 in SRAM 308.

[0080] Since im2col operates in the order of NCHW, storage core 307 transposes input data 902 from NHWC to NCHW.

[0081] Next, the input data and weight data are expanded using im2col. Figure 10 The diagram illustrates the im2col expansion using the reconstructed input data 902 as an example, assuming the following parameters: Hi = 2, Wi = 2, Ho = 2, Wo = 2, H dimension step size = 2, W dimension step size = 2. Since the W dimension step size = 2, if im2col expansion is performed on the input data before reconstruction, the original data transfer size is only one data block. After reconstruction, the expanded matrix 1001 has the shape [Kh×Kw, Ho×Wo], eliminating the need for skipping values ​​according to the step size, allowing for continuous data transfer. Therefore, the data transfer size becomes Wo, which is Wo times larger than the reconstructed input data 902. The bandwidth for data transfer is significantly improved. The expanded matrix 1001 is temporarily stored in SRAM 308. The gradient of the output neuron is also expanded using the aforementioned method.

[0082] At this point, the input data (input neuron) and weight data (output neuron gradient) have been expanded into left and right matrices in a matrix multiplication operation.

[0083] Next, matrix multiplication is performed on the input data and weight data after Im2col expansion. First, the input data after Im2col expansion needs to be moved from SRAM 308 to NRAM 431 based on the reorganized transport data size, and the weight data after Im2col expansion needs to be moved from SRAM 308 to WRAM 432. Taking the movement of the im2col expanded matrix 1001 from SRAM 308 to NRAM 431 as an example, since the first data group 903 is the data required by the current operation module 42 after reorganization according to the step size and transport data size, it is moved to NRAM 431 first, before the second data group 904. NRAM 431 moves and stores the data block of the first data group 903 from SRAM 308 according to the reorganized transport data size (Wo). After moving the first data group 903 to NRAM 431, NRAM 431 moves and stores the second data group 904 from SRAM 308 according to the reorganized transport data size. Specifically, such as Figure 10 As shown, the data block group 1002 of the first data group 903 preferentially moves the NRAM 431 from the SRAM 308, and then the data block group 1003 of the second data group 904 moves the NRAM 431 from the SRAM 308.

[0084] In one scenario, the data size of the first data group 903 is not an integer multiple of the bandwidth of the MVDMA 434. For example, the MVDMA 434 can transmit two data blocks per clock cycle, while the Wo dimension of the first data group 903 is three data blocks. The NRAM 431 will move the input data 902 to the NRAM 431 in the following order: First, move the data blocks in the first data group 903 whose data size is a multiple of the maximum bandwidth, i.e., data block group 1002. Next, move the data blocks in the second data group 904 whose data size is a multiple of the maximum bandwidth, i.e., data block group 1003. Then, move the data blocks in the first data group 903 that have not yet been moved, i.e., data block group 1004. Finally, move the data blocks in the second data group 904 that have not yet been moved, i.e., data block group 1005. Although some data blocks in data block groups 1004 and 1005 are moved repeatedly with data blocks in data block groups 1002 and 1003, the efficiency is higher than moving the input data before reassembly.

[0085] After the recombined input data 902 is transferred to NRAM 431 in the aforementioned order, the arithmetic module 42 retrieves the first data group 903 from NRAM 431 for operation. After operating on the first data group 903, it retrieves the second data group 904 from NRAM 431 for operation. Specifically, the matrix multiplication formula is as follows:

[0086] C m×n =A m×k ×Bk×n

[0087] Among them, A m×k Let B be the matrix 1001 obtained by expanding the input data using im2col. k×n The weight data (output neuron gradient) after im2col expansion has a shape of [Ho×Wo,Co] or [Co,Ho×Wo], where C m×n The output data is also called the output neuron, which is the weight gradient, m = Ci × Kh × Kw, k = Ho × Wo, n = Co.

[0088] To avoid redundant input / output and improve computational efficiency, this embodiment adopts an on-chip residency approach, storing the partial sums obtained from each calculation on-chip and directly accumulating them with the next partial sum, thereby reducing the number of data interaction operations with DRAM 204.

[0089] Due to limitations in on-chip storage space, a single computation cluster 305 cannot fully accommodate large amounts of weight gradient data. Therefore, the computation module 42 splits the weight gradient based on dimensions Ci and Co. However, splitting may reduce the bandwidth for reading data and decrease the output channel size of the computation cluster 305. To avoid these issues affecting computational efficiency, the computation module 42 first determines whether the space of a single computation cluster 305 can fully accommodate the weight gradient. If so, the computation module 42 performs matrix multiplication without splitting the data.

[0090] If the space of a single computation cluster 305 cannot completely hold the weight gradient, the operation module 42 further determines whether the sum of the spaces of all computation clusters 305 in the computing device 201 (i.e., the total number of NRAMs 431, where the total number refers to the number of NRAMs 431 in the computing device 201) can completely hold the weight gradient. If so, the operation module 42 selects the computation clusters 305 participating in the operation based on the number of computation clusters 305, using factors of that number (splitting number), and then splits the weight gradient using the splitting number, where the splitting number refers to the number of computation clusters 305 participating in the matrix multiplication operation. For example, if the total number of computation clusters 305 in the computing device 201 is 8, then the splitting number is set to factors of 8: 2, 4, and 8, meaning the number of computation clusters 305 participating in the matrix multiplication operation can be 2, 4, and 8.

[0091] Furthermore, this embodiment performs the above-mentioned splitting on both the input channel dimension (Ci) and the output channel dimension (Co). The operation module 42 is split into M parts along the input channel dimension of the weight gradient and into N parts along the output channel dimension of the weight gradient, where M and N are factors of the number of matrix multiplication operations participated in by the computation cluster 305.

[0092] This embodiment also includes determining the number of times the weight gradient is split based on the storage space of the computing clusters and the size of the weight gradient. The splitting is evaluated from smallest to largest, in this order, until a splitting condition is found that satisfies the splitting criteria. The splitting condition refers to whether the data after splitting the weight gradient according to the number of splits can reside in the storage space of the computing clusters. If a lower number of splits can satisfy the condition for residing the weight gradient, the operation module 42 will not attempt a higher number of splits. Based on this principle, in this step, the operation module 42 prioritizes determining the minimum number of splits, i.e., the case where the weight gradient resides in two computing clusters 305. The weight gradient is split into two computing clusters, meaning the storage space of the two computing clusters is sufficient to satisfy the weight gradient's residency. The operation module 422 determines whether this weight gradient can be supported by two computing clusters 305. If the weight gradient can reside in two computing clusters, the case of four or eight computing clusters 305 participating in the operation is no longer considered, and the weight gradient is directly split using the specified number of splits. Optionally, the cases where the weight gradient resides in two computation clusters include combinations of weight gradient splits along dimensions Ci and Co [M,N] of [1,2] and [2,1], where [1,2] indicates that the input channel dimension is not split and is operated by one computation cluster 305, and the output channel dimension is split and operated by two computation clusters 305; [2,1] indicates that the input channel dimension is split and operated by two computation clusters 305, and the output channel dimension is not split and is operated by one computation cluster 305.

[0093] In one scenario, the aforementioned weight gradient can be supported by two computational clusters 305, meaning there are multiple splitting combinations of M and N that satisfy the condition. The computation module 42 will select the most efficient splitting combination based on the Co dimension as the splitting method, and the efficiency is determined using the following relationship:

[0094]

[0095] Where CoL is the length in the Co direction, and LT_NUM is the number of computation modules 42. ceil rounds up. A larger value indicates higher efficiency, and the calculation module 42 selects the larger value for splitting and combining as the input data splitting method.

[0096] If the two computation clusters 305 fail to satisfy the residency weight gradient, the operation module 422 further determines whether the four computation clusters 305 satisfy the residency weight gradient. If the four computation clusters satisfy the residency weight gradient, the weight gradient is split according to the number of splits corresponding to the four computation clusters. At this time, there are three possible split combinations of the weight gradient: [1,4], [2,2], and [4,1]. The judgment method is the same as before and will not be repeated. If the four computation clusters 305 also fail to satisfy the residency weight gradient, the operation module 422 then determines whether the eight computation clusters 305 can satisfy it. At this time, there are four possible split combinations of the weight gradient: [1,8], [2,4], [4,2], and [8,1]. The judgment is also performed in the same way as above.

[0097] If the sum of the spaces of all computational clusters 305 cannot completely contain the weight gradient, the operation module 42 adjusts the number of weight gradient splits to an integer multiple (not including 1) of the number of computational clusters 305. Furthermore, the operation module 42 also splits the weight gradient into M parts along the Ci dimension and into N parts along the Co dimension, where M and N are multiples of the number of computational clusters 305. For example, if the number of computational clusters 305 in the computing device 201 is 8, the number of weight gradient splits can be 16, 32, or 64, i.e., M or N can be 16, 32, or 64, and the splitting method with the highest computational efficiency is selected. The splitting principle also follows the principle of increasing the number of splits. If multiple splitting combinations satisfy the input and output dimensions for the same number of splits, the values ​​are calculated based on the above relationship, and the operation module 42 selects the splitting combination with the larger value as the weight gradient splitting method.

[0098] Since the weight gradient splitting method is already determined, the computation module 42 splits the weight data based on the number of splits of the weight gradient. Specifically, the number of splits of the weight data is the same as the number of splits of the weight gradient, and only the same computation cluster 305 has the corresponding input neurons and weight data for computation. Based on the above principle, the computation module 42 determines the number of parts to split the weight data, and each part is handed over to a different computation cluster 305 for matrix multiplication to obtain intermediate results.

[0099] If the number of weight gradient splits is not less than the total number of computational clusters 305, the result calculated by each computational cluster 305 has no overlap with other computational clusters 305, and can be directly output as the final calculation result. If the number of weight gradient splits is less than the total number of computational clusters 305, after each computational cluster 305 calculates its partial sum, the partial sum needs to be reduced to obtain the final calculation result (weight gradient).

[0100] In this embodiment, CDMA 310 is based on Figure 5The operation mode shown undertakes the data interaction work of the accumulation task. Utilizing the communication function of SRAM 308 between computing clusters 305, the computing module 42 adopts the Ring All-Reduce algorithm to perform reduction and summation calculation. On the one hand, it makes full use of the communication bandwidth based on the bidirectional communication function of CDMA 310, and on the other hand, it avoids the problem of low hardware utilization caused by tree reduction.

[0101] This completes the convbpfilter operation in this embodiment.

[0102] Another embodiment of this disclosure is a method for reconstructing input data. Figure 11 This diagram illustrates a flowchart of a matrix multiplication operation implemented with a step size greater than 1 and larger than the original data size. The input data in this embodiment consists of input neurons, the weight data is the weight data from the forward propagation during neural network inference, and the result data is the output neurons.

[0103] In step 1101, the input neuron is transposed. Figure 12 This diagram illustrates the reorganization of the input neuron in this embodiment. Using a step size of 2, the input neuron 1201 (Wi = 10, Hi = 2) is sequentially read in blocks C1, C3, C5, and C7, and so on, with the original data transport size being one block. In this step, the input neuron 1201 is transposed to generate a transposed input neuron 1202. The lower dimension of the transposed input neuron 1202 becomes Hi, and the data transport size is now two blocks. This embodiment only reorganizes the input neuron when the transported data size after transposition is larger than the original transported data size.

[0104] In step 1102, multiple data blocks in the transposed input neuron are sampled at intervals decreasing by one step size. Data blocks in the transposed input neuron 1202 are sampled at intervals decreasing by one step size, resulting in 10 data blocks: C1, C3, C5, ..., C19.

[0105] In step 1103, the input neurons are reorganized. The data blocks sampled in step 1101 are arranged adjacently to form a first data group, and the remaining data blocks in the input data that are not in the first data group are arranged adjacently to form a second data group. Specifically, the data blocks are arranged adjacently based on their order in the transposed input neurons 1202 to form a first data group 1203, and then the remaining data blocks in the transposed input neurons 1202 that are not in the first data group 1203 are arranged adjacently to form a second data group 1204.

[0106] In step 1104, the reconstructed input neurons are stored.

[0107] In step 1105, the recombined input neuron is transposed. The transposed input neuron forms 1205, with a data transport scale of Wo. Compared to input neuron 1201, the data transport scale of input neuron 1205 increases by a factor of Wo. Figure 12 The explanation uses a two-dimensional matrix, but in reality, im2col operates in the order of NCHW. Therefore, in this step, the input neuron is transposed from NHWC to NCHW.

[0108] In step 1106, the input neurons and weight data are expanded using im2col. As mentioned earlier, the size of the reconstructed input data is larger than the original size.

[0109] In step 1107, matrix multiplication is performed on the expanded input neurons and weight data of Im2col. This step is also based on the matrix multiplication method of the previous embodiment, so it will not be described in detail.

[0110] This completes the matrix multiplication operation in this embodiment.

[0111] Another embodiment of this disclosure is a computer-readable storage medium storing computer program code that processes input data in an artificial intelligence accelerator. When the computer program code is run by a processing device, it executes... Figure 11 The method shown. Another embodiment of this disclosure is a computer program product including a computer program that processes input data in an artificial intelligence accelerator, the computer program being executed by a processor to implement... Figure 11 The steps of the method are shown. Another embodiment of this disclosure is a computer device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement... Figure 11 The steps of the method shown.

[0112] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0113] The purpose of executing computer programs in some embodiments of this disclosure is to process external technical data (such as computer vision, speech, natural language processing, data mining, etc.). By executing neural network processing programs, the computer reorganizes the technical data according to natural laws before transferring the data, thereby achieving faster data transfer and more efficient use of input / output bandwidth. Furthermore, the purpose of executing computer programs in some embodiments of this disclosure is to improve the internal performance of a computer system. By executing performance improvement programs that reorganize and transfer data, the computer performs a series of settings or adjustments on various components of the computer system according to natural laws, thereby achieving faster data transfer and more efficient use of input / output bandwidth to improve the internal performance of the computer system.

[0114] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0115] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0116] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0117] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0118] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0119] The foregoing can be better understood in accordance with the following terms:

[0120] Clause A1. An artificial intelligence accelerator for reorganizing input data to accelerate data transfer, characterized in that it comprises: a processing module for: sampling multiple data blocks in the input data at intervals of one less than the step size when the step size is greater than 1 and greater than the original data transfer size; reorganizing the input data, wherein the multiple data blocks are arranged adjacently to form a first data group; a temporary storage module for storing the reorganized input data; a storage module for preferentially transferring and storing the first data group from the temporary storage module according to the data transfer size of the reorganized input data; and a computation module for performing computation on the first data group obtained from the storage module.

[0121] Clause A2, the artificial intelligence accelerator as described in Clause A1, wherein the processing module is arranged adjacently in the input data based on the order of the plurality of data blocks.

[0122] Clause A3, the artificial intelligence accelerator according to Clause A1, wherein the processing module, when reorganizing the input data, arranges the remaining data blocks in the input data that are not the first data group adjacently to form a second data group.

[0123] Clause A4, the artificial intelligence accelerator as described in Clause A3, wherein the processing module, when reorganizing the input data, arranges the first data group before the second data group.

[0124] Clause A5. The artificial intelligence accelerator according to Clause A1, wherein after the storage module moves the first data group to the storage module, the second data group is moved from the temporary storage module to the storage module according to the size of the reorganized input data.

[0125] Clause A6. The artificial intelligence accelerator according to Clause A5, wherein the computing module performs calculations on the first data group based on the reorganized transported data size, and then performs calculations on the second data group obtained by the storage module based on the reorganized transported data size.

[0126] Clause A7. In the artificial intelligence accelerator described in Clause A5, when the data volume of the first data group is not an integer multiple of the bandwidth, the storage module moves the input data to the storage module in the following order: data blocks in the first data group whose data volume is a multiple of the maximum bandwidth; data blocks in the second data group whose data volume is a multiple of the maximum bandwidth; and data blocks in the first data group that have not yet been moved.

[0127] Clause A8, the artificial intelligence accelerator as described in Clause A1, wherein when the input data involves an im2col operation, the processing module initiates the reorganization of the input data.

[0128] Clause A9, the artificial intelligence accelerator as described in Clause A1, wherein the reorganized transported data size is the maximum data length required for computation.

[0129] Clause A10, the artificial intelligence accelerator according to Clause A1, wherein when the high-dimensional size of the input data is greater than the original transported data size, the processing module reassembles the input data.

[0130] Clause A11. An artificial intelligence accelerator, characterized in that it comprises: a processing module, configured to: sample multiple data blocks in input data at intervals of one less than the step size when the step size is greater than 1 and greater than the original transported data size; reassemble the input data, wherein the multiple data blocks are arranged adjacently to form a data group; perform im2col expansion on the reassembled input data; a storage module, configured to move and store the data group in the order after im2col expansion according to the reassembled transported data size; and a computation module, configured to obtain the data group from the storage module and perform matrix multiplication operations.

[0131] Clause A12, the artificial intelligence accelerator as described in Clause A11, wherein the processing modules are arranged adjacently in the input data based on the order of the plurality of data blocks.

[0132] Clause A13, the artificial intelligence accelerator according to Clause A11, wherein the input data is an input neuron, and the computing module performs matrix multiplication on the gradients of the input neuron and the output neuron to obtain the weight gradient.

[0133] Clause A14. The artificial intelligence accelerator according to Clause A13 includes multiple computing clusters, wherein the computing module is configured to: compare the sum of the storage spaces of the multiple computing clusters with the magnitude of the weight gradient, wherein the sum of the storage spaces is the total storage space of the neuron storage units in the artificial intelligence accelerator; split the weight gradient based on a split number of the weight gradient, wherein: if the sum of the storage spaces is not less than the magnitude of the weight gradient, the split number is a factor of the number of the multiple computing clusters; if the sum of the storage spaces is less than the magnitude of the weight gradient, the split number of the weight gradient is a multiple of the number of the multiple computing clusters.

[0134] Clause A15, the artificial intelligence accelerator according to Clause A14, wherein when the sum of the storage spaces is less than the size of the weight gradient, after the computation module performs matrix multiplication to obtain intermediate results, all intermediate results are accumulated to obtain the computation result.

[0135] Clause A16, the artificial intelligence accelerator as described in Clause A11, wherein the reorganized transport data size is larger than the original transport data size.

[0136] Clause A17, the artificial intelligence accelerator as described in Clause A11, wherein when the scale of the high-dimensional transported data of the input data is greater than the scale of the original transported data, the processing module reassembles the input data.

[0137] Clause A18, an integrated circuit device including an artificial intelligence accelerator as described in any one of Clauses A1 to A17.

[0138] Clause A19, a board including an integrated circuit device as described in Clause A18.

[0139] Clause A20. A method for reorganizing input data to accelerate data transfer, characterized in that the method includes: sampling multiple data blocks in the input data at intervals of one less than the step size when the step size is greater than 1 and greater than the original data transfer size; reorganizing the input data, wherein the multiple data blocks are arranged adjacently to form a first data group; transferring the first data group preferentially according to the reorganized data transfer size; and performing calculations on the transferred first data group.

[0140] Clause A21. A computer-readable storage medium having stored thereon computer program code for processing input data in an artificial intelligence accelerator, wherein when the computer program code is run by a processing device, it performs the method described in Clause A20.

[0141] Clause A22. A computer program product comprising a computer program for processing input data in an artificial intelligence accelerator, characterized in that, when the computer program is executed by a processor, it implements the steps of the method described in Clause A20.

[0142] Clause A23. A computer apparatus comprising a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described in Clause A20.

[0143] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An artificial intelligence accelerator that reorganizes input data to accelerate transfer, characterized in that, include: The processing module is used to: When the step size is greater than 1 and greater than the original data size, multiple data blocks in the input data are sampled at intervals of the step size minus one. The original transport data size is the transport data size before reorganization; When the high-dimensional size of the input data is larger than the original transported data size, the input data is reorganized, wherein the multiple data blocks are arranged adjacently to form a first data group; and When reorganizing the input data, the processing module arranges the remaining data blocks that are not in the first data group adjacently to form a second data group. The temporary storage module is used to store the reconstructed input data; Storage module, The temporary storage module preferentially moves and stores the first data group according to the size of the reorganized input data; After the first data group is moved to the storage module, the second data group is moved from the temporary storage module to the storage module according to the size of the reorganized input data. When the data volume of the first data group is not an integer multiple of the bandwidth, the storage module moves the input data to the storage module according to the following order: The first data group contains data blocks whose data size is a multiple of the maximum bandwidth. The second data group contains data blocks whose data size is a multiple of the maximum bandwidth. Data blocks in the first data group that have not yet been moved; Data blocks in the second data group that have not yet been moved; The processing module obtains the first data group from the storage module and performs processing.

2. The artificial intelligence accelerator according to claim 1, wherein the processing module is arranged adjacently in the input data based on the order of the plurality of data blocks.

3. The artificial intelligence accelerator according to claim 1, wherein the processing module arranges the first data group before the second data group when reorganizing the input data.

4. The artificial intelligence accelerator according to claim 1, wherein the computing module performs calculations on the first data group based on the transported data scale of the recombined input data, and then obtains the second data group from the storage module based on the transported data scale of the recombined input data for calculation.

5. The artificial intelligence accelerator of claim 1, wherein when the input data involves an im2col operation, the processing module initiates the reconstructing of the input data.

6. The artificial intelligence accelerator according to claim 1, wherein the scale of the transported data after reorganization of the input data is the maximum data length required for computation.

7. An artificial intelligence accelerator, characterized in that... include: The processing module is used to: When the step size is greater than 1 and larger than the original transported data size, multiple data blocks in the input data are sampled at intervals of one step size minus one. The original transport data size is the transport data size before reorganization; When the high-dimensional size of the input data is larger than the original transported data size, the input data is reorganized, wherein the multiple data blocks are arranged adjacently to form a first data group; The recombined input data is expanded using im2col; and When reorganizing the input data, the processing module arranges the remaining data blocks that are not in the first data group adjacently to form a second data group. The storage module moves and stores the first data group in the order after im2col expansion, according to the size of the reorganized input data. After the first data group is moved to the storage module, the second data group is moved to the storage module according to the size of the reorganized input data. When the data volume of the first data group is not an integer multiple of the bandwidth, the storage module moves the input data to the storage module according to the following order: The first data group contains data blocks whose data size is a multiple of the maximum bandwidth. The second data group contains data blocks whose data size is a multiple of the maximum bandwidth. Data blocks in the first data group that have not yet been moved; Data blocks in the second data group that have not yet been moved; The computation module obtains the data group from the storage module and performs matrix multiplication operations.

8. The artificial intelligence accelerator of claim 7, wherein the processing module is arranged adjacently in the input data based on the order of the plurality of data blocks.

9. The artificial intelligence accelerator according to claim 7, wherein the input data is an input neuron, and the computing module performs matrix multiplication on the gradients of the input neuron and the output neuron to obtain the weight gradient.

10. The artificial intelligence accelerator of claim 9, comprising a plurality of computing clusters, wherein the computing module is used to: The sum of the storage spaces of the plurality of computing clusters is compared with the magnitude of the weight gradient, wherein the sum of the storage spaces is the total storage space of the neuron storage units in the artificial intelligence accelerator; The weight gradient is split based on the number of splits of multiple weight gradients, wherein: If the sum of the storage spaces is not less than the size of the weight gradient, the number of splits is a factor of the number of the plurality of computation clusters; If the sum of the storage spaces is less than the size of the weight gradient, the number of splits of the weight gradient is a multiple of the number of the plurality of computation clusters.

11. The artificial intelligence accelerator according to claim 10, wherein when the sum of the storage spaces is less than the size of the weight gradient, after the operation module performs matrix multiplication to obtain intermediate results, all intermediate results are accumulated to obtain the calculation result.

12. The artificial intelligence accelerator according to claim 7, wherein the scale of the transported data after reorganization is larger than the scale of the original transported data.

13. An integrated circuit device comprising an artificial intelligence accelerator according to any one of claims 1 to 12.

14. A board comprising the integrated circuit device according to claim 13.

15. A method for reorganizing input data to accelerate data transfer, characterized in that, The method includes: When the step size is greater than 1 and greater than the original transported data size, multiple data blocks in the input data are sampled at intervals of the step size minus one; the original transported data size is the transported data size before reassembly. When the high-dimensional size of the input data is larger than the original transported data size, the input data is reorganized, wherein the multiple data blocks are arranged adjacently to form a first data group; When reorganizing the input data, the remaining data blocks in the input data that are not the first data group are arranged adjacently to form a second data group; The first data group will be moved first, based on the size of the reorganized input data. After the first data group is moved to the storage module, the second data group is moved to the storage module according to the size of the reorganized input data. When the data volume of the first data group is not an integer multiple of the bandwidth, the input data is moved to the storage module according to the following order: The first data group contains data blocks whose data size is a multiple of the maximum bandwidth. The second data group contains data blocks whose data size is a multiple of the maximum bandwidth. Data blocks in the first data group that have not yet been moved; Data blocks in the second data group that have not yet been moved; Perform calculations on the first data set after the relocation.

16. A computer-readable storage medium having stored thereon computer program code for processing input data in an artificial intelligence accelerator, wherein when the computer program code is run by a processing device, it performs the method of claim 15.

17. A computer program product comprising a computer program that processes input data in an artificial intelligence accelerator, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 15.

18. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 15.

Images