Apparatus and method for adaptive split optimization

Abstract

The present disclosure relates to an artificial intelligence accelerator, an integrated circuit device, a board, a method, a readable storage medium, a computer program product and a computer device for adaptively splitting and optimizing matrix multiplication operations, wherein the computer device of the present disclosure is included in the integrated circuit device, which includes a general-purpose interconnection interface and other processing devices. The computer device and the other processing devices interact with each other to jointly complete a user-specified computing operation. The integrated circuit device can also include a storage device connected to the computer device and the other processing devices, respectively, for data storage of the computer device and the other processing devices.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computers. More particularly, the present disclosure relates to an artificial intelligence accelerator for adaptive split optimization and convolution operation, an integrated circuit device, a board card, a method, a readable storage medium, a computer program product and a computer device. BACKGROUND

[0002] Convolution operation is a general and basic operation of neural network, which plays an irreplaceable key role in current convolutional neural network. However, the convolution operation has a large amount of calculation, and has a high requirement on the hardware performance of the artificial intelligence accelerator.

[0003] The convolution operation involves a large number of matrix multiplications. Due to the on-chip storage space of the artificial intelligence accelerator, the matrix multiplication cannot complete the operation at one time, and the intermediate results obtained by multiple operations need to be accumulated. The prior art outputs these intermediate results to the off-chip space for temporary storage, and then loads them from the off-chip to the artificial intelligence accelerator for accumulation when the next operation is needed. Such operation leads to a large amount of redundant output and input.

[0004] Therefore, a technical solution for reducing off-chip output and input is urgently needed. SUMMARY

[0005] In order to at least solve one or more technical problems mentioned above, the present disclosure proposes an artificial intelligence accelerator for adaptive split optimization and convolution operation, an integrated circuit device, a board card, a method, a readable storage medium, a computer program product and a computer device in multiple aspects.

[0006] In a first aspect, the present disclosure provides an artificial intelligence accelerator for adaptive split optimization, comprising a plurality of computing clusters, each computing cluster comprising a plurality of processor cores. Each processor core comprises a storage module and an operation module. The storage module comprises a neuron storage unit for storing result data. The operation module is configured to: compare a sum of storage spaces of the plurality of computing clusters and a size of the result data, wherein the sum of storage spaces is a total storage space of the neuron storage units in the artificial intelligence accelerator; if the sum of storage spaces is not less than the size of the result data, obtain a split number of the result data based on a number of the plurality of computing clusters, wherein the split number of the result data is a factor of the number of the plurality of computing clusters; and if the sum of storage spaces is less than the size of the result data, obtain the split number of the result data as a multiple of the number of the plurality of computing clusters.

[0007] In a second aspect, this disclosure provides an artificial intelligence accelerator for convolution operations, comprising multiple computing clusters, each computing cluster including multiple processor cores, and each processor core including a processing module and a computation module. The processing module is used to: sample multiple data blocks from the input data according to a stride interval; reassemble the input data, wherein the multiple data blocks are arranged adjacently to form a data cluster; and perform im2col expansion on the reassembled input data and weight data. The computation module is used to: compare the sum of the storage spaces of the multiple computing clusters with the size of the result data to determine the number of splits, wherein the sum of the storage spaces is the total storage space of the multiple computing clusters, and the result data is the result of matrix multiplication of the input data and weight data; split the im2col expanded weight data according to the number of splits; and perform matrix multiplication on the im2col expanded input data and the split weight data by different computing clusters to obtain the result data.

[0008] In a third aspect, this disclosure provides an integrated circuit device including the aforementioned artificial intelligence accelerator, and a board including the aforementioned integrated circuit device.

[0009] In the fourth aspect, this disclosure provides a method for adaptive split optimization using an artificial intelligence accelerator, the artificial intelligence accelerator comprising multiple computing clusters, each computing cluster comprising multiple processor cores. The method includes: comparing the sum of the storage spaces of the multiple computing clusters with the size of the result data, wherein the sum of the storage spaces is the total storage space of the multiple computing clusters; if the sum of the storage spaces is not less than the size of the result data, obtaining a number of splits for the result data based on the number of multiple computing clusters, wherein the number of splits for the result data is a factor of the number of multiple computing clusters; if the sum of the storage spaces is less than the size of the result data, obtaining a number of splits for the result data that is a multiple of the number of multiple computing clusters.

[0010] In a fifth aspect, this disclosure provides a computer-readable storage medium having stored thereon computer program code utilizing an artificial intelligence accelerator, wherein the above-described method is executed when the computer program code is run by a processing device.

[0011] In a sixth aspect, this disclosure provides a computer program product including a computer program utilizing an artificial intelligence accelerator, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0012] In a seventh aspect, this disclosure provides a computer apparatus including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0013] This disclosure improves computational efficiency by adopting an on-chip residency approach, where each computational result is stored on-chip and then split into multiple computation clusters based on the size of the result data. Different splitting strategies are adopted based on the storage space of the computation clusters and the size of the result data. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0015] Figure 1 This is a structural diagram of the board card shown in the embodiment of this disclosure;

[0016] Figure 2 This is a structural diagram illustrating an integrated circuit device according to an embodiment of this disclosure;

[0017] Figure 3 This is a schematic diagram illustrating the internal structure of a computing device according to an embodiment of the present disclosure;

[0018] Figure 4 This is a schematic diagram illustrating the internal structure of the processor core in an embodiment of this disclosure;

[0019] Figure 5 This is a diagram illustrating when one processor core wants to write data to another processor core in a different cluster;

[0020] Figure 6 and Figure 7 This shows a schematic diagram of the im2col operation;

[0021] Figure 8 This is a schematic diagram illustrating the execution of im2col on input data of dimension NCHW;

[0022] Figure 9 This is a schematic diagram illustrating the input data of an embodiment of this disclosure;

[0023] Figure 10 This is a schematic diagram illustrating the im2col expansion of the recombined input data in the embodiments disclosed herein;

[0024] Figure 11 This is a flowchart illustrating matrix multiplication according to another embodiment of this disclosure;

[0025] Figure 12 This is a schematic diagram illustrating the reorganization of neuronal data according to another embodiment of this disclosure. Detailed Implementation

[0026] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0027] It should be understood that the terms “comprising” and “including” used in this disclosure and claims indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0029] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0030] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0031] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0032] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0033] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0034] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0035] The computing device 201 is an artificial intelligence accelerator configured to perform user-specified operations. It is mainly implemented as a single-core or multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0036] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0037] Processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0038] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0039] Figure 3 A schematic diagram of the internal structure of computing device 201 is shown. Computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. As a system-on-a-chip, computing device 201 includes multiple computing clusters (computing clusters 305), and each computing cluster includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchical structure of system-on-a-chip, computing clusters, and processor cores.

[0040] From the perspective of system-on-a-chip hierarchy, such as Figure 3 As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple computing clusters 305.

[0041] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to respond to access requests from the processor core to access external storage devices, such as… Figure 2The DRAM 204 in the chip allows the computing device 201 to read data from external sources or write data to external sources. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external memory controller 301, the peripheral communication module 302, and multiple computing clusters 305 to transmit data and control signals between modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each computing cluster and ensure information synchronization. The multiple computing clusters 305 are the computing core of the computing device 201. Four are shown exemplary in the figure; however, with hardware development, the computing device 201 disclosed herein may also include 8, 16, 64, or even more computing clusters 305. The computing clusters 305 are used to efficiently execute deep learning algorithms.

[0042] From the perspective of the hierarchy of computational clusters, such as Figure 3 As shown, each computing cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0043] Four processor cores 306 are shown in the figure as an example; this disclosure does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0044] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0045] The computation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0046] Storage module 43 is used to store or move related data, including neuron RAM (NRAM) 431, weight RAM (WRAM) 432, input / output direct memory access (IODMA) 433, and move direct memory access (MVDMA) 434. NRAM 431 stores input, output data, and intermediate results for computation by processor core 306; WRAM 432 stores the weights of the deep learning network; IODMA 433 controls memory access between NRAM 431 / WRAM 432 and DRAM 204 via broadcast bus 309; MVDMA 434 controls memory access between NRAM 431 / WRAM 432 and SRAM 308. It should be noted that NRAM and WRAM here can be two storage regions formed by dividing the same memory in logical storage space, or they can be two independent memories; no specific limitation is made here.

[0047] Back Figure 3 The storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between computing clusters 305 and DRAM 204, communication between computing clusters 305, and communication between processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0048] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a compute cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same compute cluster 305 does not need to be obtained from DRAM 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip input / output access.

[0049] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between computing clusters 305, and data transfer between computing clusters 305 and DRAM 204, respectively. These will be explained separately below.

[0050] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the computing cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission. Multicast is a communication method that transmits split data from SRAM 308 to a few specific processor cores 306. Broadcast is a communication method that transmits split data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0051] CDMA 310 is used to control access to SRAM 308 between different computing clusters 305 within the same computing device 201. Figure 5 This diagram illustrates the operation of CDMA 310 when one processor core attempts to write data to another processor core in a different computing cluster. In this application scenario, the same computing device includes multiple computing clusters. For simplicity, only computing cluster 0 and computing cluster 1 are shown in the diagram. Both clusters include multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown for computing cluster 0, and only processor core 1 is shown for computing cluster 1. Processor core 0 intends to write data to processor core 1.

[0052] First, processor core 0 sends a unicast write request to write data into local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transfer the data to SRAM 1 of computing cluster 1. Then, the slave sends a write response B as a response. Finally, processor core 1 of computing cluster 1 sends a unicast read request to read the data from SRAM 1.

[0053] Back Figure 3The GDMA 311, in conjunction with the external memory controller 301, controls memory access from the SRAM 308 of the computing cluster 305 to the DRAM 204, or reads data from the DRAM 204 into the SRAM 308. As described above, communication between the DRAM 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between the DRAM 204 and the NRAM 431 or WRAM 432 via the IODAM 433; the second channel involves first transmitting data between the DRAM 204 and SRAM 308 via the GDMA 311, and then transmitting data between the SRAM 308 and the NRAM 431 or WRAM 432 via the MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between the DRAM 204 and the NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments disclosed herein allow for the selection of data transmission channels based on the hardware capabilities available.

[0054] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, this disclosure treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure.

[0055] The computing device 201 is often the computational bottleneck of the entire neural network operation when processing convolution operations, especially large-scale convolution operations. Therefore, it is crucial to make full use of the hardware's computing performance to accelerate convolution operations.

[0056] When performing convolution operations using matrix multiplication, the computing device 201 frequently uses `im2col` to expand the input data (neuronal data) from computer vision, speech, natural language processing, and data mining, transforming high-dimensional data into two-dimensional data for computation. For example, this is done when updating the convolution kernel during backpropagation in neural network training. Backpropagation for kernel updates (convolution backward filter) is generally represented as `convbpfilter`, and the output of `convbpfilter` is the convolution gradient. When performing `convbpfilter`, the computing device 201 typically uses the `im2col` operation to expand the gradients of the input and output neurons into two matrices, and then uses matrix multiplication to accelerate computation.

[0057] The purpose of im2col is to optimize convolution operations. For each small window that the convolution kernel needs to process, it is expanded into a row (column) of a new matrix. The number of columns (rows) of the new matrix is ​​the number of times the convolution operation is performed on the input data, which is the number of times the convolution kernel slides.

[0058] Figure 6 and Figure 7 The diagram illustrates the operation of im2col, using a 3×3 kernel, 4×4 input data (A1 to A16), and a stride of 1 as an example. Figure 6 As shown, since the convolution kernel size is 3×3, during im2col, the 3×3 matrix in the upper left corner of the input data is first expanded to form the first column 601, including A1, A2, A3, A5, A6, A7, A9, A10, and A11; then the 3×3 matrix in the upper right corner of the input data is expanded to form the second column 602, including A2, A3, A4, A6, A7, A8, A10, A11, and A12. Figure 7 As shown, the 3×3 matrix in the lower left corner of the input data is then expanded to form a third column 701 containing A5, A6, A7, A9, A10, A11, A13, A14, and A15; finally, the 3×3 matrix in the lower right corner of the input data is expanded to form a fourth column 702 containing A6, A7, A8, A10, A11, A12, A14, A15, and A16. Since the convolution kernel slides a total of 4 times, the im2col operation results in a 9×4 matrix 703, which the computing device 201 then multiplies based on matrix 703 to obtain the operation result.

[0059] In this embodiment, the storage core 307 determines whether the input data involves the im2col instruction operation. When the input data involves the im2col instruction operation, the storage core 307 initiates the reconstruction of the input data. Figure 8An exemplary diagram illustrates the execution of im2col on input data of dimension NCHW, where N represents the batch size, C represents the number of feature map channels, H represents the height of the feature map, and W represents the width of the feature map. Taking image data as an example, N represents the number of images in the batch, H represents the number of pixels in the vertical direction of the image, W represents the number of pixels in the horizontal direction, and C represents the number of channels (e.g., C is 1 for a black and white image and C is 3 for an RGB color image).

[0060] For ease of explanation, for a single image (N=1), the input data can be considered as a two-dimensional matrix 801 of size Hi×Wi with input channels (Ci), where the subscript i represents the input. If the stride is 1, the processing module (storage kernel 307) selects Wo elements along the Wi dimension each time, and moves them from the temporary storage module (SRAM 308) to the NRAM 431 in the storage module 43 based on the move instruction (move) (the gradient of the output neuron is moved to WRAM 432). Assuming the dimension of the convolution kernel is Kw×Kh, the operation is repeated Kw times in the Wi dimension and Kh times in the Hi dimension to obtain the submatrix 802 after im2col expansion, which has the shape [Kh×Kw, Ho×Wo]. Since the input data has Ci channels, the storage kernel 307 repeats the above operation Ci times to obtain the matrix after im2col expansion, which has the shape [Ci×Kh×Kw, Ho×Wo], where the subscript o represents the output.

[0061] If the step size is greater than 1, the situation becomes much more complicated. Figure 9 Another schematic diagram of the input data is shown. As shown, the size of a single input data sheet 901 is Hi×Wi×Ci. Assuming the bandwidth of MVDMA 434 can move 2 data blocks per frame, then in the first frame, data blocks B1 and B2 are moved from SRAM 308 to NRAM 431; in the second frame, data blocks B3 and B4 are moved; in the third frame, data blocks B5 and B6 are moved, and so on. Further assuming a step size of 2, the data blocks actually required by the computation module 42 are no longer consecutive, but sequentially B1, B3, B5, etc. In other words, B2 moved in the first frame is redundant data not currently needed, B4 moved in the second frame is redundant data not currently needed, and B6 moved in the third frame is redundant data not currently needed, resulting in a decrease in effective bandwidth. In other words, when the step size is greater than 1, the data size of the input data is only 1 data block. In this embodiment, the data size refers to the amount of data moved per frame, that is, the maximum data length required for computation that can be moved per frame.

[0062] To avoid a decrease in effective bandwidth, this embodiment reorganizes the input data based on the step size and the size of the transported data to generate reorganized input data 902. In this way, each data block transported in each cycle contains mostly the data needed at that moment, reducing redundant data. Furthermore, as mentioned earlier, when the step size is 2 and the transported data size is 1 (i.e., when the step size is greater than the transported data size), this embodiment can be used to reorganize the input data to reduce redundant data. To clearly distinguish the transported data size before and after reorganization, in this embodiment, the transported data size before reorganization is referred to as the original transported data size.

[0063] Furthermore, this embodiment only reassembles the input data when the higher dimension of the input data is larger than the original transported data size. Taking two-dimensional input data as an example, the first dimension is the lower dimension, and the second dimension is the higher dimension. Data transmission occurs along the first dimension. This embodiment only reassembles the input data when the second dimension of the input data is larger than the original transported data size.

[0064] The following demonstration uses a convbpfilter operation with a step size greater than 1 and larger than the original data size to illustrate the process of reconstructing the input data in this embodiment, with the input data arranged in the order of NHWC as an example. When performing the convbpfilter operation, the input data (neuron data) consists of the input neurons during backpropagation in the neural network training, and the weight data consists of the gradients of the output neurons during backpropagation in the neural network training.

[0065] For a single image, input data 901 has a three-dimensional structure (Hi×Wi×Ci). Without considering the Ci dimension, input data 901 has a two-dimensional structure (Hi×Wi) as shown in the figure. When the step size is greater than 1, storage kernel 307 samples data blocks in input data 901 at intervals one less than the step size. Taking a step size of 2 as an example, storage kernel 307 samples input data 901 at intervals of one data block, that is, sampling 15 data blocks such as B1, B3, B5, ..., B29. Therefore, the original transported data size is one data block. Further considering the Ci dimension, the actual three-dimensional step size is step size × Ci. Storage kernel 307 repeats sampling multiple times based on the three-dimensional step size. The number of samplings depends on the smaller of the step size and Kw, with the starting position shifted to the right by Ci each time. Repeating the above procedure Hi times yields all sampled data blocks for a single image.

[0066] Storage core 307 arranges the sampled data blocks adjacently to form a structure such as Figure 9The first data group 903 shown is used to arrange the remaining data blocks in the input neurons that are not in the first data group 903 adjacently to form the second data group 904. Again, using a two-dimensional structure (Hi×Wi) without considering the Ci dimension, the storage core 307 arranges the data blocks adjacently based on their order in the input data 901. Taking B1 to B6 as an example, B1, B3, and B5 are data blocks in the first data group 903, and B2, B4, and B6 are data blocks in the second data group 904. During recombination, the storage core 307 arranges B1, B3, and B5 adjacently in the first data group 903 according to their order in the input data 901, and arranges B2, B4, and B6 adjacently in the second data group 904 according to their order in the input data 901. Furthermore, when recombinating the input neurons, the storage core 307 arranges the first data group 903 before the second data group 904, ultimately forming the recombined input data 902. Those skilled in the art can easily deduce the recombined three-dimensional or higher-dimensional structure, so it will not be elaborated upon.

[0067] Storage core 307 stores the reconstructed input data 902 in SRAM 308.

[0068] Since im2col operates in the order of NCHW, storage core 307 transposes input data 902 from NHWC to NCHW.

[0069] Next, im2col expansion is performed on the gradients of the input and output neurons. In this embodiment, the weight data are the gradients of the output neurons during backpropagation in the neural network training. Figure 10 The diagram illustrates the im2col expansion using the recombined input data 902 as an example, assuming the relevant parameters are: Hi = 2, Wi = 2, Ho = 2, Wo = 2, H dimension step size = 2, W dimension step size = 2. Since the W dimension step size = 2, if im2col expansion is performed on the input neuron before recombination, the original data transport size is only one data block. After recombination, the im2col expanded matrix 1001 has the shape [Kh×Kw, Ho×Wo], no longer requiring skipping values ​​according to the step size, and can continuously transmit data. Therefore, its data transport size becomes Wo, which is Wo times larger than the transport size of the recombined input data 902, greatly increasing the bandwidth of data transport. The im2col expanded matrix 1001 is temporarily stored in SRAM 308. The gradient of the output neuron is also expanded using the aforementioned method.

[0070] At this point, the weight gradient and the output neuron gradient have been expanded into left and right matrices in a matrix multiplication operation.

[0071] Matrix multiplication is performed on the gradients of the input and output neurons after Im2col expansion. First, the input neurons after Im2col expansion need to be moved from SRAM 308 to NRAM 431 based on the reorganized transported data size, and the gradients of the output neurons after Im2col expansion need to be moved from SRAM 308 to WRAM 432. Taking the movement of the im2col expanded matrix 1001 from SRAM 308 to NRAM 431 as an example, since the first data group 903 is the data required by the current computation module 42 after reorganization according to the step size and transported data size, it is moved to NRAM 431 before the second data group 904. NRAM 431 moves and stores the data blocks of the first data group 903 from SRAM 308 in the reorganized order. After moving the first data group 903 to NRAM 431, NRAM 431 moves and stores the second data group 904 from SRAM 308 according to the reorganized transported data size (Wo). Specifically, as follows... Figure 10 As shown, the data block group 1002 of the first data group 903 preferentially moves the NRAM 431 from the SRAM 308, and then the data block group 1003 of the second data group 904 moves the NRAM 431 from the SRAM 308.

[0072] In one scenario, the data size of the first data group 903 is not an integer multiple of the bandwidth of the MVDMA 434. For example, the MVDMA 434 can transmit two data blocks per clock cycle, while the Wi dimension of the first data group 903 is three data blocks. The NRAM 431 will move the input data 902 to the NRAM 431 in the following order: First, move the data blocks in the first data group 903 whose data size is a multiple of the maximum bandwidth, i.e., data block group 1002. Next, move the data blocks in the second data group 904 whose data size is a multiple of the maximum bandwidth, i.e., data block group 1003. Then, move the data blocks in the first data group 903 that have not yet been moved, i.e., data block group 1004. Finally, move the data blocks in the second data group 904 that have not yet been moved, i.e., data block group 1005. Although some data blocks in data block groups 1004 and 1005 are moved repeatedly with data blocks in data block groups 1002 and 1003, the benefit is higher than moving the weight gradient before reorganization.

[0073] After the recombined input data 902 is transferred to NRAM 431 in the aforementioned order, the arithmetic module 42 retrieves the first data group 903 from NRAM 431 for operation. After operating on the first data group 903, it retrieves the second data group 904 from NRAM 431 for operation. Specifically, the matrix multiplication formula is as follows:

[0074] C m×n =A m×k ×Bk×n

[0075] Among them, A m×k Let B be the matrix 1001 after the input neuron is expanded using im2col. k×n The gradient of the output neuron after im2col expansion has a shape of [Ho×Wo,Co] or [Co,Ho×Wo], where C m×n The output neuron is the result of matrix multiplication, which is the weight gradient, m = Ci × Kh × Kw, k = Ho × Wo, n = Co.

[0076] The computation module 42 performs matrix multiplication based on the above formula to obtain the convolution kernel gradient. If N is greater than 1, it accumulates along the N-dimensional axis to obtain the intermediate result of the current computation cluster 305.

[0077] To avoid redundant input / output and improve computational efficiency, this embodiment adopts an on-chip residency approach, storing the partial sums obtained from each calculation on-chip and directly accumulating them with the next partial sum, thereby reducing the number of data interaction operations with DRAM 204.

[0078] First, it is determined whether the storage space of a single computation cluster 305 is not less than the size of the output neuron. Due to the limitation of on-chip storage space, when the data volume of the output neuron is large, a single computation cluster 305 cannot fully accommodate it. Therefore, the operation module 42 obtains the number of gradient splits of the output neuron based on the dimensions Ci and Co. However, splitting may reduce the bandwidth for reading data and make the output channel of the computation cluster 305 smaller. To avoid the above situations affecting the operation efficiency, the operation module 42 determines whether the space of a single computation cluster 305 can fully accommodate the output neuron. Furthermore, although the output neuron is the result of matrix multiplication and the actual data is unknown before the operation, the size of the output neuron is known, which is m×n. Therefore, this embodiment can determine whether the on-chip space is sufficient to accommodate the output neuron based on its size.

[0079] If the storage space of a single computation cluster 305 is not less than the size of the output neuron, there is no need to split the output neuron. The computation module 42 performs matrix multiplication on the gradients of the input neuron and the output neuron without splitting them to obtain the partial sum of the output neuron.

[0080] If the space of a single computing cluster 305 cannot fully accommodate the output neuron, the operation module 42 determines whether the sum of the storage spaces of all computing clusters 305 is not less than the size of the output neuron, that is, whether the total storage space of all NRAMs 431 in the computing device 201 can fully accommodate the output neuron.

[0081] If the sum of the storage spaces is not less than the size of the output neuron, it means that the total storage space of all NRAM 431 can completely house the output neuron. The number of splits for the output neuron is obtained based on the number of computation clusters 305, allowing the output neuron to be split based on this number. The operation module 42 selects the computation clusters 305 participating in the operation based on the number of computation clusters 305, using factors of that number (splitting number), and then obtains the number of splits for the output neuron based on that number. The number of splits for the output neuron refers to the number of computation clusters 305 participating in matrix multiplication operations. For example, if the number of all computation clusters 305 in the computing device 201 is 8, then the number of splits is set to factors of 8: 2, 4, and 8, meaning that the number of computation clusters 305 participating in matrix multiplication operations can be 2, 4, and 8.

[0082] Furthermore, this embodiment performs the above-mentioned splitting on both the input channel dimension (Ci) and the output channel dimension (Co). The operation module 42 is split into M parts along the input channel dimension of the output neuron and into N parts along the output channel dimension of the output neuron, where M and N are factors of the number of computation clusters 305 participating in matrix multiplication operations.

[0083] This embodiment also follows an evaluation process that prioritizes the number of splits from fewest to most, continuing in this order until the splitting condition is met. The splitting condition refers to the number of splits required to divide the result data into a number of computing clusters; that is, how many computing clusters are needed to satisfy the requirement for the result data to reside. For example, in an AI accelerator with 8 computing clusters, and where the sum of the on-chip space of all computing clusters can completely hold the result data, then when determining the number of splits for the result data, all possible splits are: 2 computing clusters, 4 computing clusters, and 8 computing clusters. First, it is determined whether 2 computing clusters can satisfy the requirement for data residency. If 2 computing clusters can, then using 4 or 8 computing clusters also satisfies the splitting condition. In this application, the evaluation starts from the minimum value of the number of splits. If yes, the number of splits for the computing clusters is selected as the splitting method for the result data; otherwise, the evaluation starts from the second smallest value of the number of splits, continuing in this order until the number of splits that satisfy the splitting condition is reached. If a lower number of splits is sufficient to satisfy the condition for residing the output neuron, then the operation module 42 will not attempt a higher number of splits. Based on this principle, the operation module 42 prioritizes determining the minimum number of splits, i.e., the case where two computation clusters 305 reside in the output neuron. The operation module 422 then determines whether the output neuron can be supported by two computation clusters 305. If the output neuron can be supported by two computation clusters 305, then the case where four or eight computation clusters 305 participate in the operation is no longer considered, and the output neuron is directly split according to the splitting condition. Specifically, the output neuron can be supported by two computation clusters 305, including the splitting combination [M,N] of the output neuron along dimensions Ci and Co as [1,2] and [2,1]. There are two splitting combinations, where [1,2] means that the input channel dimension is not split and is operated by one computation cluster 305, and the output channel dimension is split and is operated by two computation clusters 305. [2,1] means that the input channel dimension is split and is operated by two computation clusters 305, and the output channel dimension is not split and is operated by one computation cluster 305.

[0084] In one scenario, when multiple splitting combinations satisfy the splitting conditions, the splitting combination with the highest computational efficiency is selected. For example, if both of the above two splitting combinations can be supported by two computation clusters 305, meaning there are multiple splitting combinations of M and N that satisfy the conditions, the operation module 42 will select the splitting combination with the highest efficiency based on the Co dimension as the splitting method. The efficiency is determined using the following relationship:

[0085]

[0086] Where CoL is the length in the Co direction, and LT_NUM is the number of computation modules 42. ceil represents rounding up. A larger value indicates higher efficiency, and the computation module 42 selects the larger value of the split combination as the splitting method for the output neuron gradient.

[0087] In another scenario, if the two computation clusters 305 fail to retain the output neuron, the operation module 422 further determines whether the four computation clusters 305 with the lowest splitting count can retain the output neuron. If all four computation clusters 305 can also fully retain the output neuron, there are three possible splitting combinations for the output neuron: [1,4], [2,2], and [4,1]. The most efficient splitting combination is then selected based on the Co dimension. The method for selecting the splitting combination is the same as before and will not be repeated. If the four computation clusters 305 also fail to fully retain the output neuron, the operation module 422 continues to determine whether the eight computation clusters 305 can retain the output neuron. If all eight computation clusters 305 can also fully retain the output neuron, there are four possible splitting combinations for the output neuron: [1,8], [2,4], [4,2], and [8,1]. The same evaluation method is used.

[0088] If the sum of the spaces of all computational clusters 305 is less than the size of the output neuron, then the number of output neurons split is a multiple (excluding 1) of the number of computational clusters 305. Furthermore, the computation module 42 also splits the output neuron into M parts along the Ci dimension and into N parts along the Co dimension, where M and N are multiples of the number of computational clusters 305. For example, if the number of computational clusters 305 in the computing device 201 is 8, then the number of splits for the output neuron can be 16, 32, or 64, i.e., M or N can be 16, 32, or 64, and the splitting method with the highest computational efficiency is selected. The splitting principle also follows the principle of increasing the number of splits. If multiple splitting combinations satisfy the same number of input and output dimensions, then the values ​​are calculated based on the above relationship, and the computation module 42 selects the splitting combination with the larger value as the splitting method for the output neuron gradient.

[0089] After determining the number of splits, the weight data is split based on that number. Since the splitting method of the output neuron is already determined, the operation module 42 splits the weight data, i.e., the output neuron gradient, based on the number of splits of the output neuron. Specifically, the number of splits of the output neuron gradient is the same as the number of splits of the output neuron, and only the same computation cluster 305 has the corresponding output neuron gradient for operation.

[0090] The computation module 42 performs matrix multiplication on the gradients of the input neuron and the split output neuron through different computation clusters 305 to obtain a partial sum. To obtain the output neuron, the computation module 42 further determines whether the number of splits of the output neuron is less than the total number of computation clusters 305. If not, the intermediate results (partial sums) calculated by each computation cluster 305 do not overlap with the partial sums of other computation clusters 305, and the computation module 42 directly sums all intermediate results to obtain the final result data.

[0091] If the number of splits in the output neuron is less than the total number of computation clusters 305, it means that the partial sum calculated by each computation cluster 305 overlaps with the partial sums of other computation clusters 305. The operation module 42 needs to perform reduction to obtain the final calculation result. Specifically, assuming there are P computation clusters 305, when the number of splits is Q, the operation module 42 reduces the intermediate results of each (P / Q) computation cluster to obtain the calculation result, which is the output neuron. Taking P=8 as an example, when Q=1 (i.e., no splitting), data interaction and reduction are required between the 8 computation clusters 305; when Q=2, reduction is required between every 4 computation clusters 305; and when Q=4, reduction is required between every 2 computation clusters 305.

[0092] In this embodiment, CDMA 310 is based on Figure 5 The operation mode shown undertakes the data interaction work of the accumulation task. Utilizing the communication function of SRAM 308 between computing clusters 305, the arithmetic module 42 uses the Ring All-Reduce algorithm to perform the above reduction summation calculation. On the one hand, it makes full use of the communication bandwidth based on the bidirectional communication function of CDMA 310, and on the other hand, it avoids the problem of low hardware utilization caused by tree reduction.

[0093] This completes the convbpfilter operation in this embodiment.

[0094] Another embodiment of this disclosure is based on Figures 1 to 5 The method of matrix multiplication using the structure. Figure 11 The flowchart of this embodiment is shown. The input data in this embodiment are the input neurons, the weight data are the weight data during the forward propagation of the neural network inference, and the result data are the output neurons.

[0095] In step 1101, it is determined whether the storage space of a single computation cluster is not less than the size of the output neuron. Due to the limitation of on-chip storage space, when the data volume of the output neuron is large, a single computation cluster cannot fully reside there. Therefore, this embodiment splits the weight data based on dimensions Ci and Co. However, splitting may reduce the bandwidth for reading data and make the output channel of the computation cluster smaller. To avoid the above situations affecting the computational efficiency, this step determines whether the space of a single computation cluster can fully reside the output neuron.

[0096] If the storage space of a single computation cluster is not less than the size of the output neuron, and no splitting is required, then step 1102 is executed to perform matrix multiplication on the gradients of the input neuron and the output neuron without splitting, so as to obtain the partial sum of the output neuron.

[0097] If the space of a single computing cluster cannot fully accommodate the output neuron, then step 1103 is executed to determine whether the sum of the storage spaces of all computing clusters is not less than the size of the output neuron, that is, to determine whether the total storage space of all NRAM431 in computing device 201 can fully accommodate the output neuron.

[0098] If the sum of the storage spaces is not less than the size of the output neuron, indicating that the total storage space of all NRAM 431 can completely house the output neuron, then step 1104 is executed to obtain the number of splits of the output neuron based on the number of splits of the computation clusters. Specifically, based on the number of computation clusters 305, the computation clusters participating in the operation are selected by factoring their number (the number of splits), and then the number of splits of the output neuron is obtained by factoring their number of splits. In this embodiment, the above splits are also performed on both the input channel dimension (Ci) and the output channel dimension (Co), splitting into M parts along the input channel dimension of the output neuron and into N parts along the output channel dimension of the output neuron, where M and N are factors of the number of computation clusters participating in the matrix multiplication operation.

[0099] This embodiment also follows the principle of evaluating the number of splits from lowest to highest, in this order until a number of splits that meets the splitting conditions is found. If a lower number of splits can satisfy the condition for retaining the output neuron, the computation module 42 will no longer attempt splitting with a higher number of splits.

[0100] Similar to the previous embodiment, if multiple splitting combinations satisfy the conditions based on the number of splits, the splitting combination with the highest efficiency is selected as the splitting method based on the Co dimension. The efficiency is also determined by the following relationship, which will not be elaborated further.

[0101]

[0102] In another case, if the minimum number of splits fails to satisfy the requirement of retaining the output neuron, then the next lowest number of splits is determined and evaluated in the manner described above.

[0103] If the sum of the spaces of all computation clusters is less than the size of the output neuron, then proceed to step 1105, where the number of output neurons is split into multiples (excluding 1) of the number of computation clusters. The splitting principle also follows the principle of increasing the number of splits. If multiple splitting combinations of the input and output dimensions can satisfy the same number of splits, then calculate their values ​​based on the above relationship and select the splitting combination with the larger value as the splitting method for the output neuron.

[0104] After steps 1104 and 1105, step 1106 is executed, which splits the weight data based on the number of splits. Specifically, the number of splits of the weight data is the same as the number of splits of the output neurons, and only the same computing cluster has corresponding weight data for computation.

[0105] In step 1107, the input neuron and the split weight data are subjected to matrix multiplication by different computation clusters 305 to obtain a partial sum.

[0106] Figure 12 This diagram illustrates the reorganization of the input neuron in this embodiment. Using a step size of 2, the input neuron 1201 (Wi = 10, Hi = 2) is sequentially read in blocks C1, C3, C5, and C7, and so on, with the original data transport size being one block. In this step, the input neuron 1201 is transposed to generate a transposed input neuron 1202. The lower dimension of the transposed input neuron 1202 becomes Hi, and the data transport size is now two blocks. This embodiment only reorganizes the input neuron when its higher dimension exceeds the original data transport size.

[0107] Next, data blocks in the transposed input neuron 1202 are sampled at intervals decreasing by one step, and 10 data blocks, namely C1, C3, C5, ..., C19, are sampled. Based on the order of the data blocks in the transposed input neuron 1202, they are arranged adjacently to form the first data group 1203. Then, the remaining data blocks in the transposed input neuron 1202 that are not in the first data group 1203 are arranged adjacently to form the second data group 1204.

[0108] Finally, a transpose is performed to form input neuron 1205, at which point the data transported is Wo. Compared to input neuron 1201, the data transported by input neuron 1205 is increased by a factor of Wo.

[0109] Next, the input neuron 1205 and the weight data are expanded using im2col to generate the left and right matrices in the matrix multiplication operation. Then, the expanded input neuron 1205 and the weight data are multiplied by different computational clusters to obtain a partial sum.

[0110] In step 1108, it is determined whether the number of splits of the output neuron is less than the total number of calculated clusters. If not, step 1109 is executed, where all intermediate results (partial sums) are directly summed to obtain the final result data. Step 1109 is also executed after step 1102.

[0111] If the number of splits of the output neuron is less than the total number of computation clusters, then step 1110 is executed to perform reduction to obtain the final computation result. Specifically, assuming there are M computation clusters 305, when the number of splits is N, the operation module 42 reduces the sum of the parts of each (M / N) computation cluster to obtain the computation result, which is the output neuron.

[0112] This completes the matrix multiplication operation in this embodiment.

[0113] This embodiment can adaptively split and optimize matrix multiplication operations. Specifically, it is based on the result data of matrix multiplication and the size of the hardware to adaptively split the data. After determining the number of splits for the result data, the weight data is split based on the number of splits to perform matrix multiplication operations.

[0114] Another embodiment of this disclosure is a computer-readable storage medium storing computer program code utilizing an artificial intelligence accelerator, which, when executed by a processing device, performs... Figure 11 The method shown. Another embodiment of this disclosure is a computer program product including a computer program utilizing an artificial intelligence accelerator, which, when executed by a processor, implements... Figure 11 The steps of the method are shown. Another embodiment of this disclosure is a computer device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement... Figure 11 The steps of the method shown.

[0115] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] The purpose of executing computer programs in some embodiments of this disclosure is to process external technical data (such as computer vision, speech, natural language processing, data mining, etc.). By executing neural network processing programs on a computer and adopting an on-chip residing approach according to natural laws, the calculated portions and residing on the chip each time are stored, thereby reducing the movement of redundant data and achieving the technical data processing effect of accelerated data movement and efficient utilization of input / output bandwidth. Furthermore, the purpose of executing computer programs in some embodiments of this disclosure is to improve the internal performance of a computer system. By adopting an on-chip residing approach, the calculated portions and performance improvement programs residing on the chip each time are used to perform a series of settings or adjustments on various components of the computer system according to natural laws, thereby reducing the movement of redundant data and achieving the internal performance improvement effect of accelerated data movement and efficient utilization of input / output bandwidth.

[0117] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0118] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0119] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0120] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0121] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0122] The foregoing can be better understood in accordance with the following terms:

[0123] Clause A1. An adaptive split-optimized artificial intelligence accelerator, comprising multiple computing clusters, each computing cluster comprising multiple processor cores, characterized in that each processor core comprises: a storage module, including neuron storage units for storing result data; and a computation module for: comparing the sum of the storage spaces of the multiple computing clusters with the size of the result data, wherein the sum of the storage spaces is the total storage space of the neuron storage units in the artificial intelligence accelerator; and obtaining a split number of the result data based on the split number of the multiple computing clusters, wherein: if the sum of the storage spaces is not less than the size of the result data, the split number of the result data is a factor of the number of the multiple computing clusters; and if the sum of the storage spaces is less than the size of the result data, the split number of the result data is a multiple of the number of the multiple computing clusters.

[0124] Clause A2, the artificial intelligence accelerator as described in Clause A1, wherein the resulting data are the output neurons of backpropagation during neural network training.

[0125] Clause A3, the artificial intelligence accelerator according to Clause A1, wherein the computing module is further configured to compare the storage space of a single computing cluster with the size of the result data, and if the storage space of a single computing cluster is not less than the size of the result data, the computing module stops obtaining the number of splits of the result data.

[0126] Clause A4. The artificial intelligence accelerator according to Clause A1, wherein the computation module is used to perform matrix multiplication on the gradients of the input neuron and the output neuron to obtain the result data, and the computation module is further used to split the gradient of the output neuron, wherein the number of splits of the gradient of the output neuron is the same as the number of splits of the result data.

[0127] Clause A5. The artificial intelligence accelerator according to Clause A1, wherein when the sum of the storage spaces is not less than the size of the result data, the computing module is split into M parts along the input channel dimension of the result data and into N parts along the output channel dimension of the result data, where M and N are factors of the number of the plurality of computing clusters.

[0128] Clause A6. The artificial intelligence accelerator according to Clause A1, wherein when the sum of the storage spaces is less than the size of the result data, the computing module is split into M parts along the input channel dimension of the result data and into N parts along the output channel dimension of the result data, where M and N are multiples of the number of the plurality of computing clusters.

[0129] Clause A7. The artificial intelligence accelerator according to Clause A5 or 6, wherein the computing module selects the most efficient splitting combination as the splitting method for the result data based on the output channel dimension for multiple splitting combinations of M and N.

[0130] Clause A8. In the artificial intelligence accelerator described in Clause A7, the computing module starts from the minimum value of the splitting number to determine whether the splitting condition is met. If so, the splitting number is selected as the splitting method for the result data. If not, the module starts from the second smallest value of the splitting number to determine whether the splitting condition is met, and so on until the splitting number that meets the splitting condition is reached.

[0131] Clause A9. In the artificial intelligence accelerator described in Clause A7, when multiple splitting combinations based on the number of splits all satisfy the splitting conditions, the computing module selects the splitting combination with the highest computational efficiency as the splitting method for the result data based on the following computational efficiency relationship:

[0132]

[0133] Where CoL is the length of the output channel, and LT_NUM is the number of computing modules in the AI ​​accelerator. ceil rounds up.

[0134] Clause A10, the artificial intelligence accelerator according to Clause A1, wherein each processor core further includes: a processing module for: sampling multiple data blocks in input data at step intervals; recombining the input data, wherein the multiple data blocks are arranged adjacently to form a data cluster; and performing im2col expansion on the recombined input data and weight data.

[0135] Clause A11, the artificial intelligence accelerator according to Clause A10, wherein the computing module splits the im2col expanded weight data according to the split number, and passes the im2col expanded input data and the split weight data to different computing clusters for matrix multiplication to obtain the result data.

[0136] Clause A12, the artificial intelligence accelerator as described in Clause A10, wherein the input data is input neurons and the weight data is the gradient of the output neurons during backpropagation during neural network training.

[0137] Clause A13. The artificial intelligence accelerator according to Clause A12, wherein the computing module determines whether the number of splits of the result data is less than the total number of computing clusters; if not, the computing module sums the intermediate results of each computing cluster into the result data.

[0138] Clause A14, the artificial intelligence accelerator according to Clause A13, wherein if the number of splits of the result data is less than the total number of computing clusters, the computing module reduces the intermediate results of each computing cluster to obtain the result data.

[0139] Clause A15. The artificial intelligence accelerator according to Clause A14, wherein there are P computing clusters, and when the number of splits is Q, the computing module reduces the intermediate results of each (P / Q) computing cluster to obtain the computing result.

[0140] Clause A16. The artificial intelligence accelerator described in Clause A10 further includes a storage module, the storage module including neuron storage units for storing the input data and the result data.

[0141] Clause A17, an integrated circuit device including an artificial intelligence accelerator as described in any one of Clauses A1 to A16.

[0142] Clause A18, a board including an integrated circuit device as described in Clause A17.

[0143] Clause A19. A method for adaptive split optimization using an artificial intelligence accelerator, the artificial intelligence accelerator comprising multiple computing clusters, each computing cluster comprising multiple processor cores, characterized in that the method comprises: comparing the sum of the storage spaces of the multiple computing clusters with the size of result data, wherein the sum of the storage spaces is the total storage space of the multiple computing clusters; if the sum of the storage spaces is not less than the size of the result data, obtaining a number of splits of the result data based on the number of the multiple computing clusters, wherein the number of splits of the result data is a factor of the number of the multiple computing clusters; if the sum of the storage spaces is less than the size of the result data, obtaining a multiple of the number of splits of the result data as a factor of the number of the multiple computing clusters.

[0144] Clause A20. A computer-readable storage medium having stored thereon computer program code utilizing an artificial intelligence accelerator, which, when run by a processing device, performs the method described in Clause A19.

[0145] Clause A21. A computer program product comprising a computer program utilizing an artificial intelligence accelerator, characterized in that, when the computer program is executed by a processor, it implements the steps of the method described in Clause A19.

[0146] Clause A22. A computer apparatus comprising a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described in Clause A19.

[0147] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An adaptive split-optimized artificial intelligence accelerator, comprising multiple computing clusters, each computing cluster comprising multiple processor cores, characterized in that, Each processor core includes: The storage module includes neuron storage units for storing result data; The computation module is used to: The storage space of a single computing cluster is compared with the size of the result data. If the storage space of a single computing cluster is not less than the size of the result data, the computing module stops obtaining the number of splits of the result data; otherwise, the sum of the storage spaces of the multiple computing clusters is compared with the size of the result data, wherein the sum of the storage spaces is the total storage space of the neuron storage unit in the artificial intelligence accelerator. Based on the number of the plurality of computing clusters, the number of splits of the result data is obtained, wherein: If the sum of the storage spaces is not less than the size of the result data, the number of splits of the result data is a factor of the number of the plurality of computing clusters; If the sum of the storage spaces is less than the size of the result data, the number of splits of the result data is a multiple of the number of the plurality of computing clusters, excluding 1.

2. The artificial intelligence accelerator according to claim 1, wherein the resulting data is the weight gradient of backpropagation during neural network training.

3. The artificial intelligence accelerator according to claim 1, wherein the computation module is used to perform matrix multiplication on the gradients of the input neuron and the output neuron to obtain the result data, and the computation module is further used to split the gradient of the output neuron, wherein the number of splits of the gradient of the output neuron is the same as the number of splits of the result data.

4. The artificial intelligence accelerator according to claim 1, wherein when the sum of the storage spaces is not less than the size of the result data, the computing module is split into M parts along the input channel dimension of the result data and into N parts along the output channel dimension of the result data, wherein M and N are factors of the number of computing clusters participating in matrix multiplication operations.

5. The artificial intelligence accelerator according to claim 1, wherein when the sum of the storage spaces is less than the size of the result data, the computing module is split into M parts along the input channel dimension of the result data and into N parts along the output channel dimension of the result data, wherein M and N are multiples of the number of the plurality of computing clusters, and the multiples do not include 1.

6. The artificial intelligence accelerator according to claim 4 or 5, wherein the computing module selects the most efficient splitting combination as the splitting method of the result data based on the output channel dimension for multiple splitting combinations of M and N that satisfy the splitting conditions.

7. The artificial intelligence accelerator according to claim 6, wherein the operation module starts from the minimum value of the splitting number to determine whether the splitting condition is met; if so, the splitting number is selected as the splitting method of the result data; if not, it starts from the second smallest value of the splitting number to determine whether the splitting condition is met, and so on until the splitting number that meets the splitting condition is reached.

8. The artificial intelligence accelerator according to claim 7, wherein when multiple splitting combinations based on the number of splits all satisfy the splitting condition, the computing module selects the splitting combination with the highest computing efficiency as the splitting method for the result data based on the following computing efficiency relationship: in, CoL is the length of the output channel, and LT_NUM is the number of computing modules in the AI ​​accelerator. ceil rounds up.

9. The artificial intelligence accelerator of claim 1, wherein each computing cluster further comprises: Storage core, used for: Sample multiple data blocks from the input data according to the step size interval; The input data is reorganized, wherein the plurality of data blocks are arranged adjacently to form a data cluster; The recombined input data and weight data are expanded using im2col.

10. The artificial intelligence accelerator according to claim 9, wherein the computing module splits the weight data after im2col expansion according to the split number, and performs matrix multiplication operations on the input data after im2col expansion and the split weight data by different computing clusters to obtain the result data.

11. The artificial intelligence accelerator according to claim 9, wherein the input data is an input neuron, and the weight data is the gradient of the output neuron during backpropagation during neural network training.

12. The artificial intelligence accelerator according to claim 11, wherein the computing module determines whether the number of splits of the result data is less than the total number of computing clusters; if not, the computing module sums the intermediate results of each computing cluster into the result data.

13. The artificial intelligence accelerator according to claim 12, wherein if the number of splits of the result data is less than the total number of computing clusters, the operation module reduces the intermediate results of each P / Q computing cluster to obtain the result data; wherein P is the number of computing clusters and Q is the number of splits.

14. The artificial intelligence accelerator according to claim 9, wherein the storage module includes a neuron storage unit for storing the input data and the result data.

15. An integrated circuit device comprising an artificial intelligence accelerator according to any one of claims 1 to 14.

16. A board comprising the integrated circuit device according to claim 15.

17. A method for adaptive partitioning optimization using an artificial intelligence accelerator, wherein the artificial intelligence accelerator comprises multiple computing clusters, each computing cluster comprising multiple processor cores, characterized in that... The method includes: The storage space of a single computing cluster is compared with the size of the result data. If the storage space of a single computing cluster is not less than the size of the result data, the number of splits of the result data is stopped; otherwise, the sum of the storage spaces of the multiple computing clusters is compared with the size of the result data, where the sum of the storage spaces is the total storage space of the multiple computing clusters. If the sum of the storage spaces is not less than the size of the result data, the number of partitions of the result data is obtained based on the number of partitions of the plurality of computing clusters, wherein the number of partitions of the result data is a factor of the number of the plurality of computing clusters; If the sum of the storage spaces is less than the size of the result data, the number of splits of the result data is a multiple of the number of the plurality of computing clusters, excluding 1.

18. A computer-readable storage medium having stored thereon computer program code utilizing an artificial intelligence accelerator, wherein when the computer program code is run by a processing device, it performs the method of claim 17.

19. A computer program product comprising a computer program utilizing an artificial intelligence accelerator, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 17.

20. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 17.

Images