Data processing method and device for AI chip, chip and computer equipment

Abstract

The embodiment of the application discloses a data processing method and device for an AI chip, a chip and computer equipment, and belongs to the technical field of chips. The method comprises the following steps: an on-chip cache in an AI chip is pre-allocated to n network layers as a buffer, and input data of the n network layers is blocked in the dimension of data processing of the network layers; whenever the data amount of unconsumed data in the buffer of a target network layer in the n network layers meets the execution condition, the target network layer is executed, and the result is written into the buffer of the next network layer. Through the above method, when realizing interlayer fusion in AI acceleration, the resources of the on-chip cache can be maximally utilized, and the processing efficiency of the AI chip on a neural network is improved.

Description

Technical Field

[0001] This application relates to the field of chip technology, and in particular to a data processing method, apparatus, chip, and computer device for AI chips. Background Technology

[0002] Artificial intelligence (AI) chips, also known as AI accelerators, are a type of hardware accelerator specifically designed to accelerate artificial intelligence algorithms.

[0003] When processing neural networks using AI chips, due to limitations in on-chip cache capacity, the input feature map of a network layer is typically divided into blocks and read into the AI ​​chip's on-chip buffer. The read-in feature data is then processed on the AI ​​chip, and the result data is output off-chip for storage after computation. Each network layer in the neural network is processed layer by layer in the same manner.

[0004] However, the above-mentioned layer-by-layer processing method is greatly limited by the bandwidth of off-chip storage and requires frequent off-chip access operations, which affects the processing efficiency of AI chips for neural networks. Summary of the Invention

[0005] This application provides a data processing method, apparatus, chip, and computer device for AI chips. The technical solution includes:

[0006] On one hand, embodiments of this application provide a data processing method for AI chips, the method comprising:

[0007] The method obtains the amount of unconsumed data in the buffers corresponding to n consecutive network layers in a neural network; the buffers are allocated from the on-chip cache of the AI ​​chip; the unconsumed data is the data used by the corresponding network layer during subsequent execution; the data in the buffers is feature data divided into blocks according to the target dimension, and the network layer processes the data according to the target dimension; n is greater than or equal to 2, and n is an integer; the unconsumed data consists of newly written data in the corresponding buffers; or, the unconsumed data consists of newly written data in the corresponding buffers and data in the buffers that were not consumed by the network layer during the previous execution.

[0008] In response to the target input data volume meeting the execution conditions, a target network layer is executed on the AI ​​chip to process the target input data and obtain target output data; the target input data is unconsumed data in the buffer corresponding to the target network layer; the target network layer is at least one of n network layers;

[0009] The target output data is written into the buffer corresponding to the next network layer of the target network layer.

[0010] On the other hand, embodiments of this application provide a data processing apparatus for an AI chip, the apparatus comprising:

[0011] The data acquisition module is used to acquire the amount of unconsumed data in the buffers corresponding to n consecutive network layers in the neural network; the buffers are allocated from the on-chip cache of the AI ​​chip; the unconsumed data is the data used by the corresponding network layer during subsequent execution; the data in the buffers is feature data divided into blocks according to the target dimension, and the network layer processes the data according to the target dimension; n is greater than or equal to 2, and n is an integer; the unconsumed data consists of newly written data in the corresponding buffers; or, the unconsumed data consists of newly written data in the corresponding buffers and data in the buffers that were not consumed by the network layer during the previous execution;

[0012] An execution module is configured to execute a target network layer on the AI ​​chip in response to the target input data volume meeting the execution conditions, to process the target input data and obtain target output data; the target input data is unconsumed data in the buffer corresponding to the target network layer; the target network layer is at least one of n network layers;

[0013] The writing module is used to write the target output data into the buffer corresponding to the next network layer of the target network layer.

[0014] On the other hand, embodiments of this application provide an AI chip, wherein the AI ​​chip is provided with a data processing device for the AI ​​chip as described above.

[0015] On the other hand, embodiments of this application provide a computer device, which includes a CPU chip, an AI chip, and a memory, wherein the CPU chip, the AI ​​chip, and the memory are connected via a bus; the AI ​​chip includes the data processing apparatus for the AI ​​chip as described above.

[0016] The technical solutions provided in this application have at least the following beneficial effects:

[0017] The on-chip cache in the AI ​​chip is pre-allocated to n network layers as buffers, and the input data of the n network layers is divided into blocks according to the dimension of data processing by the network layer. Whenever the amount of unconsumed data in the buffer of the target network layer among the n network layers meets the execution condition, the target network layer is executed, and the result is written to the buffer of the next network layer. According to the above execution method and cache reuse mechanism, when realizing inter-layer fusion for AI acceleration, the on-chip cache resources can be used efficiently, and the redundant calculations caused by the traditional block method are eliminated, thereby improving the processing efficiency of the AI ​​chip for neural networks. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a hardware structure involved in this application;

[0019] Figure 2 This is a schematic diagram of a feature map block involved in this application;

[0020] Figure 3 This is a flowchart of a data processing method for an artificial intelligence (AI) chip provided in an exemplary embodiment of this application;

[0021] Figure 4 This is a flowchart of a data processing method for an artificial intelligence (AI) chip provided in an exemplary embodiment of this application;

[0022] Figure 5 yes Figure 4 The illustrated embodiment is a schematic diagram of an on-chip cache partitioning method.

[0023] Figure 6 yes Figure 4 The illustrated embodiment is a schematic diagram of the network layer execution order.

[0024] Figure 7 yes Figure 4 The illustrated embodiment is a schematic diagram of cache management.

[0025] Figure 8 This is a schematic diagram of the structure of a data processing apparatus for an artificial intelligence (AI) chip provided in an exemplary embodiment of this application;

[0026] Figure 9 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0028] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0029] With the continuous development and widespread application of AI technology, AI networks are becoming increasingly powerful. Consequently, their structures are becoming more complex, and their computational demands are increasing. Therefore, the requirements for processing speed in AI networks are also rising. AI chips are hardware devices developed to meet this demand.

[0030] In one possible scenario, please refer to Figure 1 The diagram illustrates a hardware structure related to this application. Figure 1 As shown, the computer device contains a Central Processing Unit (CPU) chip 13 and an AI chip 12. When data in memory 11 needs to be processed via an AI network, the data in memory 11 can be written to the AI ​​chip 12. The AI ​​chip 12 then performs AI calculations on the written data and writes the results back to memory 11 for the CPU chip 13 to access. Because the AI ​​chip is designed specifically for AI networks, performing AI network calculations via the AI ​​chip 12 significantly improves the execution speed of the AI ​​network compared to executing AI network calculations in software via the CPU chip 13.

[0031] Deep neural networks (DNNs) typically contain a large number of layers, each with a high data access volume. For AI accelerators (i.e., the AI ​​chips mentioned above), especially in edge devices (such as mobile phones and embedded systems), the limited capacity of on-chip cache (Static Random-Access Memory, SRAM) usually prevents the processing of the entire feature map at once. Traditional implementation methods include:

[0032] The layer's output feature map is tiling (the tiling strategy is constrained by SRAM capacity) based on one or more of the four dimensions of NHWC (feature map layout, where N represents the number of features, H represents the height, W represents the width, and C represents the channels). This divides the input / output feature map into several small blocks of the same size. For each small block, it is read from off-chip memory, typically Dynamic Random Access Memory (DRAM), into the AI ​​accelerator's SRAM for caching. After computation, it is output to off-chip memory. This process is repeated for each small block until the layer has finished processing. Each layer in the network processes features in this manner according to the topological order of the neural network.

[0033] The above-mentioned layer-by-layer processing method requires saving all the output of the previous layer to an off-chip location, and then reading the feature map from the off-chip location when executing the next layer. This results in a large amount of data transmission. On the one hand, the performance of the neural network may be reduced due to the bandwidth limitation of the off-chip storage. On the other hand, a large number of accesses to the off-chip storage will also increase the power consumption of the system, resulting in low energy efficiency of the AI ​​accelerator.

[0034] Based on the aforementioned block-based approach, this application proposes a layer fusion method to reduce access to off-chip memory, thereby improving energy efficiency. Considering the locality of data between layers—that is, the output of the previous layer is the input of the next layer (or several layers later)—and since the computation of a portion (block) of data in the next layer does not require the entire output of the previous layer, cross-layer execution (typically convolutional or pooling layers) can be achieved within the capacity of the on-chip cache (SRAM) to reduce the amount of off-chip data transfer.

[0035] However, the above-mentioned inter-layer fusion has the following problems:

[0036] 1) The additional computation and off-chip memory access caused by block partitioning: the more layers are fused, the more additional computation and off-chip memory access there will be, which in turn will reduce the energy efficiency ratio.

[0037] 2) As can be seen from point 1) above, more fusion layers are not necessarily better. Therefore, choosing the optimal fusion method requires a transformation based on a specific search space (2). n The algorithm problem of finding the optimal (best performing) combination from various combinations:

[0038] a) The algorithm needs to evaluate the performance cost of each combination based on the cost model. However, the cost model is an estimation method and has its own errors, which will also lead to some deviation in the final search results.

[0039] b) The search space is exponential. Even if an algorithm with a time complexity of polynomial can be found, its running time is relatively long (ranging from tens of minutes to several hours, depending on the size of the network).

[0040] The following is a brief explanation of the additional computational overhead and off-chip memory access caused by block partitioning. Please refer to [link / reference]. Figure 2 This illustrates a feature map block diagram related to this application. For example... Figure 2 As shown, the example is the fusion of two convolutional layers (Conv1 and Conv2, with n=2 fusion layers). The output (Activation2) of Conv2 has a size of R2 rows, C2 columns, and N2 channels (hereinafter referred to as R2xC2xN2), the output (Activation1) of Conv1 has a size of R1xC1xN1, and the input tensor has a size of R0xC0xN0.

[0041] For simplicity, assume that the kernel size of Conv1 / 2 is K and the stride S is 1. Perform a two-part division on the rows and columns of Activation2 (block factor f = 2), that is, the size of each small block is C2 / 2 x R2 / 2 x N2, which corresponds to the size of Activation1 as (C2 / 2 + K - 1) x (R2 / 2 + K - 1) x N1, and further corresponds to the size of InputTensor as (C2 / 2 + 2(K - 1)) x (R2 / 2 + 2(K - 1)) x N0.

[0042] Without considering additional calculations, the total computational cost (including Conv1 / 2) is:

[0043] K 2 ×N0×N1×C1×R1+K 2 ×N1×N2×C2×R2 (1-1)

[0044] Ignoring additional off-chip memory accesses, the total memory accesses (including reads and writes) are:

[0045] C0×R0×N0+C2×R2×N2 (1-2)

[0046] Considering the additional computation caused by block partitioning and off-chip memory access, the total computational cost and memory access cost are as follows:

[0047] K 2 ×N0×N1×(C1×R1+(K-1)(C1+R1)+(K-1) 2 )+K 2 ×N1×N2×C2×R2(1-3)

[0048] N0×(C0×R0+2(K-1)(C0+R0)+(2K-2) 2)+C2×R2×N2 (1-4)

[0049] Compared to formulas (1-1) and (1-2), the additional computation and off-chip memory access caused by block partitioning are respectively:

[0050] K 2 ×N0×N1×((K-1)(C1+R1)+(K-1) 2 (1-5)

[0051] as well as,

[0052] N0×(2(K-1)(C0+R0)+(2K-2) 2 (1-6)

[0053] As can be seen from the above formula, the additional computation and memory access overhead increases polynomially with the increase of the partitioning factor f and the number of fusion layers n.

[0054] To address the aforementioned problems, subsequent embodiments of this application propose an execution scheme for inter-layer fusion in AI chips, which can improve the processing efficiency of neural networks in AI chips.

[0055] Please refer to Figure 3 This document illustrates a method flowchart for a data processing method for an artificial intelligence (AI) chip, provided in an exemplary embodiment of this application. The method may include the following steps.

[0056] Step 301: Obtain the amount of unconsumed data in the buffers corresponding to n consecutive network layers in the neural network; the buffers are allocated from the on-chip cache of the AI ​​chip; the unconsumed data is the data used by the corresponding network layer during subsequent execution; the data in the buffers is feature data divided into blocks according to the target dimension, and the network layer processes the data according to the target dimension; n is greater than or equal to 2, and n is an integer.

[0057] Among them, the above n network layers are consecutive network layers in the neural network and meet the requirements of inter-layer fusion.

[0058] In the context of neural networks, "continuous" means that in n network layers, the output data of the previous layer is the input data of the next layer.

[0059] In this embodiment, the AI ​​chip can pre-allocate corresponding buffers for each of the n network layers from the on-chip cache, for caching the data to be processed by the corresponding network layer.

[0060] Optionally, the data in the buffer is feature data that is divided into blocks by row, and the network layer processes the data row by row.

[0061] The data in the aforementioned buffer is segmented according to the dimensions followed by which the network layer processes the data. For example, if the network layer processes the data in the feature map row by row, the data in the buffer is also segmented by row. This ensures that the network layer can process the data in the buffer to the maximum extent possible during execution.

[0062] The aforementioned unconsumed data refers to the data that the corresponding network layer needs to use during subsequent execution, while the consumed data refers to the data that the corresponding network layer no longer needs to use during subsequent execution.

[0063] In this embodiment, the unconsumed data consists of newly written data in the corresponding buffer, or the unconsumed data consists of newly written data in the corresponding buffer and data in the buffer that was not consumed by the network layer during the previous execution.

[0064] In neural networks, some data in the feature map of a network layer may be processed multiple times by that layer. For example, taking a convolutional layer as an example, assuming the kernel size is 3×3, when the convolutional layer processes the data in the feature map row by row, in the first execution, the convolutional layer performs convolution processing on rows 1-3 of the feature map. In the second execution, the convolutional layer performs convolution processing on rows 2-4 of the feature map. In these two processing sessions, row 1 of the feature map is processed once, while rows 2 and 3 are each processed twice. After the first execution, the data in row 1 will not participate in subsequent convolution processing. At this point, the data in row 1 is considered consumed data. Although the data in row 2 has been processed once, it still needs further processing and remains unconsumed data. In other words, after the first execution, the data in lines 2 and 3 is not consumed and is retained in the buffer. It then forms new unconsumed data with the data in line 4, which is written later. This allows for the reuse of the data already written in the buffer, eliminating the need to reread the data in lines 2 and 3 from outside the chip when the network layer is executed next time.

[0065] Step 302: In response to the target input data volume meeting the execution conditions, the target network layer is executed on the AI ​​chip to process the target input data and obtain the target output data; the target input data is the unconsumed data in the buffer corresponding to the target network layer; the target network layer is at least one of n network layers.

[0066] The amount of the target input data meets the execution conditions, including the amount of the target input data meeting the data requirements for the target network layer to perform at least one processing operation per row.

[0067] For example, if the target network layer requires 3 rows of data to perform one processing step, then when the target input data reaches 3 rows, its data volume meets the execution condition.

[0068] In this embodiment of the application, whenever the amount of unconsumed data in the buffer of a certain network layer in the on-chip cache of the AI ​​chip meets the execution conditions, the AI ​​chip can execute that network layer.

[0069] Step 303: Write the target output data into the buffer corresponding to the next network layer of the target network layer.

[0070] In this embodiment of the application, after the AI ​​chip executes a network layer, it needs to write the obtained output data into the buffer corresponding to the next network layer so that the next network layer can process it during subsequent execution.

[0071] In summary, the solution shown in this application pre-allocates the on-chip cache of the AI ​​chip to n network layers as buffers, and the input data of the n network layers is divided into blocks according to the dimension of data processing by the network layers. Whenever the amount of unconsumed data in the buffer of the target network layer among the n network layers meets the execution condition, the target network layer is executed, and the result is written to the buffer of the next network layer, thereby improving the processing efficiency of the AI ​​chip for the neural network.

[0072] To execute a specific neural network, in addition to applying the above... Figure 3 The layer fusion method shown requires first dividing the graph into subgraphs and then selecting an appropriate grouping fusion strategy.

[0073] Because neural networks contain a wide variety of layers, not all layers can be executed on AI accelerators, and even those that can may not be fused. Therefore, by using subgraph partitioning based on the topological order of the neural network, it can be roughly divided into the following parts:

[0074] 1) Layers that can be merged together and executed by an AI accelerator (such as Conv / Pool);

[0075] 2) Layers that cannot be integrated for the time being but can still be executed by AI accelerators (such as fully connected layers / long short-term memory neural network layers);

[0076] 3) Layers that cannot or are inconvenient to execute on AI accelerators (such as non-maximum suppression layers).

[0077] For the first layer, the fusion method described in this technology can be used for fusion. Currently, most neural networks are based on the first layer, and the main computational load of the entire network is concentrated in the convolutional layers. Therefore, adopting the technical solution shown in this application can significantly improve the energy efficiency of the entire neural network.

[0078] Since the on-chip cache capacity of AI accelerators is limited and fixed, when processing subgraphs that have been partitioned and handed over to the AI ​​accelerator by the subgraph partitioning function, a large on-chip cache may be required due to the large number of layers, making it impossible to process the entire subgraph at once. Therefore, it is necessary to further divide the subgraph into several groups, and each group is fused based on the inter-layer fusion method of this technical solution. Subsequent embodiments of this application will further introduce the solution involved in this application in conjunction with the grouping strategy and the inter-layer fusion scheme.

[0079] Please refer to Figure 4 This document illustrates a method flowchart for a data processing method for an artificial intelligence (AI) chip, provided in an exemplary embodiment of this application. The method may include the following steps.

[0080] Step 401: Obtain at least two effective partitioning schemes for allocating the on-chip cache of the AI ​​chip to n consecutive network layers in the neural network; the effective partitioning scheme refers to a scheme in which the buffer capacity allocated to each of the n network layers is greater than the amount of input data required for the corresponding network layer to execute once.

[0081] Since grouping incurs additional overhead—each additional group requires an additional off-chip memory access (including writes and reads)—even if adding a new group does not increase computation (based on the inter-layer fusion method of this technical solution), the extra off-chip memory access will still reduce the system's energy efficiency ratio. Therefore, an appropriate grouping fusion strategy is needed to find a suitable grouping method for off-chip memory access.

[0082] The grouping and fusion strategy is as follows: Assuming a subgraph has N layers and the AI ​​accelerator's on-chip cache capacity is C, layers are added sequentially according to the network topology, starting from the first layer, until the on-chip cache cannot hold them all. These layers are then grouped and fused, with this group serving as the split point. The same process is repeated for subsequent groups until all layers have been traversed. Please refer to [reference needed]. Figure 5 This illustrates a schematic diagram of on-chip cache partitioning according to an embodiment of this application. Figure 5As shown, consider a simple case where the subgraph is divided into two groups as described above, denoted as group 0 and group 1. Group 0 consists of layers 0 to 1k, occupying approximately C of on-chip cache, with off-chip memory access of f0. Group 1 consists of layers 1k+1 to 1N-1, occupying approximately CM of on-chip cache, with off-chip memory access of f1. The total off-chip memory access under this combination is f0 + f1. Note that since group 1 does not fully utilize the on-chip cache, the starting position of this group can be adjusted layer by layer backward, increasing the required on-chip cache accordingly until the capacity reaches C. Assuming this corresponds to a callback to layer 1k', there are k-k' combinations from layer 1k to layer 1k'.

[0083] Step 402: Obtain the off-chip memory access amounts corresponding to at least two valid partitioning schemes.

[0084] Based on the above Figure 5 For example, there are k-k'+1 possible external memory accesses (plus f0+f1 corresponding to the original group). Generalizing, suppose there are P original groups, denoted as group0 / 1 / ... / P-1; let r = k-k' be the number of layers groupP-1 can call back to; after layer adjustment, let O(P) be the smallest external memory access of the P groups; groupP-1 can be adjusted r times, and let f(P-1) be the external memory access corresponding to groupP-1. i Let i = 0, 1, ..., r, where i = 0 indicates no adjustment is made; each time group P-1 is adjusted, the minimum out-of-chip memory access of the first P-1 groups is denoted as O(P-1). i ,i=0,1,…,r,whereO(P-1)0=∑ p∈[0,P-2] If f(p) ≥ 0, then for group P-1, the optimal adjustment position jP-1 can be expressed as:

[0085] j P-1 =argmin i∈[0,r] (O(P-1) i +f(P-1) i (2-1)

[0086] In formula (2-1), f(P-1) i It is definite, O(P-1). i It is a problem to be solved. In fact, O(P-1) i This is a subproblem similar to (2-1), which requires solving r j... P-2 And so on, until the solution reaches j1.

[0087] Step 403: Based on the off-chip memory access corresponding to at least two of the effective partitioning schemes, determine the target partitioning scheme from the at least two effective partitioning schemes.

[0088] In one possible implementation, the AI ​​chip can determine the target partitioning scheme as the one with the minimum off-chip memory access among at least two valid partitioning schemes.

[0089] The problem of finding the optimal partitioning scheme described above can be solved using a recursive algorithm. The algorithm's time complexity is polynomial, therefore it does not consume much execution time. Algorithms for searching the optimal grouping strategy are shown in Table 1.

[0090] Table 1

[0091]

[0092]

[0093] The algorithm pseudocode (Python-like syntax representation) is shown in Table 1 above. Line 25, InitGroup, follows a similar structure. Figure 5The method (e.g., layer 0 to k is group 0, layer k to N-1 is group 1) initializes the original groups and their corresponding off-chip memory accesses. Line 26 passes these two arrays to SearchGroup for searching (orig_g_array is a two-dimensional array, the first dimension is the number of groups, and the second dimension is the number of layers within a group); line 3 accumulates the off-chip memory accesses of all groups; line 4 determines the number of layers r that the rightmost group can be adjusted backward (left) under the current grouping method by calling CheckAvailRollback. After each adjustment, the off-chip memory accesses of the left and right groups (a total of r possibilities) are represented as one-dimensional arrays f_rb_lhs and f_rb_rhs, and the left group is group 1. The r possible values ​​are represented by a two-dimensional array g_rb. Line 5 checks if r is 0; if so, there is no room for adjustment, the callback level is recorded as 0, and the function returns directly. Otherwise, the search continues. Lines 9 to 19 iterate over the r adjustable ranges. When the first dimension of g_array (number of groups) is greater than 2, lines 11 and 12 determine all adjusted (sub)groups and their corresponding off-chip memory access values. Line 13 passes these values ​​as parameters to SearchGroup for recursive calls. The recursive function SearchGroup returns the minimum total off-chip memory access value among all possible adjustments in the current group, and records the callback level of the last (right) group as j, saving it along with the current group g_array to j_map. After the algorithm completes the search, the position that each group needs to be adjusted can be traced back through j_map, which is the optimal group with the minimum total off-chip memory access value.

[0094] Step 404: According to the target partitioning scheme, allocate the buffer for n network layers from the on-chip cache of the AI ​​chip.

[0095] In this embodiment of the application, after determining the target partitioning scheme, the AI ​​chip can allocate the on-chip cache to n network layers as a buffer.

[0096] In another possible implementation, the process of determining the target partitioning scheme in steps 401 to 403 above can also be executed by other chips besides the AI ​​chip (such as by the CPU), and the determined target partitioning scheme can be notified to the AI ​​chip, so that the AI ​​chip can allocate cache.

[0097] Step 405: Obtain the amount of unconsumed data in the buffers corresponding to each of the n network layers; the buffers are allocated from the on-chip cache of the AI ​​chip; the unconsumed data is the data used by the corresponding network layer during subsequent execution; the data in the buffers is feature data divided into blocks according to the target dimension, and the network layer processes the data according to the target dimension; n is greater than or equal to 2, and n is an integer.

[0098] In this embodiment, the AI ​​chip can maintain a set of pointers for each network layer to indicate the consumption progress and production progress of the data in the buffer corresponding to that network layer, thereby indicating the amount of unconsumed data in the buffers corresponding to the n network layers respectively.

[0099] Step 406: In response to the fact that the amount of unconsumed data in the buffer corresponding to the first network layer in the n network layers does not meet the execution condition, a new data block is written into the buffer corresponding to the first network layer. The data block is the feature data segmented from the input feature map of the first network layer according to the target dimension.

[0100] The aforementioned unconsumed data consists of newly written data in the corresponding buffer, or it consists of newly written data in the corresponding buffer and data in the buffer that was not consumed by the network layer during the previous execution.

[0101] Specifically, when the data in the buffer corresponding to a certain network layer first reaches the execution condition of the network layer, the unconsumed data at this time are all newly written data in the buffer. When the data in the buffer corresponding to the network layer is processed and subsequently reaches the execution condition of the network layer again, the unconsumed data may contain both newly written data and data that was not consumed by the network layer in the previous execution.

[0102] In this embodiment of the application, taking the feature map divided into data blocks by row as an example, whenever the amount of unconsumed data in the buffer corresponding to the first network layer does not meet the execution conditions, the AI ​​chip can read a new row of data blocks from outside the chip and write the new row of data blocks into the buffer corresponding to the first network layer until all the data in the feature map corresponding to the first network layer has been read, or the amount of unconsumed data in the buffer corresponding to the first network layer meets the execution conditions.

[0103] When reading in a new data block, if there is no free buffer space in the buffer corresponding to the first network layer, the new data block is written to the buffer space of consumed data in the buffer corresponding to the first network layer in an overwrite manner.

[0104] In this embodiment of the application, when there is no free buffer space in the buffer and the amount of unconsumed data does not meet the execution conditions, it indicates that consumed data occupies the buffer space. At this time, a new data block can be used to overwrite the consumed data. The new data block and the unconsumed data in the buffer form the data used by the network layer in the next execution, thereby realizing the reuse of data in the buffer.

[0105] Accordingly, if there is free buffer space in the buffer corresponding to the first network layer, and the capacity of the free buffer space is not less than the data size of the new data block, then the data block is written into the free buffer space.

[0106] Step 407: In response to the target input data volume meeting the execution conditions, the target network layer is executed on the AI ​​chip to process the target input data and obtain the target output data; the target input data is the unconsumed data in the buffer corresponding to the target network layer; the target network layer is at least one of n network layers.

[0107] Wherein, in response to the aforementioned data volume of the target input data satisfying the execution conditions, the target network layer is executed on the AI ​​chip to process the target input data and obtain target output data, including:

[0108] In response to the fact that the number of the target network layers is at least two, at least two of the target network layers are executed sequentially on the AI ​​chip in the order from deep to shallow in the neural network.

[0109] Step 408: Write the target output data into the buffer corresponding to the next network layer of the target network layer.

[0110] In one possible implementation, writing the target output data to the buffer corresponding to the next network layer of the target network layer includes:

[0111] In response to the absence of free buffer space in the buffer corresponding to the next network layer of the target network layer, the target output data is written to the buffer space of consumed data in the buffer corresponding to the next network layer of the target network layer in an overwrite manner; the consumed data is data that will not be used in subsequent executions of the corresponding network layer.

[0112] In this embodiment, since the output of the previous network layer is the input of the next network layer, in order to ensure that the data cached on the chip can be executed normally, when multiple network layers meet the execution conditions at the same time, the AI ​​chip will execute the multiple network layers in the order from back to front (i.e. from deep to shallow in the neural network). After executing each network layer, the output result data of that network layer is written into the buffer corresponding to the next network layer. This ensures that the output result of each network layer will not overwrite the unconsumed data in the next layer after being written into the next layer.

[0113] Taking the data in the buffer as feature data that is divided into blocks according to rows, and the network layer processes the data by row as an example, the solution shown in this application proposes an inter-layer fusion approach based on row-by-row block division, row-by-row execution, and cache reuse to solve the shortcomings of traditional block division methods, such as additional computation, off-chip memory access, and the resulting long search time and inaccurate results. At the same time, the cache reuse mechanism based on the circular buffer can further reduce the demand for on-chip cache.

[0114] Let's take the fusion of two convolutional layers as an example again. Row-by-row partitioning refers to dividing the feature tensor along the R dimension, with each block corresponding to a row. Please refer to [reference needed]. Figure 6 This illustrates a schematic diagram of the network layer execution sequence involved in an embodiment of this application. For example... Figure 6 As shown, for example, the input tensor InputTensor has a block size of 1×C0×N0. The execution order is as follows:

[0115] 1) Execute Conv1. The input is Input, which has a size of 3xC0xN0 and corresponds to three lines of R0 = 0 / 1 / 2. The output is Activation1 (output 1), which has a size of 1×C1×N1 and corresponds to one line of R1 = 0.

[0116] 2) Execute Conv1. The input is Input, which has a size of 3×C0×N0 and corresponds to three lines of R0=1 / 2 / 3; the output is Activation1, which has a size of 1×C1×N1 and corresponds to one line of R1=1.

[0117] 3) Execute Conv1. The input is Input, which has a size of 3×C0×N0 and corresponds to three lines of R0=2 / 3 / 4; the output is Activation1, which has a size of 1×C1×N1 and corresponds to one line of R1=2.

[0118] 4) Execute Conv2. The input is Activation1, with a size of 3×C1×N1, corresponding to three rows of R1=0 / 1 / 2; the output is Activation2 (output Activation2), with a size of 1×C2×N2, corresponding to one row of R2=0;

[0119] 5) Execute Conv1. The input is Input, which has a size of 3×C0×N0 and corresponds to three lines of R0=3 / 4 / 5; the output is Activation1, which has a size of 1×C1×N1 and corresponds to one line of R1=3.

[0120] 6) Execute Conv2. The input is Activation1, which has a size of 3×C1×N1 and corresponds to three rows of R1=1 / 2 / 3. The output is Activation2, which has a size of 1×C2×N2 and corresponds to one row of R2=1.

[0121] In this embodiment, the line-by-line execution process is not simply executed layer by layer according to the fusion order. Instead, it determines which layer should execute based on the input of the relevant layer. For example, in step 4 above, since the input of Conv2 (R1 = 0 / 1 / 2, output by Conv1) is sufficient to output a new line (R2 = 0), the execution right is transferred from Conv1 to Conv2. In step 5, since Conv2 has no new input and cannot output a new line, the execution right is transferred back to Conv1. In step 6, Conv2 receives a new line of input (R1 = 3), the execution right is transferred back to Conv2, and a new line is output (R2 = 1).

[0122] Cache reuse is managed based on the ring buffer; please refer to [reference needed]. Figure 7 It illustrates a schematic diagram of cache management involved in an embodiment of this application; as shown Figure 7 As shown, for Input and Activation1, only a minimum of 3 rows of space need to be allocated for buffering. In step 1, all 3 rows of space in the input buffer are filled, and Conv1 outputs to the first row of space in Activation1 (R1=0). In step 2, a new row (R0=3) is loaded from outside the chip into the input buffer, overwriting R0=0, while reusing R0=1 / 2, and Conv1 outputs to the second row of space in Activation1 (R1=1). Step 3 is similar to step 2, the input buffer continues to load a new row and generate output; the management of the output 1 buffer (Activation1 buffer) is similar to the management of the input buffer. The row block execution logic can be shown in Table 2 below:

[0123] Table 2

[0124]

[0125]

[0126] As shown in the pseudocode in Table 2, for each layer, a pair of pointers (w_ptr and r_ptr) are maintained, representing the progress of input buffer production and consumption, respectively, in rows; stride is the step size of each layer, representing the number of rows consumed in one execution of that layer; thr is the threshold for the number of unconsumed rows in the input buffer, indicating the input threshold that the layer needs to meet to execute; out_rows represents the number of output rows for each layer in one execution; ring_buf_rows records the maximum number of rows required for each layer to input the ring buffer; cur_layer represents the currently executing layer; stack_layer stores the next layer that can be executed if cur_layer does not meet the execution conditions. The InitEnv function sets variables such as stride, thr, and out_rows according to the network being executed. Line 9 of the code checks if `cur_layer` meets the conditions for execution. If not, it attempts to execute `stack_layer` (line 29). Lines 12 to 15 are the assignment logic for `stack_layer`. If the layer above the current layer (`cur_layer-1`) also meets the execution conditions, the layer above it is used as `stack_layer` if the current layer is executed first. Otherwise, `stack_layer` is 0 (returning to the first layer for execution). Line 19 indicates the execution of `cur_layer`. Lines 18 and 21 update the `r_ptr` and `w_ptr` of the current layer and the next layer, respectively. Line 22 updates `ring_buf_rows` with the latest `w_ptr` and `r_ptr`, and allocates `ringbuffer` space with the final value. Line 31 checks if the last layer has finished outputting. If so, it means the entire network execution has ended, and the execution logic exits.

[0127] The solution described in the above embodiments of this application proposes an inter-layer fusion approach based on row-by-row block partitioning, row-by-row execution, and cache reuse to solve the shortcomings of traditional block partitioning methods, such as additional computation, off-chip memory access, and the resulting long search time and inaccurate results. At the same time, the cache reuse mechanism based on loops can further reduce the demand for on-chip cache.

[0128] Based on this, this technical solution also proposes a grouping search algorithm, which can accurately find the optimal grouping scheme in a short time (polynomial complexity) when the on-chip SRAM is limited, reduce the off-chip memory access to the lowest possible level, thereby reducing system power consumption and improving energy efficiency.

[0129] In summary, the solution shown in this application embodiment, in addition to improving the processing efficiency of the AI ​​chip for the neural network through the above-mentioned execution method, also determines a suitable grouping scheme through a grouping search scheme, thereby minimizing the off-chip memory access during subsequent network layer execution.

[0130] The technical solutions described in the above embodiments of this application have significant effects on improving the performance and energy efficiency of AI accelerators, especially in handheld devices (such as mobile phones and embedded devices) running deep neural networks. On the one hand, they improve the execution efficiency of AI accelerators by eliminating unnecessary redundant computations; on the other hand, they improve the performance and energy efficiency of AI accelerators by reducing unnecessary off-chip memory accesses. Simultaneously, the cache reuse mechanism can reduce the demand for on-chip SRAM capacity by AI accelerators to a certain extent, thereby reducing costs. Therefore, the solutions described in this application have a very positive effect on improving overall system performance and reducing energy consumption and costs.

[0131] The inter-layer fusion strategy described in this application can schedule instruction generation through software and run on existing hardware platforms, exhibiting a certain degree of versatility. Furthermore, these fusion strategies can also be implemented in hardware, further improving performance and energy efficiency, making them suitable for certain relatively fixed scenarios with higher performance requirements.

[0132] Please refer to Figure 8 This illustration shows a structural schematic diagram of a data processing apparatus for an artificial intelligence (AI) chip provided in an exemplary embodiment of this application. The apparatus includes:

[0133] The data acquisition module 801 is used to acquire the amount of unconsumed data in the buffers corresponding to n consecutive network layers in the neural network; the buffers are allocated from the on-chip cache of the AI ​​chip; the unconsumed data is the data used by the corresponding network layer in subsequent execution; the data in the buffers is feature data divided into blocks according to the target dimension, and the network layer processes the data according to the target dimension; n is greater than or equal to 2, and n is an integer; the unconsumed data consists of newly written data in the corresponding buffers; or, the unconsumed data consists of newly written data in the corresponding buffers and data in the buffers that were not consumed by the network layer in the previous execution;

[0134] The execution module 802 is configured to execute a target network layer on the AI ​​chip in response to the target input data volume meeting the execution conditions, so as to process the target input data and obtain target output data; the target input data is unconsumed data in the buffer corresponding to the target network layer; the target network layer is at least one of the n network layers;

[0135] The writing module 803 is used to write the target output data into the buffer corresponding to the next network layer of the target network layer.

[0136] In one possible implementation, the execution module 802 is configured to execute at least two of the target network layers sequentially on the AI ​​chip in order from deep to shallow in the neural network, in response to the number of target network layers being at least two.

[0137] In one possible implementation, the writing module 803 is used for,

[0138] In response to the absence of free buffer space in the buffer corresponding to the next network layer of the target network layer, the target output data is written to the consumed data buffer space in the buffer corresponding to the next network layer of the target network layer in an overwrite manner; the consumed data is data that will not be used in subsequent executions of the corresponding network layer.

[0139] In one possible implementation, the writing module 803 is further configured to write a new data block into the buffer corresponding to the first network layer in response to the fact that the amount of unconsumed data in the buffer of the first network layer among the n network layers does not meet the execution condition. The data block is feature data segmented from the input feature map of the first network layer according to the target dimension.

[0140] In one possible implementation, the device further includes:

[0141] The partitioning scheme acquisition module is used to acquire at least two effective partitioning schemes for allocating the on-chip cache of the AI ​​chip to the n network layers before the data volume acquisition module acquires the amount of unconsumed data in the buffers corresponding to the n consecutive network layers in the neural network. The effective partitioning scheme refers to a scheme in which the buffer capacity allocated to each of the n network layers is greater than the amount of input data required for the corresponding network layer to execute once.

[0142] The memory access acquisition module is used to acquire the off-chip memory access corresponding to at least two of the effective partitioning schemes;

[0143] The partitioning scheme determination module is used to determine a target partitioning scheme from at least two effective partitioning schemes based on the off-chip memory access volume corresponding to at least two effective partitioning schemes respectively;

[0144] A buffer allocation module is used to allocate buffers for n network layers from the on-chip cache of the AI ​​chip according to the target partitioning scheme.

[0145] In one possible implementation, the partitioning scheme determination module is used to determine the scheme with the minimum off-chip memory access among at least two effective partitioning schemes as the target partitioning scheme.

[0146] In one possible implementation, the data in the buffer is feature data divided into blocks by row, and the network layer processes the data row by row.

[0147] In summary, the solution presented in this application, through group search and inter-layer fusion execution combined with cache reuse, can effectively reduce the use of on-chip cache resources and reduce off-chip memory access, thereby greatly improving the processing efficiency of AI chips for neural networks.

[0148] This application also provides an AI chip, which includes a data processing device for artificial intelligence AI chips as shown in the above embodiments. When the AI ​​chip needs to perform neural network acceleration tasks, the on-chip cache of the AI ​​chip can be allocated to n network layers as buffers. The input data of the n network layers is divided into blocks according to the dimension of data processing by the network layer. Whenever the amount of unconsumed data in the buffer of the target network layer among the n network layers meets the execution condition, the AI ​​chip executes the target network layer and writes the result to the buffer of the next network layer.

[0149] Please refer to Figure 9 The diagram illustrates the structure of a computer device provided in an exemplary embodiment of this application. The computer device includes a CPU chip 910, an AI chip 920, and a memory 930. The CPU chip 910, AI chip 920, and memory 930 are connected via a bus 940. The AI ​​chip 920 is equipped with the data processing device for artificial intelligence (AI) chips provided in the above embodiment.

[0150] like Figure 9 As shown, the AI ​​chip 920 has an on-chip cache 922. When the AI ​​chip needs to perform neural network acceleration tasks, the on-chip cache 922 can be allocated to n network layers as a buffer.

[0151] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0152] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method for an artificial intelligence (AI) chip, comprising: The method comprises: obtaining the data amount of unconsumed data in the buffer area corresponding to each of the continuous n network layers in the neural network; the buffer area is allocated from the on-chip cache of the AI chip; the unconsumed data is data used in subsequent execution of the corresponding network layer; the data in the buffer area is feature data blocked according to a target dimension, and the network layer processes data according to the target dimension; n is greater than or equal to 2, and n is an integer; the unconsumed data is composed of newly written data in the corresponding buffer area; or, the unconsumed data is composed of newly written data in the corresponding buffer area and data not consumed by the network layer in the last execution in the buffer area; in response to the data amount of target input data meeting an execution condition, executing a target network layer on the AI chip to process the target input data and obtain target output data; the target input data is unconsumed data in the buffer area corresponding to the target network layer; the target network layer is at least one of the n network layers; writing the target output data into the buffer area corresponding to the next network layer of the target network layer.

2. The method of claim 1, wherein, The method comprises: in response to the number of target network layers being at least two, executing at least two target network layers on the AI chip in the order from deep to shallow in the neural network.

3. The method of claim 2, wherein, The method comprises: in response to the buffer area corresponding to the next network layer of the target network layer not having free buffer space, writing the target output data into the buffer space of consumed data in the buffer area corresponding to the next network layer of the target network layer in an overlaid manner; the consumed data is data not used in subsequent execution of the corresponding network layer.

4. The method of claim 1, wherein, The method further comprises: in response to the data amount of unconsumed data in the buffer area corresponding to the first network layer of the n network layers not meeting the execution condition, writing a new data block into the buffer area corresponding to the first network layer; the data block is feature data cut from the input feature map of the first network layer according to the target dimension.

5. The method of claim 1, wherein, The method further comprises: obtaining at least two effective division schemes for allocating the on-chip cache of the AI chip to the n network layers; the effective division scheme refers to a scheme in which the buffer capacity allocated to each of the n network layers is greater than the data amount of input data required for execution of the corresponding network layer once; obtaining the off-chip memory access amount corresponding to each of the at least two effective division schemes; determining a target division scheme from the at least two effective division schemes based on the off-chip memory access amount corresponding to each of the at least two effective division schemes; According to the target division scheme, the buffer is allocated for n network layers from the on-chip cache of the AI chip.

6. The method of claim 5, wherein, The target division scheme is determined from at least two effective division schemes based on the off-chip memory access amount corresponding to each of the at least two effective division schemes. The scheme corresponding to the minimum off-chip memory access amount is determined as the target division scheme from the at least two effective division schemes.

7. The method according to any one of claims 1 to 6, characterized in that, The data in the buffer is feature data blocked by rows, and the network layer processes data by rows.

8. A data processing device for artificial intelligence (AI) chips, characterized in that, The device comprises: The data amount acquisition module is configured to acquire data amounts of unconsumed data in buffers corresponding to consecutive n network layers in a neural network; the buffers are allocated from an on-chip cache of an AI chip; the unconsumed data is data used in subsequent execution of the corresponding network layer; the data in the buffer is feature data blocked by a target dimension, and the network layer processes data by the target dimension; n is greater than or equal to 2 and is an integer; the unconsumed data is composed of newly written data in the corresponding buffer; or, the unconsumed data is composed of newly written data in the corresponding buffer and data not consumed by the network layer in the last execution in the buffer; The execution module is configured to execute a target network layer on the AI chip to process target input data and obtain target output data in response to the data amount of the target input data satisfying an execution condition; the target input data is unconsumed data in the buffer corresponding to the target network layer; the target network layer is at least one of the n network layers; The writing module is configured to write the target output data into the buffer corresponding to a next network layer of the target network layer.

9. The apparatus of claim 8, wherein, The execution module is configured to execute at least two target network layers on the AI chip in sequence according to a deep-to-shallow order in the neural network in response to the number of target network layers being at least two.

10. The apparatus of claim 9, wherein, The writing module is configured to, write the target output data into consumed data buffer space in the buffer corresponding to the next network layer of the target network layer in an overwriting manner in response to the buffer space being not free in the buffer corresponding to the next network layer of the target network layer; the consumed data is data not used in subsequent execution of the corresponding network layer.

11. The apparatus of claim 8, wherein, The writing module is further configured to write a new data block into the buffer corresponding to the first network layer in response to the data amount of unconsumed data in the buffer corresponding to the first network layer in the n network layers not satisfying the execution condition; the data block is feature data cut from an input feature map of the first network layer according to the target dimension.

12. The apparatus of claim 8, wherein, The device further comprises: The division scheme acquisition module is configured to acquire at least two effective division schemes for allocating on-chip buffers of the AI chip to n network layers before the data amount acquisition module acquires data amounts of unconsumed data in the buffer areas corresponding to the n network layers in the neural network. The memory access amount acquisition module is configured to acquire off-chip memory access amounts corresponding to the at least two effective division schemes. The division scheme determination module is configured to determine a target division scheme from the at least two effective division schemes based on the off-chip memory access amounts corresponding to the at least two effective division schemes. The buffer allocation module is configured to allocate the buffer areas from the on-chip buffers of the AI chip to the n network layers according to the target division scheme.

13. The apparatus of claim 12, wherein, The division scheme determination module is configured to determine the scheme corresponding to the minimum off-chip memory access amount from the at least two effective division schemes as the target division scheme.

14. The apparatus of any one of claims 8 to 13, wherein, The data in the buffer area is feature data that is blocked by rows, and the network layer processes data by rows.

15. An artificial intelligence, AI, chip, wherein The AI chip is provided with the data processing device according to any one of claims 8 to 14.

16. A computer device, comprising: The computer device includes a central processing unit (CPU) chip, an AI chip, and a memory, and the CPU chip, the AI chip, and the memory are connected through a bus. The AI chip includes the data processing device according to any one of claims 8 to 14.

Images